David Arnold (Jan 22 2024 at 16:43): David Arnold (Jan 22 2024 at 16:43):Shrink the initrd, how?
The documentation states a minimum 1.5GB RAM requirement, but my target host has only 900MB (yeah!, needs to be a cheap proxy).
Does anyone know how I can shrink the initrd (makeInitrdNG?)? — and even more importantly test that reliably?
What is all that is there in an initrd and how can I minimalize that which there is? — I've never touched this topic and am finding myself on the bottom of a climb that I wouldn't want to do on my own, alone.
Shivaraj B H (Jan 22 2024 at 18:00):kexec image uses netboot-base.nix module which depends on profiles/base.nix, and this base module has a lot of packages by default, you can probably override it to remove unwanted base packages, which will reduce the kexec image size.
Shivaraj B H (Jan 22 2024 at 18:04):(deleted)
David Arnold (Jan 22 2024 at 18:07):I think it's here in the squashfsStore where it get's heavy:
# Create the initrd
system.build.netbootRamdisk = pkgs.makeInitrdNG {
inherit (config.boot.initrd) compressor;
prepend = [ "${config.system.build.initialRamdisk}/initrd" ];
contents =
[ { object = config.system.build.squashfsStore;
symlink = "/nix-store.squashfs";
}
];
};
and then here:
initialRamdisk = pkgs.makeInitrdNG {
name = "initrd-${kernel-name}";
inherit (config.boot.initrd) compressor compressorArgs prepend;
inherit (cfg) strip;
contents = map (path: { object = path; symlink = ""; }) (subtractLists cfg.suppressedStorePaths cfg.storePaths)
++ mapAttrsToList (_: v: { object = v.source; symlink = v.target; }) (filterAttrs (_: v: v.enable) cfg.contents);
};
with
contents = {
"/tmp/.keep".text = "systemd requires the /tmp mount point in the initrd cpio archive";
"/init".source = "${cfg.package}/lib/systemd/systemd";
"/etc/systemd/system".source = stage1Units;
"/etc/systemd/system.conf".text = ''
[Manager]
DefaultEnvironment=PATH=/bin:/sbin
${cfg.extraConfig}
ManagerEnvironment=${lib.concatStringsSep " " (lib.mapAttrsToList (n: v: "${n}=${lib.escapeShellArg v}") cfg.managerEnvironment)}
'';
"/lib".source = "${modulesClosure}/lib";
"/etc/modules-load.d/nixos.conf".text = concatStringsSep "\n" config.boot.initrd.kernelModules;
# We can use either ! or * to lock the root account in the
# console, but some software like OpenSSH won't even allow you
# to log in with an SSH key if you use ! so we use * instead
"/etc/shadow".text = "root:${if isBool cfg.emergencyAccess then optionalString (!cfg.emergencyAccess) "*" else cfg.emergencyAccess}:::::::";
"/bin".source = "${initrdBinEnv}/bin";
"/sbin".source = "${initrdBinEnv}/sbin";
"/etc/sysctl.d/nixos.conf".text = "kernel.modprobe = /sbin/modprobe";
"/etc/modprobe.d/systemd.conf".source = "${cfg.package}/lib/modprobe.d/systemd.conf";
"/etc/modprobe.d/ubuntu.conf".source = pkgs.runCommand "initrd-kmod-blacklist-ubuntu" { } ''
${pkgs.buildPackages.perl}/bin/perl -0pe 's/## file: iwlwifi.conf(.+?)##/##/s;' $src > $out
'';
"/etc/modprobe.d/debian.conf".source = pkgs.kmod-debian-aliases;
"/etc/os-release".source = config.boot.initrd.osRelease;
"/etc/initrd-release".source = config.boot.initrd.osRelease;
} // optionalAttrs (config.environment.etc ? "modprobe.d/nixos.conf") {
"/etc/modprobe.d/nixos.conf".source = config.environment.etc."modprobe.d/nixos.conf".source;
};
If you have an embedded usecade, clever's not-os might be a better choice fundamentally, as this is its goal from the outset.
David Arnold (Jan 22 2024 at 18:08):That'd be fun, but in my case not really necessary. I just got a cheap IPv6-only 900mb vps collocated in Bogota (which is key, and otherwise very expensive).
Shivaraj B H (Jan 22 2024 at 19:49):I doubt it is the initialRamdisk, on my nixos machine when I check the size of initrd it shows just 12 MB:
ls -lah /nix/store/l8qssxmf3ff1c6g2iq6klcfahclxxgzb-initrd-linux-6.1.71/initrd
.r--r--r-- root root 12 MB Thu Jan 1 05:30:01 1970 /nix/store/l8qssxmf3ff1c6g2iq6klcfahclxxgzb-initrd-linux-6.1.71/initrd
This is how I built the initrd:
nix build .#nixosConfigurations.nixos.config.system.build.initialRamdisk
I'll re-check tomorrow. I somehow need a better testing handle on this piece of software to be able to actually assess the max- memory consumption.
Last time I felt lucky and just deployed, then the VM got stuck and hanged and I had to wait two business days for a manual intervention. Not fun :smile:
Tim DeHerrera (Jan 23 2024 at 00:36):900mb is pretty tight. Definitely use an OOM killer daemon on that thing. That will at least help you avoid another manual intervention :sweat_smile:
sedlund (Jan 23 2024 at 05:52):this will get you much further
zramSwap = {
enable = true;
memoryPercent = 150;
};
Yeah, I just need it as a reverse proxy with stable IP as failover route in front of a mobile network backup internet link for a site with normally static IP wire-line link, where the site tunnels via wireguard onto that machine to establishes a stable route in case of issue with the main link.
I think an OOM daemon might not do much, because it's already after the kexec when this happens and apparently during the loading of the initrd.
@sedlund , I'll try your suggestion, although the docs say that swap wouldn't help much and that the requirement is 1.5gb (exclusive of swap). I didn't know about zram, this is really interesting!
They don't explain _why_ "no swap" and maybe they should be more precise and change that to "no disk swap" if your suspicion is right.
Shivaraj B H (Jan 23 2024 at 17:51):With nixos-anywhere you are looking to format your disk while being on the ram and swap exists on the disk, which is why it doesn’t help.
David Arnold (Jan 23 2024 at 19:37):I haven't come around to test this yet, but it looks like zramSwap is a compressed swap on ram, effectively making the RAM apparently bigger than it really is through compression. If that work's that'd be fun and really nice.
sedlund (Jan 24 2024 at 03:06):right, 150% is what google uses on ChromeOS. you will see typically 4:1 compression in regular workloads. the setting is not particular to nixos-anywhere (when building remotely), just useful on ram limited machines and in general. Windows does ram compression by default as does Fedora with zram
Tim DeHerrera (Jan 24 2024 at 03:22):fwiw I also use a cheap RAM limited vps from digitalocean (1G). I haven't yet installed nixos on it (intended to, but I was in a rush at the time).
Its only use is as a nebula lighthouse to create a network between some of my personal devices, and it hasn't had any issue since I set it up so I kinda forgot about it.
Might have to circle back around at some point, if just for usage comparison's sake.
Last updated: Nov 15 2024 at 12:33 UTC