This should be a significant disk space saving for most NixOS
installations. This method is a bit more complicated than doing it in
the postInstall for the firmware derivations, but this way it's
automatic, so each firmware package doesn't have to separately
implement its compression.
Currently, only xz compression is supported, but it's likely that
future versions of Linux will additionally support zstd, so I've
written the code in such a way that it would be very easy to implement
zstd compression for those kernels when they arrive, falling back to
xz for older (current) kernels.
I chose the highest possible level of compression (xz -9) because even
at this level, decompression time is negligible. Here's how long it took
to decompress every firmware file my laptop uses:
i915/kbl_dmc_ver1_04.bin 2ms
regulatory.db 4ms
regulatory.db.p7s 3ms
iwlwifi-7265D-29.ucode 62ms
9d71-GOOGLE-EVEMAX-0-tplg.bin 22ms
intel/dsp_fw_kbl.bin 65ms
dsp_lib_dsm_core_spt_release.bin 6ms
intel/ibt-hw-37.8.10-fw-22.50.19.14.f.bseq 7ms
And since booting NixOS is a parallel process, it's unlikely (but
difficult to measure) that the time to user interaction was held up at
all by most of these.
Fixes (partially?) #148197
cpio includes the number of directory hard links in archives it creates.
Some filesystems, like btrfs, do not count directory hard links the same
way as more common filesystems like ext4 or tmpfs, so archives built
when /tmp is on such a filesystem do not reproduce. This patch replaces
cpio with bsdtar, which does not have this issue. The specific
invocation is from this page:
https://reproducible-builds.org/docs/archives/
The `platform` field is pointless nesting: it's just stuff that happens
to be defined together, and that should be an implementation detail.
This instead makes `linux-kernel` and `gcc` top level fields in platform
configs. They join `rustc` there [all are optional], which was put there
and not in `platform` in anticipation of a change like this.
`linux-kernel.arch` in particular also becomes `linuxArch`, to match the
other `*Arch`es.
The next step after is this to combine the *specific* machines from
`lib.systems.platforms` with `lib.systems.examples`, keeping just the
"multiplatform" ones for defaulting.
Since fdf32154fc, we no longer allow
missing modules in the initrd. Unfortunately since before this commit,
the modules-closure script would also fail on missing firmware, which
is a very common case (e.g. xhci-pci.ko.xz lists renesas_usb_fw.mem as
dependent firmware). Fix this by only issuing a warning instead.
- Generate a link to the initramfs file with an appropriate file
extension, guessed based on the compressor by default
- Use correct metadata in u-boot images if generated, up to now this
was hardcoded to gzip and would silently generate an erroneous image
if another compressor was specified
- Document all the parameters
- Improve cross-building compatibility, by allowing passing either a
string as before, or a function taking a package set and returning the
path to a compressor in the "compressor" argument of the
function.
- Support more compression algorithms
- Place compressor executable function and arguments in passthru, for
reuse when appending initramfses
Co-Authored-By: Dominik Xaver Hörl <hoe.dom@gmx.de>
Before this commit, the firmware information would be loaded from the
currently running kernel, not from the kernel to be loaded.
This commit ensures the correct kernel version and modules are read.
After a recent upgrade of modinfo, its output is now incorrect for
builtin modules. This commit filters out the output until a fix is made
available upstream
The previous code discarded entire dependency trees if the first entry in the dependency list compiled by `modprobe --show-depends` is a builtin and otherwise handled its output in a rather hackish way.
depmod looks for files modules.order and modules.builtin which are
generated at kernel build time but were previously not passed to
the modules-shrunk derivation
not all valid file names are valid derivation names. This can cause
troubles when, for example, trying to place systemd template unit
files, which contain an '@' in their name, in an initrd.
Fixes#53987
Pull request #38470 added support for running/building kernels without
modules. This got merged in 38e04bbf29 but
unfortunately while this works perfectly on kernels without modules it
also makes sure that *every* kernel gets no modules.
So all of our VM tests fail since that merge with something like this:
machine# loading module loop...
machine# modprobe: FATAL: Module loop not found in directory /lib/modules/4.14.33
machine# loading module vfat...
machine# modprobe: FATAL: Module vfat not found in directory /lib/modules/4.14.33
machine# loading module nls_cp437...
machine# modprobe: FATAL: Module nls_cp437 not found in directory /lib/modules/4.14.33
machine# loading module nls_iso8859-1...
machine# modprobe: FATAL: Module nls_iso8859-1 not found in directory /lib/modules/4.14.33
machine# loading module fuse...
machine# modprobe: FATAL: Module fuse not found in directory /lib/modules/4.14.33
machine# loading module dm_mod...
machine# modprobe: FATAL: Module dm_mod not found in directory /lib/modules/4.14.33
I shortly tested this against the "misc" VM test and the test is working
again.
In the long term (and I currently don't have time for this) it would be
better to also have a VM test which tests a kernel without modules.
Signed-off-by: aszlig <aszlig@nix.build>
Cc: @roberth, @7c6f434c
Resolved the following conflicts (by carefully applying patches from the both
branches since the fork point):
pkgs/development/libraries/epoxy/default.nix
pkgs/development/libraries/gtk+/3.x.nix
pkgs/development/python-modules/asgiref/default.nix
pkgs/development/python-modules/daphne/default.nix
pkgs/os-specific/linux/systemd/default.nix
For a while now, the only thing the 'uboot' attribute does is to tell
whether to add ubootTools to kernel/initrd builds. That can be
determined with platform.kernelTarget == "uImage" just as well.
To achieve reproducible results, `cpio` archive members are added in
sorted order and inodes renumbered.
The `cpio-clean.pl` script is made obsolete by setting mtimes via
`touch` & using `cpio --reproducible`. Suggested by @dezgeg in
https://github.com/NixOS/nixpkgs/pull/21273#issuecomment-268116605.
Note that using `--reproducible` means that initial ramdisk creation now
requires at least `cpio` version 2.12 (released in 2015).
See the comments at f67015cae4
for more information.
Please note: this makes initrd unrepeatable again, but most people will prefer that above an unbootable system.
Conflicts:
pkgs/applications/networking/browsers/chromium/default.nix
pkgs/top-level/all-packages.nix
Merge conflicts seemed trivial, but a look from viric and aszlig would be nice.
