Closes#302291Closes#301536
The following things have changed:
* For 5.7+: ZSWAP compressor uses zstd now.
* For 5.11+: ZRAM compressor uses zstd now.
* For 5.13+: kernel modules are compressed with zstd instead of xz.
* For 5.19+: support zstd-compressed firmware.
The modules-closure functionality needed explicit support for copying
over `.zst` files. Also, the VM image builder used busybox's `insmod`
before which doesn't support zstd. Switched to `kmod` and added xz/zstd
as dependencies for it, similar to how it's done for the actual stage1
in d33e52b253. The use of `kmod` here
doesn't seem to be such a big deal since it's only a build-time
dependency.
The nixpkgs-unstable channel's programs.sqlite was used to identify
packages producing exactly one binary, and these automatically added
to their package definitions wherever possible.
The previous version didn't work in the case of relative symlinks in
subdirectories. If "foo/bar" was a link to "baz", it would check for
a link to "baz" in the root, rather than under "foo".
We don't need to dereference the symlink ourselves for [ anyway, as it
dereferences its arguments itself, so all we need to do to fix this is
to pass it the link.
Fixes: 14f83d5c6f ("compressFirmwareXz: fix links to directories")
The previous find invocation didn't match the root directory, so the
root directory's access and modification time wasn't set to a
deterministic value and the build time leaked into the output.
`make-initrd` replaced `cpio` with `bsdtar` in #165892 because the
former includes the number of hardlinks in the created archive, which
depends on the filesystem (and can also be influenced by `nix-store
--optimise`). The same problem applies to `make-initrd-ng`, so this
commit replaces `cpio` with `libarchive`'s `bsdtar`.
A handful of kernel modules use glob patterns to express their
firmware dependencies. (`git grep 'MODULE_FIRMWARE.*\*'`)
Previously, we weren't handling these patterns. Now, we are.
It's extremely frustrating seeing "Error: Os { code: 13, kind:
PermissionDenied, message: "Permission denied" }" without any hint as to
where exactly that occurred.
This commit fixes that by adding context to most errors.
Now the tool will only strip binaries if a strip executable is passed
via the STRIP environment variable. This is exposed via the strip
option for makeInitrdNG and the NixOS option boot.initrd.systemd.strip.
This should be a significant disk space saving for most NixOS
installations. This method is a bit more complicated than doing it in
the postInstall for the firmware derivations, but this way it's
automatic, so each firmware package doesn't have to separately
implement its compression.
Currently, only xz compression is supported, but it's likely that
future versions of Linux will additionally support zstd, so I've
written the code in such a way that it would be very easy to implement
zstd compression for those kernels when they arrive, falling back to
xz for older (current) kernels.
I chose the highest possible level of compression (xz -9) because even
at this level, decompression time is negligible. Here's how long it took
to decompress every firmware file my laptop uses:
i915/kbl_dmc_ver1_04.bin 2ms
regulatory.db 4ms
regulatory.db.p7s 3ms
iwlwifi-7265D-29.ucode 62ms
9d71-GOOGLE-EVEMAX-0-tplg.bin 22ms
intel/dsp_fw_kbl.bin 65ms
dsp_lib_dsm_core_spt_release.bin 6ms
intel/ibt-hw-37.8.10-fw-22.50.19.14.f.bseq 7ms
And since booting NixOS is a parallel process, it's unlikely (but
difficult to measure) that the time to user interaction was held up at
all by most of these.
Fixes (partially?) #148197
cpio includes the number of directory hard links in archives it creates.
Some filesystems, like btrfs, do not count directory hard links the same
way as more common filesystems like ext4 or tmpfs, so archives built
when /tmp is on such a filesystem do not reproduce. This patch replaces
cpio with bsdtar, which does not have this issue. The specific
invocation is from this page:
https://reproducible-builds.org/docs/archives/
The `platform` field is pointless nesting: it's just stuff that happens
to be defined together, and that should be an implementation detail.
This instead makes `linux-kernel` and `gcc` top level fields in platform
configs. They join `rustc` there [all are optional], which was put there
and not in `platform` in anticipation of a change like this.
`linux-kernel.arch` in particular also becomes `linuxArch`, to match the
other `*Arch`es.
The next step after is this to combine the *specific* machines from
`lib.systems.platforms` with `lib.systems.examples`, keeping just the
"multiplatform" ones for defaulting.
Since fdf32154fc, we no longer allow
missing modules in the initrd. Unfortunately since before this commit,
the modules-closure script would also fail on missing firmware, which
is a very common case (e.g. xhci-pci.ko.xz lists renesas_usb_fw.mem as
dependent firmware). Fix this by only issuing a warning instead.
- Generate a link to the initramfs file with an appropriate file
extension, guessed based on the compressor by default
- Use correct metadata in u-boot images if generated, up to now this
was hardcoded to gzip and would silently generate an erroneous image
if another compressor was specified
- Document all the parameters
- Improve cross-building compatibility, by allowing passing either a
string as before, or a function taking a package set and returning the
path to a compressor in the "compressor" argument of the
function.
- Support more compression algorithms
- Place compressor executable function and arguments in passthru, for
reuse when appending initramfses
Co-Authored-By: Dominik Xaver Hörl <hoe.dom@gmx.de>
