Originally, we switched to bsdtar from libarchive to solve a reproducibility issue related to hardlinks
As of gnu cpio 2.14 the --ignore-dirnlink option is introduced and now included in --reproducible, which solves this issue
By switching back, we are in turn solving an issue in libarchive >=3.7.5 erroring out with "Error reading archive -: (null)"
Change-Id: Ib6140d599b6547d8e941b0251ce996e303c41fa6
The current error message is hard to debug because the error is in the
nix store:
> Error: failed to parse JSON in
"/build/.attr-1s42g1c76fxb77skzq0b4wdhcrg8jmzb54czmxvh1qm7psgsbcni"
>
> Caused by:
> missing field `source` at line 1 column 102
>
> Location:
> src/main.rs:329:10
modules-closure.sh seems to consider everything under
/lib/firmware to be a kernel module,
this change adds a special handling of `edid` directory,
which does not contain kernel modules
fixes#279739
Closes#267442
$ nix path-info -Sh /nix/store/qj1dm7wfw5m3mxf1gn3fdm0az9y1h5ny-linux-firmware-20240312-xz
/nix/store/qj1dm7wfw5m3mxf1gn3fdm0az9y1h5ny-linux-firmware-20240312-xz 440.3M
$ nix path-info -Sh /nix/store/c3szcjxb3g990dbiz7llwmkaf0bi98j2-linux-firmware-20240312-zstd
/nix/store/c3szcjxb3g990dbiz7llwmkaf0bi98j2-linux-firmware-20240312-zstd 460.6M
This is an increase of 4.4%, but OTOH zstd has a significantly higher
decompression speed[1].
[1] https://gregoryszorc.com/blog/2017/03/07/better-compression-with-zstandard/
Closes#302291Closes#301536
The following things have changed:
* For 5.7+: ZSWAP compressor uses zstd now.
* For 5.11+: ZRAM compressor uses zstd now.
* For 5.13+: kernel modules are compressed with zstd instead of xz.
* For 5.19+: support zstd-compressed firmware.
The modules-closure functionality needed explicit support for copying
over `.zst` files. Also, the VM image builder used busybox's `insmod`
before which doesn't support zstd. Switched to `kmod` and added xz/zstd
as dependencies for it, similar to how it's done for the actual stage1
in d33e52b253. The use of `kmod` here
doesn't seem to be such a big deal since it's only a build-time
dependency.
The nixpkgs-unstable channel's programs.sqlite was used to identify
packages producing exactly one binary, and these automatically added
to their package definitions wherever possible.
The previous version didn't work in the case of relative symlinks in
subdirectories. If "foo/bar" was a link to "baz", it would check for
a link to "baz" in the root, rather than under "foo".
We don't need to dereference the symlink ourselves for [ anyway, as it
dereferences its arguments itself, so all we need to do to fix this is
to pass it the link.
Fixes: 14f83d5c6f ("compressFirmwareXz: fix links to directories")
The previous find invocation didn't match the root directory, so the
root directory's access and modification time wasn't set to a
deterministic value and the build time leaked into the output.
`make-initrd` replaced `cpio` with `bsdtar` in #165892 because the
former includes the number of hardlinks in the created archive, which
depends on the filesystem (and can also be influenced by `nix-store
--optimise`). The same problem applies to `make-initrd-ng`, so this
commit replaces `cpio` with `libarchive`'s `bsdtar`.
A handful of kernel modules use glob patterns to express their
firmware dependencies. (`git grep 'MODULE_FIRMWARE.*\*'`)
Previously, we weren't handling these patterns. Now, we are.
It's extremely frustrating seeing "Error: Os { code: 13, kind:
PermissionDenied, message: "Permission denied" }" without any hint as to
where exactly that occurred.
This commit fixes that by adding context to most errors.
Now the tool will only strip binaries if a strip executable is passed
via the STRIP environment variable. This is exposed via the strip
option for makeInitrdNG and the NixOS option boot.initrd.systemd.strip.
This should be a significant disk space saving for most NixOS
installations. This method is a bit more complicated than doing it in
the postInstall for the firmware derivations, but this way it's
automatic, so each firmware package doesn't have to separately
implement its compression.
Currently, only xz compression is supported, but it's likely that
future versions of Linux will additionally support zstd, so I've
written the code in such a way that it would be very easy to implement
zstd compression for those kernels when they arrive, falling back to
xz for older (current) kernels.
I chose the highest possible level of compression (xz -9) because even
at this level, decompression time is negligible. Here's how long it took
to decompress every firmware file my laptop uses:
i915/kbl_dmc_ver1_04.bin 2ms
regulatory.db 4ms
regulatory.db.p7s 3ms
iwlwifi-7265D-29.ucode 62ms
9d71-GOOGLE-EVEMAX-0-tplg.bin 22ms
intel/dsp_fw_kbl.bin 65ms
dsp_lib_dsm_core_spt_release.bin 6ms
intel/ibt-hw-37.8.10-fw-22.50.19.14.f.bseq 7ms
And since booting NixOS is a parallel process, it's unlikely (but
difficult to measure) that the time to user interaction was held up at
all by most of these.
Fixes (partially?) #148197
cpio includes the number of directory hard links in archives it creates.
Some filesystems, like btrfs, do not count directory hard links the same
way as more common filesystems like ext4 or tmpfs, so archives built
when /tmp is on such a filesystem do not reproduce. This patch replaces
cpio with bsdtar, which does not have this issue. The specific
invocation is from this page:
https://reproducible-builds.org/docs/archives/
