Right now our bootstrapFiles-selecting algorithm uses the
`loongson2f.nix` bootstrapFiles (which were not built by Hydra).
These bootstrapFiles don't work anymore. They were added in 2010 by
40405d03ac.
This commit causes mipsel-linux native builds to use the Hydra-built
bootstrap files from this PR instead:
https://github.com/NixOS/nixpkgs/pull/183487
Hydra job building them: https://hydra.nixos.org/build/208909151
The bootstrap files can be reproduced on the commit 21ec906463, e.g. by:
cat $(nix-build pkgs/top-level/release.nix -QA stdenvBootstrapTools.aarch64-linux.dist)/nix-support/hydra-build-products
file tarball /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
file busybox /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/busybox
and the hashes as well:
nix hash file /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
sha256-aJvtsWeuQHbb14BGZ2EiOKzjQn46h3x3duuPEawG0eE=
nix hash path /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/busybox
sha256-0MuIeQlBUaeisqoFSu8y+8oB6K4ZG5Lhq8RcS9JqkFQ=
You can check this on any machine, as the builds are on cache.nixos.org
but also you can reproduce the hashes when rebuilt on aarch64-linux HW.
PR #208478 added a lot of documentation about which packages were
rebuilt in each stage of the stdenv bootstrap. However nothing
checks that these comments agree with reality; they can bitrot over
time. This PR rewrites those comments as assertions, so they cannot
bitrot.
This conversion did expose some ambiguity in our scheme for naming
the stages. Suppose that `pkgs.stdenv.name=="stdenv-stage4", then
which of these is "the stage4 coreutils"?
```
pkgs.coreutils
pkgs.stdenv.__bootPackages.coreutils
```
The choice is arbitrary, and both choices have confusing corner
cases. We should revisit this at some point.
Hydra job building them: https://hydra.nixos.org/build/208909151
The bootstrap files can be reproduced on the parent commit, e.g. by:
cat $(nix-build pkgs/top-level/release.nix -QA stdenvBootstrapTools.aarch64-linux.dist)/nix-support/hydra-build-products
file tarball /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
file busybox /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/busybox
and the hashes as well:
nix hash file /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
sha256-aJvtsWeuQHbb14BGZ2EiOKzjQn46h3x3duuPEawG0eE=
nix hash path /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/busybox
sha256-0MuIeQlBUaeisqoFSu8y+8oB6K4ZG5Lhq8RcS9JqkFQ=
You can check this on any machine, as the builds are on cache.nixos.org
but also you can reproduce the hashes when rebuilt on aarch64-linux HW.
`builtins.baseNameOf` retains any string context, causing the test
derivation to incorrectly depend on `pkgs.glibc`. All we really want is
to know what the dynamicLinker is called, but we don't need it to be
present in store.
Thanks to Adam Joseph for spotting this.
After https://github.com/NixOS/nixpkgs/pull/209054 we started moving
libstdc++.so out of default glibc's paths. This exposed bootstrap
tools build failure as:
$ nix build --no-link -f ./pkgs/stdenv/linux/make-bootstrap-tools.nix
...
>
.../bin/bar: error while loading shared libraries: libstdc++.so.6:
cannot open shared object file: No such file or directory
Note that bootstrap itself did not break. The change only expands
handcrafted `-rpath` entries.
The stdenv bootstrap creates several wrappers around binutils, and
gives them the exact same drvName as the binutils package itself.
These wrappers cost almost nothing to create (they are just file
copies and patchelf runs, not builds), so we should distinguish them
from expensive binutils builds with a unique pname. This commit
does that.
Right now we build gettext several times during the bootstrap.
Gettext's build process is "embarrassingly serial", so avoiding
rebuilding it speeds things up considerably.
This change allows building new gcc during bootstrap without fear of
pulling in outdated libstdc++.so after g++ switched from bootstrapTools
to freshly built g++.
Noticed when tried to add early bootstrap stage to rebuild `gcc` before
`glibc` is fully untangled from `bootstrapTools` as a failure to built
`binutils`:
ld: dwp.o: in function `__gnu_cxx::new_allocator<gold::Dwp_output_file::Contribution>::allocate(unsigned long, void const*)':
/nix/store/...-gcc-11.3.0/include/c++/11.3.0/ext/new_allocator.h:116: undefined reference to `std::__throw_bad_array_new_length()'
The change survives existing bootstrap and unblockes early `gcc` bootstrap.
Just a few comments added:
- added a few one-liners to explore which tools are rebuilt at each
stdenv iteration during bootstrap
- explicitly listed available toolchains and their sources for on each
bootstrap step: glibc, binutils, gcc, coreutils.
- added mention of static libraries linked into gcc
Co-authored-by: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com>
This PR provides the Hydra-generated bootstrap tarballs for mips64el-linux-gnuabin32.
With this PR we now have the bootstrap-file for all three little-endian ABIs on mips: n64, n32, and o32. I do not currently plan to do big-endian mips unless some motivation arises; all mips chips are bi-endian and Debian has dropped big-endian support due to lack of interest.
I'll be following the script used in #151399, #168199, and #183487.
Files came from [this](https://hydra.nixos.org/build/188389586#tabs-summary) Hydra build, which used nixpkgs revision 97d9c84e1d to instantiate:
```
/nix/store/hakn8s85s9011v61r6svp5qy8x1y64fv-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabin32.drv
```
and then built:
```
/nix/store/rjgybpnf3yiqyhvl2n2lx31jf800fii2-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabin32
```
I downloaded these files from Hydra and prefetched them into the nix store with the following commands:
```
STOREPATH=rjgybpnf3yiqyhvl2n2lx31jf800fii2-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabin32
OPTIONS="--option binary-caches https://cache.nixos.org --option trusted-public-keys cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
nix store prefetch-file \
file://$(nix store add-file --name bootstrap-tools.tar.xz $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/bootstrap-tools.tar.xz)
nix store prefetch-file --executable \
file://$(nix store add-path --name busybox $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/busybox)
```
These commands produced the following output:
```
Downloaded 'file:///nix/store/w6zzd2fx2vhmjfcf5h5zc01m0swldpbw-bootstrap-tools.tar.xz' to '/nix/store/6w0f0mqblrghvh6yjwcb4xdqq9x50lbl-w6zzd2fx2vhmjfcf5h5zc01m0swldpbw-bootstrap-tools.tar.xz' (hash 'sha256-LWrpN6su2yNVurUyhZP34OiZyzgh7MfN13fIIbou8KI=').
Downloaded 'file:///nix/store/nqagw1kgdz1zlmqi00qfjrmwqk3g3bgd-busybox' to '/nix/store/i361xhbdhhnvg7zd637xpm63vbl80s0s-nqagw1kgdz1zlmqi00qfjrmwqk3g3bgd-busybox' (hash 'sha256-4N3G1qYA7vitjhsIW17pR6UixIuzrq4vZXa8F0/X4iI=').
```
I used the hashes from the output above to create the `fetchurl` invocation which is part of this commit.
I then started the bootstrap with the following command:
```
nix build -L -f . --arg localSystem '(import ./lib).systems.examples.mips64el-linux-gnuabin32' hello
```
As @lovesegfault requested, here are the the `sha256sum`s of all the `on-server` components for extra verification:
```
sha256sum /nix/store/${STOREPATH}/on-server/*
```
which produced the following output:
```
2d6ae937ab2edb2355bab5328593f7e0e899cb3821ecc7cdd777c821ba2ef0a2 /nix/store/rjgybpnf3yiqyhvl2n2lx31jf800fii2-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabin32/on-server/bootstrap-tools.tar.xz
01633f71135cb9ab1b5ce3ebb67e80cbf288739729bffc1350c1552f6f8df34b /nix/store/rjgybpnf3yiqyhvl2n2lx31jf800fii2-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabin32/on-server/busybox
```
gcc stopped using libelf in commit 48215350c24 ("re PR lto/46273 (Failed
to bootstrap)") around 2010, before gcc-4.6.0.
Bootstrap tools don't use it either.
This PR updates the Hydra-generated bootstrap tarballs for
powerpc64le-linux. The bootstrap-files referenced prior to this
commit will only bootstrap in a nixpkgs which has
https://github.com/NixOS/nixpkgs/pull/181802. That PR was closed in
favor of https://github.com/NixOS/nixpkgs/pull/181943, which is a
cleaner solution but which requires regenerating the bootstrap-files.
I'll be following the script established in
https://github.com/NixOS/nixpkgs/pull/151399, which I previously used
in https://github.com/NixOS/nixpkgs/pull/168199.
Files came from [this](https://hydra.nixos.org/build/186237511) Hydra build, which used nixpkgs revision ac43c44478 to instantiate:
```
/nix/store/nhjbza9vlcyhp9zxfz6lwpc3m2ghrpzj-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu.drv
```
and then built:
```
/nix/store/fklpm7fy6cp5wz55w0gd8wakyqvzapjx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu
```
I downloaded these files from Hydra with the following commands:
```
STOREPATH=fklpm7fy6cp5wz55w0gd8wakyqvzapjx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu
OPTIONS="--option binary-caches https://cache.nixos.org --option trusted-public-keys cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
nix store add-file \
--name bootstrap-tools.tar.xz \
$(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/bootstrap-tools.tar.xz
nix store add-path \
--name busybox \
$(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/busybox
```
I then prefetched them into `/nix/store` with:
```
$ nix store prefetch-file --executable file:///nix/store/p9lz8r81zp3a4sl2qq2v4j69syjzryn2-busybox
Downloaded 'file:///nix/store/p9lz8r81zp3a4sl2qq2v4j69syjzryn2-busybox' to '/nix/store/a42qf2kf5hychcsw5sz0pvghy9vli1im-p9lz8r81zp3a4sl2qq2v4j69syjzryn2-busybox' (hash 'sha256-jtPEAsht4AUAG4MLK8xocQSfveUR4ppU1lS4bGI1VN4=').
$ nix store prefetch-file file:///nix/store/y4530zpk7ia4szf5cdi4zpyy5lpjv3iv-bootstrap-tools.tar.xz
Downloaded 'file:///nix/store/y4530zpk7ia4szf5cdi4zpyy5lpjv3iv-bootstrap-tools.tar.xz' to '/nix/store/kgzyq9q08nll28ccqjcbv8angq5hyvdp-y4530zpk7ia4szf5cdi4zpyy5lpjv3iv-bootstrap-tools.tar.xz' (hash 'sha256-MpIDnpZUK3M17qlnuoxfnK0EgxRosm3TMW1WfPZ1+jU=').
```
And started the bootstrap with the following command:
```
nix build -f . -L hello
```
As @lovesegfault requested in #151399, here are the the `sha256sum`s of all the `on-server` components for extra verification:
```
$ sha256sum /nix/store/fklpm7fy6cp5wz55w0gd8wakyqvzapjx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/*
3292039e96542b7335eea967ba8c5f9cad04831468b26dd3316d567cf675fa35 /nix/store/fklpm7fy6cp5wz55w0gd8wakyqvzapjx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/bootstrap-tools.tar.xz
3d078dff7b4087d82442937667c91dace3321493aae4d3a4160d046b7eabcc2c /nix/store/fklpm7fy6cp5wz55w0gd8wakyqvzapjx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/busybox
```
Before the change an attempt to use `gnumake.override { guileSupport = true; }`
caused recursion in bootstrap stages as guileSupport pulls in guile and it's
dependencies.
To restore the bootstrap the change unconditionally sets
`guileSupport = false;` for `gnumake`.
Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
This PR adds Hydra-generated bootstrap tarballs for mips64el-linux. I'll be following the script established in https://github.com/NixOS/nixpkgs/pull/151399, which I previously used in https://github.com/NixOS/nixpkgs/pull/168199.
Files came from [this](https://hydra.nixos.org/build/182757245) Hydra build, which used nixpkgs revision ef3fe254f3 to instantiate:
```
/nix/store/a2bvv663wjnyhq8m7v84aspsd3sgf9h6-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64.drv
```
and then built:
```
/nix/store/aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64
```
I downloaded these files from Hydra with the following commands:
```
STOREPATH=aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64
OPTIONS="--option binary-caches https://cache.nixos.org"
nix store add-file \
--name bootstrap-tools.tar.xz \
$(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/bootstrap-tools.tar.xz
nix store add-path \
--name busybox \
$(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/busybox
```
I then prefetched them into `/nix/store` with:
```
$ nix store prefetch-file --executable file:///nix/store/aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64/on-server/busybox
Downloaded 'file:///nix/store/aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64/on-server/busybox' to '/nix/store/ai30ss23914syz6j8m95arkwffbbx44k-busybox' (hash 'sha256-sTE58ofjqAqX3Xtq1g9wDxzIe6Vo//GHbicfqJoivDI=').
$ nix store prefetch-file file:///nix/store/aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64/on-server/bootstrap-tools.tar.xz
Downloaded 'file:///nix/store/aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64/on-server/bootstrap-tools.tar.xz' to '/nix/store/nr6zjrbwbxcxs6brf042zsyqllwbwj9v-bootstrap-tools.tar.xz' (hash 'sha256-tTgjeXpd2YgnfP4JvRuO0bXd2j8GqzBcd57JI3wH9x0=').
```
And started the bootstrap with the following command (the `--arg localSystem` is needed because #161159 has not merged):
```
nix build -f . -L hello --arg localSystem '(import ./lib).systems.examples.mips64el-linux-gnuabi64'
```
As @lovesegfault requested in #151399, here are the the `sha256sum`s of all the `on-server` components for extra verification:
```
$ sha256sum /nix/store/aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64/on-server/*
b53823797a5dd988277cfe09bd1b8ed1b5ddda3f06ab305c779ec9237c07f71d /nix/store/aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64/on-server/bootstrap-tools.tar.xz
598e05abb69b2c1a0db46585cd2131212077c0937ce2a665daf3811f059ae767 /nix/store/aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64/on-server/busybox
```
In 5643714dea, I changed pkgs/stdenv/linux/make-bootstrap-tools.nix
to take a package set instead of system and localSystem arguments, but
I forgot to update make-bootstrap-tools-cross.nix.
Fixes: 5643714dea ("stdenvBootstrapTools: inherit {cross,local}System")
