See docs.
Follow-up work:
- Existing packages should be converted
- `defaultPkgConfigPackages` should assert on `meta.pkgConfigModules`
and let `tests.pkg-config` alone test the build results.
CC @sternenseemann
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
platform.uname.processor seems to be what we want in many more cases
than what we were using before — it does the right thing for aarch64,
x86_64, riscv32, riscv64, mips, mips64, powerpc, and powerpc64 (the
latter three of which were broken before).
This fixes cross-compilation of systemd for PowerPC/POWER platforms.
Derivations listed as disallowedReferences or disallowedRequisites,
currently end up as build-time dependencies.
This is problematic since the disallowed derivations will be built by nix as
build-time dependencies, while those derivations might take a very long time
to build, or might not even build successfully on the platform used.
However, in order to scan for disallowed references in the final output,
knowing the out path is sufficient, and the out path can be calculated from
the derivation without needing to build it, saving time and resources.
While the problem is less severe for allowedReferences and allowedRequisites,
since we want the derivation to be built eventually, we would still like to
get the error early and without having to wait while nix builds a derivation
that might not be used (e.g. if we listed the wrong one).
A few potentially disruptive changes:
- binutils does not embed ${binutils-unwrapped}/lib as a default library
search path anymore. This will cause link failures for -lbfd -lopcodes
users that did not declare their dependency on those libraries. They
will need to add `libbfd` and `libopcodes` attributes to build inputs.
- `libbfd` and `libopcodes` attributes now just reference
`binutils-unwrapped.{dev,lib}` pair of attributes without patching
`binutils` build system.
We don't patch build system anymore and use multiple outputs out of
existing `binutils` build. That makes the result more maintainable: no
need to handle ever growing list of dependencied of `libbfd`. This time
new addition was `libsframe`.
To accomodate `out` / `lib` output split I had to remove `lib` -> `bin`
backreference by removing legacy lookup path for plugins.
I also did not enable `zstd` just yet as `nixpkgs` version of `zstd`
package pulls in `cmake` into bootstrap sequence.
Changes: https://lists.gnu.org/archive/html/info-gnu/2023-01/msg00003.html
FreeBSD doesn't use LLVM's cxxabi implementation, for backwards
compatibility reasons. Software expects the libcxxrt API when
building on FreeBSD. This fixes the build of
pkgsCross.x86_64-freebsd.boost.
Some other packages, for example ruby gems via buildRubyGem, use a
variable called "type" internally, which is overwritten here and
causes failures like:
failure: $gempkg path unspecified
Fix for changes in 11c3127e38.
this is intentional to support both structuredAttrs and non
In pkgs/stdenv/generic/setup.sh line 614:
for pkg in ${depsBuildBuild[@]} ${depsBuildBuildPropagated[@]}; do
^------------------^ SC2068 (error): Double quote array expansions to avoid re-splitting elements.
In pkgs/stdenv/generic/setup.sh line 521:
local varRef="$varVar[$((targetOffset - hostOffset))]"
^-- SC1087 (error): Use braces when expanding arrays, e.g. ${array[idx]} (or ${var}[.. to quiet).
exit -1 == exit 255 but we don't have a reason to use 255
In pkgs/stdenv/generic/setup.sh line 518:
(( hostOffset <= targetOffset )) || exit -1
^-- SC2242 (error): Can only exit with status 0-255. Other data should be wri
tten to stdout/stderr.
`builtins.baseNameOf` retains any string context, causing the test
derivation to incorrectly depend on `pkgs.glibc`. All we really want is
to know what the dynamicLinker is called, but we don't need it to be
present in store.
Thanks to Adam Joseph for spotting this.
we use [*] to support structuredAttrs and non
In pkgs/stdenv/generic/setup.sh line 1542:
for curPhase in ${phases[*]}; do
^----------^ SC2048 (warning): Use "${array[@]}" (with quotes) to prevent whitespace problems.
In pkgs/stdenv/generic/setup.sh line 101:
source "$hookName"
^---------^ SC1090 (warning): ShellCheck can't follow non-constant source. Use a directive to specify location.
In pkgs/stdenv/generic/setup.sh line 166:
mkdir -p "$out/nix-support"
^--^ SC2154 (warning): out is referenced but not assigned.
In pkgs/stdenv/generic/setup.sh line 407:
PATH=
^--^ SC2123 (warning): PATH is the shell search path. Use another name.
In pkgs/stdenv/generic/setup.sh line 452:
declare -a pkgBuildAccumVars=(pkgsBuildBuild pkgsBuildHost pkgsBuildTarget)
^---------------^ SC2034 (warning): pkgBuildAccumVars appears unused. Verify use (or export if used e
xternally).
because pkgBuildAccumVars is used
In pkgs/stdenv/generic/setup.sh line 235:
nameref="$* ${nameref-}"
^-----^ SC2178 (warning): Variable was used as an array but is now assigned a string.
because we theres a useArray conditional
In pkgs/stdenv/generic/setup.sh line 36:
: ${outputs:=out}
^-------------^ SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
After https://github.com/NixOS/nixpkgs/pull/209054 we started moving
libstdc++.so out of default glibc's paths. This exposed bootstrap
tools build failure as:
$ nix build --no-link -f ./pkgs/stdenv/linux/make-bootstrap-tools.nix
...
>
.../bin/bar: error while loading shared libraries: libstdc++.so.6:
cannot open shared object file: No such file or directory
Note that bootstrap itself did not break. The change only expands
handcrafted `-rpath` entries.
The stdenv bootstrap creates several wrappers around binutils, and
gives them the exact same drvName as the binutils package itself.
These wrappers cost almost nothing to create (they are just file
copies and patchelf runs, not builds), so we should distinguish them
from expensive binutils builds with a unique pname. This commit
does that.
`hasUnsupportedPlatform` was not updated with #37395, so it does not
understand attrsets in `meta.[bad]platforms`. In particular,
attrsets in `meta.badPlatforms` will "fail open" and be ignored.
Let's use `lib.meta.availableOn` instead of duplicating its logic.
Thanks to @alyssais for [noticing][1].
[1][https://github.com/NixOS/nixpkgs/pull/194148#discussion_r990817610]
Co-authored-by: sternenseemann <sternenseemann@systemli.org>
Right now we build gettext several times during the bootstrap.
Gettext's build process is "embarrassingly serial", so avoiding
rebuilding it speeds things up considerably.
otherwise the build just fails with 'make: *** No rule to make target 'install'. Stop.'
and update buildPhase message
i don't know if the 'makefile may have been created in buildPhase' is
true but i guess it might be possible
This change allows building new gcc during bootstrap without fear of
pulling in outdated libstdc++.so after g++ switched from bootstrapTools
to freshly built g++.
Noticed when tried to add early bootstrap stage to rebuild `gcc` before
`glibc` is fully untangled from `bootstrapTools` as a failure to built
`binutils`:
ld: dwp.o: in function `__gnu_cxx::new_allocator<gold::Dwp_output_file::Contribution>::allocate(unsigned long, void const*)':
/nix/store/...-gcc-11.3.0/include/c++/11.3.0/ext/new_allocator.h:116: undefined reference to `std::__throw_bad_array_new_length()'
The change survives existing bootstrap and unblockes early `gcc` bootstrap.
Just a few comments added:
- added a few one-liners to explore which tools are rebuilt at each
stdenv iteration during bootstrap
- explicitly listed available toolchains and their sources for on each
bootstrap step: glibc, binutils, gcc, coreutils.
- added mention of static libraries linked into gcc
Co-authored-by: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com>
The documentation for `meta.sourceProvenance` in
`doc/stdenv/meta.chapter.md` says: "the `meta.sourceProvenance`
attribute should be a list containing one or more value..."
Let's update check-meta.nix to require that `meta.sourceProvenance` is
a list, as the documentation says, rather than a single element.
Adding two extra keystrokes `[` and `]` when filling out this field is
an insignificant burden for package authors, and being able to assume
that the `meta.sourceProvenance` field is always a list greatly
simplifies any code that acts on the value of this field.
Since `meta.sourceProvenance` was just merged a few hours ago now is
the easiest time to fix this: nobody is using the feature yet.
Without the change we don't propagate `enableParallelBuilding = true`
and leave most builds sequential.
Noticed on `mythtv` package which did not specify parallelism and
`config.enableParallelBuildingByDefault = true` had no effect.
Derivations not using `__structuredAttrs` should not attempt to set
environment variables from `env`.
Derivations using `__structuredAttrs` should fail if `env` is not
exportable.
Co-authored-by: Robin Gloster <mail@glob.in>
stdenv: print message if structuredAttrs is enabled
stdenv: add _append
reduces the chance of a user doing it wrong
fix nix develop issue
output hooks don't work yet in nix develop though
making $outputs be the same on non-structuredAttrs and structuredAttrs
is too much trouble.
lets instead make a function that gets the output names
reading environment file '/nix/store/2x7m69a2sm2kh0r6v0q5s9z1dh41m4xf-xz-5.2.5-env-bin'
nix: src/nix/develop.cc:299: std::string Common::makeRcScript(nix::ref<nix::Store>, const BuildEnvironment&, const Path&): Assertion `outputs != buildEnvironment.vars.end()' failed.
use a function to get all output names instead of using $outputs
copy env functionality from https://github.com/NixOS/nixpkgs/pull/76732/commits
In specific cases, combining the `checkMeta` and `checkMetaRecursively`
config options would result in `error: infinite recursion encountered`
fixes#193296
This is needed in order to mark a certain derivation containing a Nix
expression tarball to Hydra so that it is recognised as a channel.
When I first got an evaluation error due to using this meta attribute, I
was under the impression that nobody outside of Vuizvui[1] is using this
feature and that we don't have any occurrence of isHydraChannel in
Nixpkgs.
However, when working around[2] the issue I assumed that it's not
something that should be included in Nixpkgs because we're not using it
there.
It turned out that my assumption was wrong and we *do* use the attribute
in Nixpkgs, namely via releaseTools.channel, which is similar to what
we're doing in Vuizvui.
Since we already include a bunch of undocumented attributes in
metaTypes, it only makes sense to add isHydraChannel as well since it's
actually documented in the Hydra documentation[3].
[1]: https://github.com/openlab-aux/vuizvui
[2]: https://github.com/openlab-aux/vuizvui/commit/e0685e81b3fdc43a272f0
[3]: 53335323ae/doc/manual/src/jobs.md (meta-fields)
Signed-off-by: aszlig <aszlig@nix.build>
Passing `-l$NIX_BUILD_CORES` improperly limits the overall system load.
For a build machine which is configured to run `$B` builds where each
build gets `total cores / B` cores (`$C`), passing `-l $C` to make will
improperly limit the load to `$C` instead of `$B * $C`.
This effect becomes quite pronounced on machines with 80 cores, with
40 simultaneous builds and a cores limit of 2. On a machine with this
configuration, Nix will run 40 builds and make will limit the overall
system load to approximately 2. A build machine with this many cores
can happily run with a load approaching 80.
A non-solution is to oversubscribe the machine, by picking a larger
`$C`. However, there is no way to divide the number of cores in a way
which fairly subdivides the available cores when `$B` is greater than
1.
There has been exploration of passing a jobserver in to the sandbox,
or sharing a jobserver between all the builds. This is one option, but
relatively complicated and only supports make. Lots of other software
uses its own implementation of `-j` and doesn't support either `-l` or
the Make jobserver.
For the case of an interactive user machine, the user should limit
overall system load using `$B`, `$C`, and optionally systemd's
cpu/network/io limiting features.
Making this change should significantly improve the utilization of our
build farm, and improve the throughput of Hydra.
Stdenv on aarch64-darwin pulls in (bootstrap-stage4) objc4, unlike
x86_64. However derivations that otherwise depend on objc4 would use a
a different objc4 derivation on top of the final stdenv.
Because this library defines an LLVM module, having multiple instances
of it in the import path will interfere with builds.
This PR provides the Hydra-generated bootstrap tarballs for mips64el-linux-gnuabin32.
With this PR we now have the bootstrap-file for all three little-endian ABIs on mips: n64, n32, and o32. I do not currently plan to do big-endian mips unless some motivation arises; all mips chips are bi-endian and Debian has dropped big-endian support due to lack of interest.
I'll be following the script used in #151399, #168199, and #183487.
Files came from [this](https://hydra.nixos.org/build/188389586#tabs-summary) Hydra build, which used nixpkgs revision 97d9c84e1d to instantiate:
```
/nix/store/hakn8s85s9011v61r6svp5qy8x1y64fv-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabin32.drv
```
and then built:
```
/nix/store/rjgybpnf3yiqyhvl2n2lx31jf800fii2-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabin32
```
I downloaded these files from Hydra and prefetched them into the nix store with the following commands:
```
STOREPATH=rjgybpnf3yiqyhvl2n2lx31jf800fii2-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabin32
OPTIONS="--option binary-caches https://cache.nixos.org --option trusted-public-keys cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
nix store prefetch-file \
file://$(nix store add-file --name bootstrap-tools.tar.xz $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/bootstrap-tools.tar.xz)
nix store prefetch-file --executable \
file://$(nix store add-path --name busybox $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/busybox)
```
These commands produced the following output:
```
Downloaded 'file:///nix/store/w6zzd2fx2vhmjfcf5h5zc01m0swldpbw-bootstrap-tools.tar.xz' to '/nix/store/6w0f0mqblrghvh6yjwcb4xdqq9x50lbl-w6zzd2fx2vhmjfcf5h5zc01m0swldpbw-bootstrap-tools.tar.xz' (hash 'sha256-LWrpN6su2yNVurUyhZP34OiZyzgh7MfN13fIIbou8KI=').
Downloaded 'file:///nix/store/nqagw1kgdz1zlmqi00qfjrmwqk3g3bgd-busybox' to '/nix/store/i361xhbdhhnvg7zd637xpm63vbl80s0s-nqagw1kgdz1zlmqi00qfjrmwqk3g3bgd-busybox' (hash 'sha256-4N3G1qYA7vitjhsIW17pR6UixIuzrq4vZXa8F0/X4iI=').
```
I used the hashes from the output above to create the `fetchurl` invocation which is part of this commit.
I then started the bootstrap with the following command:
```
nix build -L -f . --arg localSystem '(import ./lib).systems.examples.mips64el-linux-gnuabin32' hello
```
As @lovesegfault requested, here are the the `sha256sum`s of all the `on-server` components for extra verification:
```
sha256sum /nix/store/${STOREPATH}/on-server/*
```
which produced the following output:
```
2d6ae937ab2edb2355bab5328593f7e0e899cb3821ecc7cdd777c821ba2ef0a2 /nix/store/rjgybpnf3yiqyhvl2n2lx31jf800fii2-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabin32/on-server/bootstrap-tools.tar.xz
01633f71135cb9ab1b5ce3ebb67e80cbf288739729bffc1350c1552f6f8df34b /nix/store/rjgybpnf3yiqyhvl2n2lx31jf800fii2-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabin32/on-server/busybox
```
inherit_errexit wasn’t available in bash 3. We have a check to show a
nice error message, but that check is after we set inherit_errexit in
setup.sh. So we can just move this to below the BASH_VERSINFO check.
gcc stopped using libelf in commit 48215350c24 ("re PR lto/46273 (Failed
to bootstrap)") around 2010, before gcc-4.6.0.
Bootstrap tools don't use it either.
This PR updates the Hydra-generated bootstrap tarballs for
powerpc64le-linux. The bootstrap-files referenced prior to this
commit will only bootstrap in a nixpkgs which has
https://github.com/NixOS/nixpkgs/pull/181802. That PR was closed in
favor of https://github.com/NixOS/nixpkgs/pull/181943, which is a
cleaner solution but which requires regenerating the bootstrap-files.
I'll be following the script established in
https://github.com/NixOS/nixpkgs/pull/151399, which I previously used
in https://github.com/NixOS/nixpkgs/pull/168199.
Files came from [this](https://hydra.nixos.org/build/186237511) Hydra build, which used nixpkgs revision ac43c44478 to instantiate:
```
/nix/store/nhjbza9vlcyhp9zxfz6lwpc3m2ghrpzj-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu.drv
```
and then built:
```
/nix/store/fklpm7fy6cp5wz55w0gd8wakyqvzapjx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu
```
I downloaded these files from Hydra with the following commands:
```
STOREPATH=fklpm7fy6cp5wz55w0gd8wakyqvzapjx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu
OPTIONS="--option binary-caches https://cache.nixos.org --option trusted-public-keys cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
nix store add-file \
--name bootstrap-tools.tar.xz \
$(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/bootstrap-tools.tar.xz
nix store add-path \
--name busybox \
$(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/busybox
```
I then prefetched them into `/nix/store` with:
```
$ nix store prefetch-file --executable file:///nix/store/p9lz8r81zp3a4sl2qq2v4j69syjzryn2-busybox
Downloaded 'file:///nix/store/p9lz8r81zp3a4sl2qq2v4j69syjzryn2-busybox' to '/nix/store/a42qf2kf5hychcsw5sz0pvghy9vli1im-p9lz8r81zp3a4sl2qq2v4j69syjzryn2-busybox' (hash 'sha256-jtPEAsht4AUAG4MLK8xocQSfveUR4ppU1lS4bGI1VN4=').
$ nix store prefetch-file file:///nix/store/y4530zpk7ia4szf5cdi4zpyy5lpjv3iv-bootstrap-tools.tar.xz
Downloaded 'file:///nix/store/y4530zpk7ia4szf5cdi4zpyy5lpjv3iv-bootstrap-tools.tar.xz' to '/nix/store/kgzyq9q08nll28ccqjcbv8angq5hyvdp-y4530zpk7ia4szf5cdi4zpyy5lpjv3iv-bootstrap-tools.tar.xz' (hash 'sha256-MpIDnpZUK3M17qlnuoxfnK0EgxRosm3TMW1WfPZ1+jU=').
```
And started the bootstrap with the following command:
```
nix build -f . -L hello
```
As @lovesegfault requested in #151399, here are the the `sha256sum`s of all the `on-server` components for extra verification:
```
$ sha256sum /nix/store/fklpm7fy6cp5wz55w0gd8wakyqvzapjx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/*
3292039e96542b7335eea967ba8c5f9cad04831468b26dd3316d567cf675fa35 /nix/store/fklpm7fy6cp5wz55w0gd8wakyqvzapjx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/bootstrap-tools.tar.xz
3d078dff7b4087d82442937667c91dace3321493aae4d3a4160d046b7eabcc2c /nix/store/fklpm7fy6cp5wz55w0gd8wakyqvzapjx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/busybox
```
This is a change for `powerpc-linux` but that is ancient and I don't
think it matters. The impure bootstrap that was previously assigned to
it has probably bitrotted anyways.
Before the change an attempt to use `gnumake.override { guileSupport = true; }`
caused recursion in bootstrap stages as guileSupport pulls in guile and it's
dependencies.
To restore the bootstrap the change unconditionally sets
`guileSupport = false;` for `gnumake`.
Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
the motivation for this is to simplify stdenv and ease the job of
reviewers due to them needing to tell contributors about the defacto
rule that mesonFlags should be a list of strings
This PR adds Hydra-generated bootstrap tarballs for mips64el-linux. I'll be following the script established in https://github.com/NixOS/nixpkgs/pull/151399, which I previously used in https://github.com/NixOS/nixpkgs/pull/168199.
Files came from [this](https://hydra.nixos.org/build/182757245) Hydra build, which used nixpkgs revision ef3fe254f3 to instantiate:
```
/nix/store/a2bvv663wjnyhq8m7v84aspsd3sgf9h6-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64.drv
```
and then built:
```
/nix/store/aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64
```
I downloaded these files from Hydra with the following commands:
```
STOREPATH=aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64
OPTIONS="--option binary-caches https://cache.nixos.org"
nix store add-file \
--name bootstrap-tools.tar.xz \
$(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/bootstrap-tools.tar.xz
nix store add-path \
--name busybox \
$(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/busybox
```
I then prefetched them into `/nix/store` with:
```
$ nix store prefetch-file --executable file:///nix/store/aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64/on-server/busybox
Downloaded 'file:///nix/store/aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64/on-server/busybox' to '/nix/store/ai30ss23914syz6j8m95arkwffbbx44k-busybox' (hash 'sha256-sTE58ofjqAqX3Xtq1g9wDxzIe6Vo//GHbicfqJoivDI=').
$ nix store prefetch-file file:///nix/store/aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64/on-server/bootstrap-tools.tar.xz
Downloaded 'file:///nix/store/aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64/on-server/bootstrap-tools.tar.xz' to '/nix/store/nr6zjrbwbxcxs6brf042zsyqllwbwj9v-bootstrap-tools.tar.xz' (hash 'sha256-tTgjeXpd2YgnfP4JvRuO0bXd2j8GqzBcd57JI3wH9x0=').
```
And started the bootstrap with the following command (the `--arg localSystem` is needed because #161159 has not merged):
```
nix build -f . -L hello --arg localSystem '(import ./lib).systems.examples.mips64el-linux-gnuabi64'
```
As @lovesegfault requested in #151399, here are the the `sha256sum`s of all the `on-server` components for extra verification:
```
$ sha256sum /nix/store/aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64/on-server/*
b53823797a5dd988277cfe09bd1b8ed1b5ddda3f06ab305c779ec9237c07f71d /nix/store/aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64/on-server/bootstrap-tools.tar.xz
598e05abb69b2c1a0db46585cd2131212077c0937ce2a665daf3811f059ae767 /nix/store/aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64/on-server/busybox
```
the motivation for this is to simplify stdenv and ease the job of
reviewers due to them needing to tell contributors about the defacto
rule that cmakeFlags should be a list of strings
Closes#178625
The `busybox` version of `mktemp` requires exactly six `X` characters
in the argument to `mktemp`, unlike the `coreutils` version of `mktemp`.
Let's accomodate packages, like `epson-escpr2`, which fool `setup.sh`
into using the `busybox` version instead of the `stdenv` version.
In 5643714dea, I changed pkgs/stdenv/linux/make-bootstrap-tools.nix
to take a package set instead of system and localSystem arguments, but
I forgot to update make-bootstrap-tools-cross.nix.
Fixes: 5643714dea ("stdenvBootstrapTools: inherit {cross,local}System")
Since 97c43828fb the `file` package has
been part of stdenv, and no longer needs to be listed explicitly as a
build input. Let's remove the platform-specific inclusion for mingw64
as suggested by @mehmooda:
https://github.com/NixOS/nixpkgs/pull/168413#issuecomment-1147370500
I traced the line removed by this commit through the `git blame`; it
was initially added in this commit (and then shuffled around a few
dozen times by refactorings):
8b292a1b35
The commit message indicates that `libpng-1.6.20` was current at the
time. Although there are [libpng
archives](https://github.com/glennrp/libpng) available in git form,
the older versions don't have their autoconfery vendored in, so I
can't link to them. Fortunately the relevant bit hasn't changed since
then:
a37d483651/configure (L5575)
```
mingw* | pw32*)
# Base MSYS/MinGW do not provide the 'file' command needed by
# func_win32_libid shell function, so use a weaker test based on 'objdump',
# unless we find 'file', for example because we are cross-compiling.
if ( file / ) >/dev/null 2>&1; then
lt_cv_deplibs_check_method='file_magic ^x86 archive import|^x86 DLL'
lt_cv_file_magic_cmd='func_win32_libid'
else
# Keep this pattern in sync with the one in func_win32_libid.
lt_cv_deplibs_check_method='file_magic file format (pei*-i386(.*architecture: i386)?|pe-arm-wince|pe-x86-64)'
lt_cv_file_magic_cmd='$OBJDUMP -f'
fi
;;
```
During stdenv bootstrapping, coreutils is built twice. This makes
troubleshooting very difficult, because both packages have
name="coreutils", so it is a hassle to figure out "which coreutils am
I using / is not building"?
The first of these builds is used only in stage4, and is not part of
the final stdenv. Let's label that one with a different `name`
attribute to make it obvious which is which.
The usage of `makeStaticLibraries` in stdenv/linux/default.nix is
prefaced by this comment:
# Link GCC statically against GMP etc. This makes sense because
# these builds of the libraries are only used by GCC, so it
# reduces the size of the stdenv closure.
However "these builds of the libraries are only used by GCC" is not
actually true. As currently written, the stage4 coreutils links
against these customized, static-ified libraries.
Beside the fact that the code doesn't actually do what it says, this
causes other problems as well. One example is #168983, which arises
because have a dynamically-linked binary (coreutils) which is built
from statically-linked libraries (libgmp.a); doing this causes mayhem
on platforms where `-fstack-protector` needs an auxiliary
`libssp.{so,a}` library; we end up with link failures because some
parts of the resulting binary want `libssp.so` and other parts want
`libssp_nonshared.a`.
Let's make the code actually do what the comment says, by moving these
definitions into the `gcc-unwrapped` override. This will cause the
stage4-coreutils to link against libgmp dynamically, rather than
statically. For this reason this commit depends on the previous
commit, which allows that to be done without creating a forbidden
reference from stdenv-final to the bootstrap-files.
As explained in the comment, this ensures that stage4-coreutils does
not leak a reference to the bootstrap-files by way of libgmp. This
will allow the next patch in this series to build stage4-coreutils
using a dynamically-linked (rather than statically-linked) libgmp.
It's expected that attributes in the top-level package set will all use
that package set, but this wasn't the case for the bootstrap tools.
This led some very confusing behaviour:
- pkgsMusl.stdenvBootstrapTools would build glibc bootstrap tools
- stdenvBootstrapTools was _always_ cross compiled, even if
Nixpkgs wasn't, because it always set crossSystem. This also didn't
match the behaviour of using make-bootstrap-tools.nix as an
entrypoint, where crossSystem would default to null.
For the Linux stdenv, I've made the ideal fix, which is to make pkgs an
argument rather than taking the arguments for pkgs, and then
re-importing it. This means it'll always use exactly the same package
set that's calling it, and should also mean faster eval due to not
importing Nixpkgs twice.
The Darwin stdenv is more complicated, and I'm not able to easily test
it, so I wasn't confident in making the same fix there. Instead, I've
just made sure crossSystem and localSystem are set to the correct values
so they're not always cross compiled and match the parent package set's.
It would still be preferable if somebody could make Darwin's
make-bootstrap-tools.nix take pkgs as an argument, rather than all the
arguments for pkgs.
it may be what the license handling code does, but it's confusing and not very useful
Co-authored-by: Adam Joseph <54836058+a-m-joseph@users.noreply.github.com>
These servers apparently no longer exist, since September 2, 2021[1].
If somebody needs this for non-Scaleway machines, they should suggest
its reintroduction with a different name.
[1]: https://news.ycombinator.com/item?id=27192757
libtool's libtool.m4 script assumes that `file` is available, and can
be found at `/usr/bin/file` (this path is hardwired). Furthermore,
the script with this assumption is vendored into the ./configure
scripts of an enormous number of packages. Without this commit, you
will frequently see errors like this during the configurePhase with
the sandbox enabled:
./configure: line 9595: /usr/bin/file: command not found
Due mostly to luck, this error does not affect native compiles on
nixpkgs' two most popular platforms, x86_64-linux and aarch64-linux.
However it will cause incorrect linker flag detection and a failure to
generate shared libraries for sandboxed cross-builds to a x86_64-linux
host as well as any sandboxed build (cross or native) for the following
hosts: x86_64-freebsd, *-hpux, *-irix, mips64*-linux, powerpc*-linux,
s390x-linux, s390x-tpf, sparc-linux, and *-solaris.
This commit fixes the problem by adding an extra line to fixLibtool()
in pkgs/stdenv/generic/setup.sh. This extra line will scan the
unpacked source code for executable files named "configure" which
contain the following text:
'GNU Libtool is free software; you can redistribute it and/or modify'
This text is taken to be an indicator of a vendored libtool.m4. When
it is found, the configure script containing it is subjected to `sed
-i s_/usr/bin/file_file_` which replaces all occurrences of
`/usr/bin/file` with `file`.
Additionally, the `file` package is now considered to be part of
`stdenv`. It has been added to `common-path.nix` so that the `file`
binary will be found in the `$PATH` of every build, except for the
bootstrap-tools and the first few stages of stdenv boostrapping.
Verified no regressions under:
nix-build --arg pkgs 'import ./. {}' ./lib/tests/release.nix
This commit allows the following commands to complete, which should
enable Hydra to produce bootstrap-files for mips64el:
nix-build \
--option sandbox true \
--option sandbox-fallback false \
pkgs/top-level/release-cross.nix \
-A bootstrapTools.mips64el-linux-gnuabi64.build
nix-build \
--option sandbox true \
--option sandbox-fallback false \
. \
-A pkgsCross.mips64el-linux-gnuabi64.nix_2_4
Instead of requiring the platforms be equal, use `isCompatible` to
determine if we can execute tests. The upside of this is that we now
can execute tests for natively cross compiled package sets like
pkgsStatic, pkgsLLVM and pkgsCross.musl64 etc.
canExecute is like isCompatible, but also checks that the Kernels are
_equal_, i.e. that both platforms use the same syscall interface. This
is crucial in order to actually be able to execute binaries for the
other platform.
isCompatible is dropped, since it has changed semantically and there's
no use case left in nixpkgs.
the motivation for this is to simplify stdenv and ease the job of
reviewers due to them needing to tell contributors about the defacto
rule that configureFlags should be a list of strings
stdenv: deprecate addCoverageInstrumentation adapter
this used to be used in nixos/tests but it hasn't been used in nixpkgs
for years
stdenv: deprecate replaceMaintainersField adapter
it was added in 2009 in 01e98e49b1
by nbp
there are no uses of it in nixpkgs now
stdenv: deprecate validateLicenses adapter
it was added in 2009 in b29073af25
unfreePredicate is now handled in ./pkgs/stdenv/generic/check-meta.nix
so this is unnecessary
This warning logs when a package has no maintainers. It will stay silent
if `meta.maintainers` is not set at all, only complaining when it is an
empty list. In the future a separate warning could be added to allow for
that stricter behavior. Or this warning could be changed.
This will allow for adding more validity types in the future, such as a
warning type. (which is in the next commit in this series)
This is NOT a breaking change because validity.valid is never exposed
outside of `stdenv.mkDerivation`.
This PR adds Hydra-generated bootstrap tarballs for powerpc64le-linux.
I'll be following the script established in PR to tarballs.nixos.org.
Files came from this Hydra build:
https://hydra.nixos.org/build/172142499
Which used nixpkgs revision 49a83445c2
to instantiate:
/nix/store/gj272sd56gsj6qpyzh4njpfzwdhviliz-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu.drv
and then built:
/nix/store/n81pljbd8m0xgypm84krc2bnvqgjrfxx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu
I downloaded these files from Hydra with the following commands:
```
STOREPATH=n81pljbd8m0xgypm84krc2bnvqgjrfxx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu
OPTIONS="--option binary-caches https://cache.nixos.org"
nix store add-file \
--name bootstrap-tools.tar.xz \
$(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/bootstrap-tools.tar.xz
nix store add-path \
--name busybox \
$(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/busybox
```
As @lovesegfault requested in #151399, here are the the `sha256sum`s
of all the `on-server` components for extra verification:
```
$ sha256sum /nix/store/n81pljbd8m0xgypm84krc2bnvqgjrfxx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/*
036d062869f7accf0ad89714d12029469dfe6af504f9b226d61eb7d808ad4735 /nix/store/n81pljbd8m0xgypm84krc2bnvqgjrfxx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/bootstrap-tools.tar.xz
6bc05832180f5075f4458c32eb0a5e2b673f605884dce01822be474f4e0a63ee /nix/store/n81pljbd8m0xgypm84krc2bnvqgjrfxx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/busybox
```
Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
This isn't really desirable in general, but given that Nix itself
currently relies on this behaviour and that we don't want to break
backwards compatibility we should support it for now, maybe
deprecating it in the future.
This reverts commit c1ef87e8d0.
I didn't realize that it would rebuild all linux stdenvs;
I certainly don't want to incur such a rebuild at this point.
At some point, I'd like to make another attempt at
71f1f4884b ("openssl: stop static binaries referencing libs"), which
was reverted in 195c7da07d. One problem with my previous attempt is
that I moved OpenSSL's libraries to a lib output, but many dependent
packages were hardcoding the out output as the location of the
libraries. This patch fixes every such case I could find in the tree.
It won't have any effect immediately, but will mean these packages
will automatically use an OpenSSL lib output if it is reintroduced in
future.
This patch should cause very few rebuilds, because it shouldn't make
any change at all to most packages I'm touching. The few rebuilds
that are introduced come from when I've changed a package builder not
to use variable names like openssl.out in scripts / substitution
patterns, which would be confusing since they don't hardcode the
output any more.
I started by making the following global replacements:
${pkgs.openssl.out}/lib -> ${lib.getLib pkgs.openssl}/lib
${openssl.out}/lib -> ${lib.getLib openssl}/lib
Then I removed the ".out" suffix when part of the argument to
lib.makeLibraryPath, since that function uses lib.getLib internally.
Then I fixed up cases where openssl was part of the -L flag to the
compiler/linker, since that unambigously is referring to libraries.
Then I manually investigated and fixed the following packages:
- pycurl
- citrix-workspace
- ppp
- wraith
- unbound
- gambit
- acl2
I'm reasonably confindent in my fixes for all of them.
For acl2, since the openssl library paths are manually provided above
anyway, I don't think openssl is required separately as a build input
at all. Removing it doesn't make a difference to the output size, the
file list, or the closure.
I've tested evaluation with the OfBorg meta checks, to protect against
introducing evaluation failures.
When implementing this reviewer request:
https://github.com/NixOS/nixpkgs/pull/161158#discussion_r822256070
to move mips64el-unknown-linux-* from platforms.nix to examples.nix, I
neglected to update the reference in make-bootstrap-tools-cross.nix.
As a result, the hydra jobs to generate a bootstrap tarball for
mips64el are not running:
https://hydra.nixos.org/eval/1750828?filter=mips&compare=1750620&full=#tabs-errors
This commit fixes the problem, so the hydra job can run. Once it
does, I will submit a PR adding the trusted bootstrap tarball hash to
pkgs/stdenv/linux/bootstrap-files/.
Co-authored-by: sterni <sternenseemann@systemli.org>
