It's expected that attributes in the top-level package set will all use
that package set, but this wasn't the case for the bootstrap tools.
This led some very confusing behaviour:
- pkgsMusl.stdenvBootstrapTools would build glibc bootstrap tools
- stdenvBootstrapTools was _always_ cross compiled, even if
Nixpkgs wasn't, because it always set crossSystem. This also didn't
match the behaviour of using make-bootstrap-tools.nix as an
entrypoint, where crossSystem would default to null.
For the Linux stdenv, I've made the ideal fix, which is to make pkgs an
argument rather than taking the arguments for pkgs, and then
re-importing it. This means it'll always use exactly the same package
set that's calling it, and should also mean faster eval due to not
importing Nixpkgs twice.
The Darwin stdenv is more complicated, and I'm not able to easily test
it, so I wasn't confident in making the same fix there. Instead, I've
just made sure crossSystem and localSystem are set to the correct values
so they're not always cross compiled and match the parent package set's.
It would still be preferable if somebody could make Darwin's
make-bootstrap-tools.nix take pkgs as an argument, rather than all the
arguments for pkgs.
At some point, I'd like to make another attempt at
71f1f4884b ("openssl: stop static binaries referencing libs"), which
was reverted in 195c7da07d. One problem with my previous attempt is
that I moved OpenSSL's libraries to a lib output, but many dependent
packages were hardcoding the out output as the location of the
libraries. This patch fixes every such case I could find in the tree.
It won't have any effect immediately, but will mean these packages
will automatically use an OpenSSL lib output if it is reintroduced in
future.
This patch should cause very few rebuilds, because it shouldn't make
any change at all to most packages I'm touching. The few rebuilds
that are introduced come from when I've changed a package builder not
to use variable names like openssl.out in scripts / substitution
patterns, which would be confusing since they don't hardcode the
output any more.
I started by making the following global replacements:
${pkgs.openssl.out}/lib -> ${lib.getLib pkgs.openssl}/lib
${openssl.out}/lib -> ${lib.getLib openssl}/lib
Then I removed the ".out" suffix when part of the argument to
lib.makeLibraryPath, since that function uses lib.getLib internally.
Then I fixed up cases where openssl was part of the -L flag to the
compiler/linker, since that unambigously is referring to libraries.
Then I manually investigated and fixed the following packages:
- pycurl
- citrix-workspace
- ppp
- wraith
- unbound
- gambit
- acl2
I'm reasonably confindent in my fixes for all of them.
For acl2, since the openssl library paths are manually provided above
anyway, I don't think openssl is required separately as a build input
at all. Removing it doesn't make a difference to the output size, the
file list, or the closure.
I've tested evaluation with the OfBorg meta checks, to protect against
introducing evaluation failures.
`TargetConditionals.h` was missing several definitions, like
`TARGET_OS_TV` that are part of SDK 10.12 at least. And one that doesn't
seem to occur in any SDK afaict, `TARGET_OS_EMBEDDED_OTHER`.
I added the definitions from SDK 10.12 verbatim and defined
`TARGET_OS_EMBEDDED_OTHER` to be equal to `0`.
I think none of this works if `darwin.Libsystem` is used to build for
linux or iOS though so maybe this needs a more thorough fix?
This reverts 336d82617f because it's no
longer necessary.
This enables the bootstrap stdenv test to specify the actual llvm
of the newly generated build instread of assuming it's the same version
as the current stdenv.
Changes to llvmPackages have caused the `libclang-cpp*.dylib` files to
be included in the `clang-unwrapped.lib` output. So we no longer need to
copy them from libclang.
`TargetConditionals.h` was missing several definitions, like
`TARGET_OS_TV` that are part of SDK 10.12 at least. And one that doesn't
seem to occur in any SDK afaict, `TARGET_OS_EMBEDDED_OTHER`.
I added the definitions from SDK 10.12 verbatim and defined
`TARGET_OS_EMBEDDED_OTHER` to be equal to `0`.
This is a modified version of a patch to avoid a stdenv rebuild.
I was having a hard time testing new bootstrapFiles because
`make-bootstrap-tools.nix` imports `pkgspath` but does not pass anything
but the current system.
This is merely for convenience and I'm not entirely certain it's a
sensible thing to do, maybe generating new bootstrapFiles while
overriding the current bootstrapFiles isn't something you're supposed to
do?
The rpath structure for the bootstrap tools was reworked to minimize
the amount of rewriting required on unpack, but the test was not
updated to match the different structure.
Additionally [1] builds that use the bootstrap version of libc++
cannot find libc++abi if the reference includes the "lib"
component (ie, libc++ refers to libc++abi with
@rpath/lib/libc++abi.dylib).
[1] https://logs.nix.samueldr.com/nix-darwin/2021-05-18#4993282
Test failure observed on Hydra: https://hydra.nixos.org/build/143130126
Also begin to start work on cross compilation, though that will have to
be finished later.
The patches are based on the first version of
https://reviews.llvm.org/D99484. It's very annoying to do the
back-porting but the review has uncovered nothing super major so I'm
fine sticking with what I've got.
Beyond making the outputs work, I also strove to re-sync the packages,
as they have been drifting pointlessly apart for some time.
----
Other misc notes, highly incomplete
- lvm-config-native and llvm-config are put in `dev` because they are
tools just for build time.
- Clang no longer has an lld dep. That was introduced in
db29857eb3, but if clang needs help
finding lld when it is used we should just pass it flags / put in the
resource dir. Providing it at build time increases critical path
length for no good reason.
----
A note on `nativeCC`:
`stdenv` takes tools from the previous stage, so:
1. `pkgsBuildBuild`: `(?1, x, x)`
2. `pkgsBuildBuild.stdenv.cc`: `(?0, ?1, x)`
while:
1. `pkgsBuildBuild`: `(?1, x, x)`
2. `pkgsBuildBuild.targetPackages`: `(x, x, ?2)`
3. `pkgsBuildBuild.targetPackages.stdenv.cc`: `(?1, x, x)`
LTO is disabled during bootstrap to keep the bootstrap tools small and
avoid unnecessary LLVM rebuilds, but is enabled in the final stdenv
stage and should be usable by normal packages.
This also updates the bootstrap tool builder to LLVM 5, but not the ones
we actually use for bootstrap. I'll make that change in a subsequent commit
so as to provide traceable provenance of the bootstrap tools.
In the extremely unlikely case that our store hash path ends in several
digits (as is the case right now), the Darwin ld will try to interpret
those digits as a version number and barf. To avoid that, we pass in the
SDK version explicitly to stop it from trying to figure it out from iffy
context.
Introduce new abstraction, `stdenv/booter.nix` for composing bootstraping
stages, and use it everywhere for consistency. See that file for more doc.
Stdenvs besides Linux and Darwin are completely refactored to utilize this.
Those two, due to their size and complexity, are minimally edited for
easier reviewing.
No hashes should be changed.
- The darwin test can now force the use of the freshly-booted darwin stdenv
- The linux test now passes enough dummy arguments
This may make debugging harder, if so, check out #20889
Revert a revert of a merge that shouldn't have been in master but was intentionally in staging.
Next time I'll do this right after the revert instead of so far down the line...
This reverts commit 9adad8612b.
This will break part of the bootstrap tools tests because the new tools
need some changes in the stdenv, but if I change them all at once, the
stdenv breaks with the old bootstrap tools. So I'm doing this first, then
will make changes to the stdenv once this bundle is built and I can use
it.
I also added some functionality to let me test one set of bootstrap tools
on another nixpkgs tree, which makes testing a lot more pleasant.
This reverts commit debd401b0f.
We must not use a single-binary build for the bootstrap since the common
binary gains a dynamic linkage to gmp (due to 'factor' and 'expr'
handling arbitrary-precision arithmetic).
