The old stdenv didn't work, and was also impure. The new one works, and
is pure. Presently, the bootstrap tools are cross compiled into one small
nar and one large tar, which is then unpacked, patched, and split into
smaller derivations. Efforts were made to make the boot process as short
as possible - there are only two clangs built, and as many packages are
propagated between stages as possible while leaving the bootstrap tools
out of the final stdenv's closure.
After https://github.com/NixOS/nixpkgs/pull/322388, darwin.cctools is just Apple’s cctools again. The replacement for what the bootstrap tools wants is darwin.binutils-unwrapped.
GNU binutils is not preferred on Darwin, and newer versions have issues building. Make it an evaluation error to use it in the Darwin stdenv bootstrap.
- Only link binaries that exist for stage 0 cctools and LLVM bintools;
- Drop cctools-llvm in favor of the updated darwin.binutils; and
- Update llvm-manages Python overrides (needed for newer versions of LLVM).
I realized what rhelmot did in 61202561d9
(specify what packages just need `stdenvNoLibc`) is definitely the right
approach for this, and adjusted NetBSD and OpenBSD to likewise use it.
With that change, we don't need these confusing and ugly `*bsdCross`
package sets at all!
We can get rid of a lot more libc-related `*Cross`, and I will do so
soon, but this is the first step.
(adapted from commit 51f1ecaa59)
For a long time, we've had `crossLibcStdenv`, `*Cross` libc attributes,
and `*bsdCross` pre-libc package sets. This was always bad because
having "cross" things is "not declarative": the naming doesn't reflect
what packages *need* but rather how we *provide* something. This is
ugly, and creates needless friction between cross and native building.
Now, almost all of these `*Cross` attributes are gone: just these are
kept:
- Glibc's and Musl's are kept, because those packages are widely used
and I didn't want to risk changing the native builds of those at this
time.
- generic `libcCross`, `theadsCross`, and friends, because these relate
to the convolulted GCC bootstrap which still needs to be redone.
The BSD and obscure Linux or freestnanding libcs have conversely all
been made to use a new `stdenvNoLibc`, which is like the old
`crossLibcStdenv` except:
1. It usable for native and cross alike
2. It named according to what it *is* ("a standard environment without
libc but with a C compiler"), rather than some non-compositional
jargon ("the stdenv used for building libc when cross compiling",
yuck).
I should have done this change long ago, but I was stymied because of
"infinite recursions". The problem was that in too many cases we are
overriding `stdenv` to *remove* things we don't need, and this risks
cyles since those more minimal stdenvs are used to build things in the
more maximal stdenvs.
The solution is to pass `stage.nix` `stdenvNoCC`, so we can override to
*build up* rather than *tear down*. For now, the full `stdenv` is also
passed, so I don't need to change the native bootstraps, but I can see
this changing as we make things more uniform and clean those up.
Finally, the BSDs also had to be cleaned up, since they have a few
pre-libc dependencies, demanding a systematic approach. I realized what
rhelmot did in 61202561d9 (specify what
packages just need `stdenvNoLibc`) is definitely the right approach for
this, and adjusted NetBSD and OpenBSD to likewise use it.
Most Linux distributions are enabling this these days and it does
protect against real world vulnerabilities as demonstrated by
CVE-2018-16864 and CVE-2018-16865.
Fix#53753.
Information on llvm version support gleaned from
6609892a2d68e07da3e5092507a730
Information on gcc version support a lot harder to gather,
but both 32bit and 64bit arm do appear to be supported
based on the test suite.
A second take at eb28e5e72e, which was reverted for the extra logging
during the internals of `nix-shell -p`. This commit does the same
logging, but to $NIX_LOG_FD instead, which is echoed during any normal
build, but not during the internals of `nix-shell -p`.
[1]: eb28e5e72e
libiconv-darwin depends on Meson, which (indirectly) depends on
libiconv. When libiconv-darwin is set as libiconv, it will cause an
infinite recursion. Avoid the infinite recursion by using libiconvReal
in stage 1. Every stage after that can use libiconv-darwin.
The cc and bintools wrapper contained ad hoc bootstrapping logic for
expand-response-params (which was callPackage-ed in a let binding). This
lead to the strange situation that the bootstrapping logic related to
expand-response-params is split between the wrapper derivations (where
it is duplicated) and the actual stdenv bootstrapping.
To clean this up, the wrappers simply should take expand-response-params
as an ordinary input: They need an adjacent expand-response-params (i.e.
one that runs on their host platform), but don't care about the how.
Providing this is only problematic during stdenv bootstrapping where we
have to pull it from the previous stage at times.
We don't need to artificially make sure that we can execute the wrapper
scripts on the build platform by using stdenv's shell (which comes from
buildPackages) since our cross infrastructure will get us the wrapper
from buildPackages. The upside of this change is that cross-compiled
wrappers (e.g. pkgsCross.aarch64-multiplatform.gcc) will actually work
when executed!
For bootstrapping this is also not a problem, since we have a long
build->build platform chain so runtimeShell is just as good as
stdenvNoCC.shell. We do fall back to old ways, though, by explicitly
using the bootstrap-tools shell in stage2, so the adjacent bash is only
used from stage4 onwards. This is unnecessary in principle (I'll try
removing this hack in the future), but ensures this change causes zero
rebuilds.
As of late, `final: prev: …` for overlays has become more prevalent in
newer code. This is also exhibited in some code (presumably added
recently) in stdenv. This change is not about any merits or demerits of
any naming convention, but rather aims to make the nomenclature in
stdenv bootstrapping consistent to lessen confusion.
I've chosen to stick to `self: super: …` convention because:
1. It is more common in the code as it stands.
2. Using `final: prev: …` makes the code more confusing, als it causes
`prev` to be in scope alongside `prevStage`. `prevStage` actually
bears no relation to `prev` even though their naming suggests it,
making it easy to confuse them (the former is the (final) package set
of the previous stage while the latter is just the `prev`/`super` of
the overlay “chaining” on a completely fresh package set, i.e. `prev`
doesn't even relate to the previous stage's `overrides` argument).
This change also corrects a naming error in stdenv/native which had no
effect, as the variables were unused.
This fixes using inputDerivation on derivations that are fixed-output.
Previously:
```
nix-repl> drv = runCommand "huh" { outputHash = "sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU="; outputHashAlgo = "sha256"; outputHashType = "flat"; } "touch $out"
nix-repl> drv.inputDerivation
«derivation /nix/store/d8mjs6cmmvsr1fv7psm6imis5pmh9bcs-huh.drv»
nix-repl> :b drv.inputDerivation
error: fixed output derivation 'huh' is not allowed to refer to other store paths.
You may need to use the 'unsafeDiscardReferences' derivation attribute, see the manual for more details.
```
Fixes: https://github.com/NixOS/nixpkgs/issues/304209
