We go out of our way (see top of file) to build a single binary
with symlinks for all of the tools, but were losing them
when preparing the bootstrap tools.
For the cc of the intermediate stages, to be precise. Doing the same for
bintools requires lots of refactoring.
This is mainly for the future extensibility as now you can change
documentation generation with impunity without rebuilding the
whole of stdenv.
Existing "mips64el" should be "mipsel".
This is just the barest minimum so that nixpkgs can recognize them as
systems - although required for building individual derivations onto
MIPS boards, it is not sufficient if you want to actually build nixos on
those targets
Aarch64 tools tested briefly with qemu-aarch64,
but neither have been actually used yet :).
For now only "host" indirectly via binary cache
at cache.allvm.org.
cc-wrapper may wrap a cc-compiler, but it doesn't need one to build
itself. (c.f. expand-response-params is a separate derivation.) This
helps avoid cycles on the cross stuff, in addition to removing a
useless dependency edge.
I could have been super careful with overrides in the stdenv to avoid
the mass rebuild, but I don't think it's worth it.
This is needed when cross-compiling for iOS (Aarch64 + Darwin). I also
changed the syntax of the Linux stdenv for visual consistency, though
that has no effect on semantics as the os is already guaranteed to be
Linux.
This reverts commit 0a944b345e, reversing
changes made to 61733ed6cc.
I dislike these massive stdenv changes with unclear motivation,
especially when they involve gratuitous mass renames like NIX_CC ->
NIX_BINUTILS. The previous such rename (NIX_GCC -> NIX_CC) caused
months of pain, so let's not do that again.
- Don't build with libsigsegv by default. The build apparently attempted
to link against it, but it never retained the reference anyway...
- Side effect: stdenv bootstrapping needs no libsigsegv anymore.
- Run checks, but only in the interactive gawk by default on Linux,
so that stdenv bootstrap isn't slowed down (by glibc locales, etc.).
- xz should be no longer needed in inputs, as we have it in stdenvs now.
The whole change was triggered by some used kernel versions still
breaking libsigsegv tests #28464.
This reverts commit eeabf85780.
This change suddenly makes tons of stdenv internals visible in
nativeBuildInputs of every derivation, which doesn't seem desirable.
E.g:
````
nix-repl> hello.nativeBuildInputs
[ «derivation /nix/store/bcfkyf6bhssxd2vzwgzmsbn7b5b9rpxc-patchelf-0.9.drv»
«derivation /nix/store/4wnshnz9wwanpfzcrdd76rri7pyqn9sk-paxctl-0.9.drv»
<< snip 10+ lines >>
«derivation /nix/store/d35pgh1lcg5nm0x28d899pxj30b8c9b2-gcc-wrapper-6.4.0.drv»
]
````
Additionally, instead of pulling them from `setup.sh`, route them via
Nix. This gets us one step closer to making stdenv be a plain attribute
set instead of a derivation.
`pkgsNoParams` was removed by me, but then #25035 was merged using it,
leading to an unbound identifier.
It would be nice to get travis to do build release-cross.nix or
something to catch these things.
Use `buildPackages.binutils` to get build = host != target binutils,
i.e. the old `binutilsCross`, and use
`buildPackages.buildPackages.binutils` to get build = host = target
binutils, i.e. the old `binutils`.
`buildPackages` chains like this are supposed to remove the need for
all such `*Cross` derivations. We start with binutils because it's
comparatively easy.
No hashes of cross-tests should be changed
Before all overrides were also pruned in the previous stage, now
only gcc and binutils are, because they alone care about about the
target platform. The rest of the overrides don't, so it's better to
preserve them in order to avoid spurious rebuilds.
Our bootstrap tools are actually broken right now due to busybox not
working when invoked directly from a store path. (It says e.g.
"0qqqw19y4gmknajw8vg4fvhx9gxdqlhz-busybox: applet not found").
Make this test actually fail in such case, the next commit will fix the
problem with busybox.
This is required for Aarch64 since a lot of source tarballs ship with
outdated configure scripts that don't recognize aarch64. Simply
replacing the config.guess and config.sub with new versions from
upstream makes them build again.
This same approach is used by at least Buildroot and Fedora. In
principle this could be enabled for all architectures but
conditionalizing this on aarch64 avoids a mass rebuild on x86.
The long term goal is a big replace:
{ inherit system platform; } => buildPlatform
crossSystem => hostPlatform
stdenv.cross => targetPlatform
And additionally making sure each is defined even when not cross compiling.
This commit refactors the bootstrapping code along that vision, but leaves
the old identifiers with their null semantics in place so packages can be
modernized incrementally.
[N.B., this package also applies to the commits that follow it in the same
PR.]
In most cases, buildPackages = pkgs so things work just as before. For
cross compiling, however, buildPackages is resolved as the previous
bootstrapping stage. This allows us to avoid the mkDerivation hacks cross
compiling currently uses today.
To avoid a massive refactor, callPackage will splice together both package
sets. Again to avoid churn, it uses the old `nativeDrv` vs `crossDrv` to do
so. So now, whether cross compiling or not, packages with get a `nativeDrv`
and `crossDrv`---in the non-cross-compiling case they are simply the same
derivation. This is good because it reduces the divergence between the
cross and non-cross dataflow. See `pkgs/top-level/splice.nix` for a comment
along the lines of the preceding paragraph, and the code that does this
splicing.
Also, `forceNativeDrv` is replaced with `forceNativePackages`. The latter
resolves `pkgs` unless the host platform is different from the build
platform, in which case it resolves to `buildPackages`. Note that the
target platform is not important here---it will not prevent
`forcedNativePackages` from resolving to `pkgs`.
--------
Temporarily, we make preserve some dubious decisions in the name of preserving
hashes:
Most importantly, we don't distinguish between "host" and "target" in the
autoconf sense. This leads to the proliferation of *Cross derivations
currently used. What we ought to is resolve native deps of the cross "build
packages" (build = host != target) package set against the "vanilla
packages" (build = host = target) package set. Instead, "build packages"
uses itself, with (informally) target != build in all cases.
This is wrong because it violates the "sliding window" principle of
bootstrapping stages that shifting the platform triple of one stage to the
left coincides with the next stage's platform triple. Only because we don't
explicitly distinguish between "host" and "target" does it appear that the
"sliding window" principle is preserved--indeed it is over the reductionary
"platform double" of just "build" and "host/target".
Additionally, we build libc, libgcc, etc in the same stage as the compilers
themselves, which is wrong because they are used at runtime, not build
time. Fixing this is somewhat subtle, and the solution and problem will be
better explained in the commit that does fix it.
Commits after this will solve both these issues, at the expense of breaking
cross hashes. Native hashes won't be broken, thankfully.
--------
Did the temporary ugliness pan out? Of the packages that currently build in
`release-cross.nix`, the only ones that have their hash changed are
`*.gcc.crossDrv` and `bootstrapTools.*.coreutilsMinimal`. In both cases I
think it doesn't matter.
1. GCC when doing a `build = host = target = foreign` build (maximally
cross), still defines environment variables like `CPATH`[1] with
packages. This seems assuredly wrong because whether gcc dynamically
links those, or the programs built by gcc dynamically link those---I
have no idea which case is reality---they should be foreign. Therefore,
in all likelihood, I just made the gcc less broken.
2. Coreutils (ab)used the old cross-compiling infrastructure to depend on
a native version of itself. When coreutils was overwritten to be built
with fewer features, the native version it used would also be
overwritten because the binding was tight. Now it uses the much looser
`BuildPackages.coreutils` which is just fine as a richer build dep
doesn't cause any problems and avoids a rebuild.
So, in conclusion I'd say the conservatism payed off. Onward to actually
raking the muck in the next PR!
[1]: https://gcc.gnu.org/onlinedocs/gcc/Environment-Variables.html
This patch add a new argument to Nixpkgs default expression named "overlays".
By default, the value of the argument is either taken from the environment variable `NIXPKGS_OVERLAYS`,
or from the directory `~/.nixpkgs/overlays/`. If the environment variable does not name a valid directory
then this mechanism would fallback on the home directory. If the home directory does not exists it will
fallback on an empty list of overlays.
The overlays directory should contain the list of extra Nixpkgs stages which would be used to extend the
content of Nixpkgs, with additional set of packages. The overlays, i-e directory, files, symbolic links
are used in alphabetical order.
The simplest overlay which extends Nixpkgs with nothing looks like:
```nix
self: super: {
}
```
More refined overlays can use `super` as the basis for building new packages, and `self` as a way to query
the final result of the fix-point.
An example of overlay which extends Nixpkgs with a small set of packages can be found at:
https://github.com/nbp/nixpkgs-mozilla/blob/nixpkgs-overlay/moz-overlay.nix
To use this file, checkout the repository and add a symbolic link to
the `moz-overlay.nix` file in `~/.nixpkgs/overlays` directory.
`gcc-unwrapped` basically replaces `gccPlain`. It may seem like an ugly
polution to stick it in all-packages, but a future PR will enshrine this
`*-unwrapped` pattern. In any event, the long term goal is stdenvs might
need to tweak how compilers are booted and wrapped, but the code to build
the unwrapped compilers themselves should be generic.
Introduce new abstraction, `stdenv/booter.nix` for composing bootstraping
stages, and use it everywhere for consistency. See that file for more doc.
Stdenvs besides Linux and Darwin are completely refactored to utilize this.
Those two, due to their size and complexity, are minimally edited for
easier reviewing.
No hashes should be changed.
On one hand, don't want to pass garbage that affects hash, on the other
hand footguns are bad.
Now, factored out the derivation so only need to pass in what is used.
- The darwin test can now force the use of the freshly-booted darwin stdenv
- The linux test now passes enough dummy arguments
This may make debugging harder, if so, check out #20889
- Non-cross stdenvs are honest and assert that `crossSystem` is null
- `crossSystem` is a mandatory argument to top-level/stage.nix, just like
`system` and `platform`
- Broken default arguments on stdenvs for testing are gone.
- All stdenvs (but little-used stdenvNix) take the same arguments for easy
testing.
This makes the flow of data easier to understand. There's little downside
because the args in question are already inspected by the stdenvs.
cross-compiling in particular is simpler because we don't need to worry
about overriding the config closed over by `allPackages`.
This commit changes the dependencies of stdenv, and clean-up the stdenv
story by removing the `defaultStdenv` attribute as well as the `bootStdenv`
parameter.
Before, the final bootstrapping stage's stdenv was provided by
all-packages, which was iterating multiple times over the
top-level/default.nix expression, and non-final bootstrapping stages'
stdenvs were explicitly specified with the `bootStdenv` parameter.
Now, all stages' stdenvs are specified with the `stdenv` parameter.
For non-final bootstrapping stages, this is a small change---basically just
rename the parameter.
For the final stage, top-level/default.nix takes the chosen stdenv and
makes the final stage with it.
`allPackages` is used to make all bootstrapping stages, final and
non-final alike. It's basically the expression of `stage.nix` (along with a
few partially-applied default arguments)
Note, the make-bootstrap-tools scripts are temporarily broken
I broke this in the cleanups I did in 171c7f0, the gcc inside the bootstrap
tarball is not getting built with the correct --with-fpu, --with-float
etc. options.
- cloog, ppl, cloogppl aren't used by recent GCCs. Kill references to them.
- Use correct versions of isl, as the current GCC depends
on non-default versions of them.
- Also clarify isl dynamic libraries are needed in cross
builds, but not in native builds
- Since aeb3d8c (bzip2: fix cross build on mingw by using autoconf patch),
it seems that the bzip2 binary depends on libbz2 when cross compiling.
So copy libbz2 into the bootstrap tarball as well.
- Curl isn't used in the bootstrap tools since e6f61b4cf3.
Our coreutils now uses single-binary-build mode where, by default,
simple shebang scripts are used for all the binaries. That doesn't work
e.g. with the Linux unpacker which only handles standard binaries and
symlinks. Let's use the symlinked mode instead for boostrapping.
This does NOT change any stdenv hashes.
I only tested the case most important to me:
$ nix-build pkgs/top-level/release.nix -A stdenvBootstrapTools.x86_64-linux.test
For some reason, the current bootstrap tools fail to build gettext:
init2.c:37: MPFR assertion failed: (64 - 0) == ((64 - 0)/8) * 8 && sizeof(mp_limb_t) == ((64 - 0)/8)
libxml/xpath.c: In function 'xmlXPathCompPathExpr':
libxml/xpath.c:10627:1: internal compiler error: Aborted
xmlXPathCompPathExpr(xmlXPathParserContextPtr ctxt) {
^
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.
make[5]: *** [libxml/libxml_rpl_la-xpath.lo] Error 1
I didn't investigate why this is the case but rebuilding the bootstrap
tools seems to help.
I used this old-ish WIP branch https://github.com/dezgeg/nixpkgs/commits/arm-bootstrap
since latest master has even more problems with cross-compiling anything.
(I will eventually push this stuff and make the ARM bootstraps build on hydra.)
I assume there's not much use for it during bootstrapping.
This fixes them as well, as curl was compiled against libnghttp2 but the
lib wasn't copied to the bootstrap tools.
The most complex problems were from dealing with switches reverted in
the meantime (gcc5, gmp6, ncurses6).
It's likely that darwin is (still) broken nontrivially.
Otherwise this fails on ARM:
/nix/store/jipqp9739n7wrjz40igbk85pqk13s0ad-binutils-2.23.1/bin/ld: /nix/store/92pdpqrqkdf8wjciq1cisvsp8kdz8p2i-gmp-5.1.3/lib/libgmp.a(mp_get_fns.o): relocation R_ARM_MOVW_ABS_NC against `__gmp_allocate_func' can not be used when making a shared object; recompile with -fPIC
/nix/store/92pdpqrqkdf8wjciq1cisvsp8kdz8p2i-gmp-5.1.3/lib/libgmp.a: could not read symbols: Bad value
collect2: error: ld returned 1 exit status
make[2]: *** [libisl.la] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory `/tmp/nix-build-isl-0.11.1.drv-3/isl-0.11.1'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/tmp/nix-build-isl-0.11.1.drv-3/isl-0.11.1'
make: *** [all] Error 2
builder for ‘/nix/store/a8ghniifd8d8agqx0cqsh41daa08v11c-isl-0.11.1.drv’ failed with exit code 2
Several places in the tree associate the ARMv7 system
with the beaglebone platform. Change them to point to
armv7l-hf-multiplatform as it supports several boards (including the
beaglebone as well)
- there were many easy merge conflicts
- cc-wrapper needed nontrivial changes
Many other problems might've been created by interaction of the branches,
but stdenv and a few other packages build fine now.
Attrnames and package names should be as close as possible to avoid confusion.
I took care not to confuse the two mpc things during the mass-replace,
so hopefully I suceeded (tarball still builds).
The static curl program is gone, replaced by curl inside of the
bootstrap tools tarball. Also, we generate a .tar.xz archive rather
than .cpio.bz2, making the download smaller. The separate
{sh,cpio,mkdir,ln,bzip2} programs have been replaced by a single
busybox program.
Make thread disabling explicit. This changes the semantics of the perl
derivation, so on other platforms it may require setting
enableThreading = false
This commit doesn't change the derivation or out hash of stdenvLinux.
The stage3.extraAttrs.glibc argument was required for this whole build
procedure to correctly work and it was very-very hard to see why (the
comment said something about gcc47, but we're using gcc48 now).
This stage3.extraAttrs.glibc goes into stage3.stdenv.glibc after some
arg passing, and in pkgs/development/compiler/gcc an (stdenv ? glibc)
boolean expression decides to override /usr/include during the GCC
build.
All of our stages are built with glibc, so this refactoring moves this
repeating specification of glibc (once for gcc wrapper and once here for
extraAttrs) to stageFun, by getting rid of wrapGCC, as we were using
that in all of the stages anyways.
Incidentally it turned out, that this stdenv.glibc inconsistency caused
some random other stuff to behave differently:
- stage1.pkgs.perl has threading disabled,
- stage4.pkgs.coreutils (the production coreutils) has testing disabled.
Leave this historical accidents as they are in this commit, so the scope
of this commit can stay as a refactoring only, these issues will be
fixed in separate commits.
This commit doesn't change the derivation or the output hash of
stdenvLinux.
Use the new allowedRequisites feature in stdenvLinux.
This way we properly check that the end-result stdenv of the quite
complicated multi-stage stdenvLinux building procedure is sane, and only
depends on the stuff that we know about.
Alternative would be to just disallowRequisites bootstrapTools, which is
the most common offender, but we have had other offenders in the past.
For these checks to actually fire, you currently have to use nixUnstable,
as the necessary feature will be released in Nix 1.8.
Previously stdenv depended on two different zlibs and there was a third
one in the top-level package set for other purposes. This commit merges
all this zlibs to one.
Actually this have been committed once as 1f2b636, but then got lost
while resolving merge conflicts. Hopefully it survives this time.
The point here is that it's always possible to debug this staging
logic by using `nix-repl pkgs/stdenv/linux'. The modular-stdenv
change introduced the lib dependency, which we now default to
../../../lib.
No derivation or out hashes of stdenvLinux is changed by this commit.
Previously stdenv depended on two different zlibs and there was a third
one in the top-level package set for other purposes. This commit merges
all this zlibs to one.
Make stages explicit and generalize the pattern of having an stdenv and
a pkgs collection for all stages to a common stage generating function
called stageFun.
Rewrite all stage handling with this new function.
This commit doesn't change the outhash (or drvhash) of the stdenv.
Don't use default parameter values, to make the callsites more readable
and for easier debuggability/changability. Also reordered the
callsites' parameter ordering for consistency.
In the final stdenv don't repeat the name of the shell.
This commit doesn't change the outhash (or drvhash) of the stdenv.
All the different stages of stdenv had the fetchurl inherited anyways,
so make this generic in stdenvBootFun.
This commit doesn't change the outhash (or drvhash) of the stdenv.
Commit 986f361946 started to use
<nix/fetchurl.nix> to "download" the bootstrap binaries from the
Nixpkgs tree, using the file:/// scheme. This has really bad
consequences:
* It makes any derivation depend on the path of the Nixpkgs tree. So
evaluating a package will produce a different .drv file when run
from different locations. No wonder Hydra evaluation has been so
slow lately: for every Nixpkgs evaluation, it had to create tens of
thousands of .drv files, even if nothing had changed.
* It requires the builder to have file system access to the Nixpkgs
tree. So if your tree is in your home directory, the stdenv
bootstrap would probably fail.
So now the binaries are downloaded from tarballs.nixos.org.
Also dropped PowerPC "support".
Binutils nowadays contains ld.gold, which depends on libstdc++. So it
needs to be built with the new GCC rather than the one from
bootstrap-tools.
Issue #1469.
GCC provides a number of libraries that are used by programs built by
GCC, in particular libgcc_s.so and libstdc++.so. This caused programs
that used these libraries to have a runtime dependency on all of GCC
(~77 MiB). Now they only depend on the "lib" output of GCC (~1.6
MiB).
With this and previous multiple-output improvements, closure sizes are
reduced a lot:
hello: 41 MiB -> 22 MiB
patchelf: 118 MiB -> 23 MiB
pan: 364 MiB -> 90 MiB
This reduces the size of closures a lot by getting rid of Glibc's
header files, static libraries and utilities. For instance, the
closure of GNU Hello went from 41 MiB to 22 MiB.
TODO: We may want to merge back in parts of Glibc that we previously
split off (such as the info files).
Since nix-1.4, nix's corepkgs contain a fetchurl suitable for
downloading the bootstrap binaries. Doing this will allow us to have a
nixpkgs with no in-tree binaries without breaking the purity of the
bootstrap (though for now, they are fetched in-tree until the binaries
are added to nixos.org somewhere). As an additional small benefit, the
in-tree binaries do not have to be hashed on every instantiation as they
do now.
The fetchurl in nix-1.2 and 1.3 isn't able to make binaries executable,
so it can't be used for this case. In that case, attempting to build the
bootstrap will show a message asking the user to manually download each
file and add it to the store with nix-store --add (but the hash is
ultimately the same, of course).
Signed-off-by: Shea Levy <shea@shealevy.com>
Due to xz being override in the last stdenv and also in the previous, the
nixpkgs xz ended up being built by bootstrap-tools, and thus depending on it
through libgcc_so.so.1. That ends up making 'nix' with a runtime
dependency on bootstrap-tools.
Although patching it made some programs run (configure tests), some others
crashed with segfault. So I don't think there is any win patching it. The
proper way to solve the bootstrap in the raspberry pi is, as far as I've been
testing, use glibc 2.17 libs in bootstrap-tools with the same ld.so name as the
bootstrapped glibc.
This is a problem inherent in our way to bootstrap, that first replaces
the glibc of a given gcc+glibc (bootstrap-tools) with gcc-wrapper tricks, and
then builds a new gcc. A nicer way would be to build a gcc without glibc,
then the glibc, then the final gcc, as we do with cross-tools.
Some comments about this problem in
https://github.com/NixOS/nixpkgs/issues/234#issuecomment-11764352
This solves the problem of the change of name in ld.so between
glibc 2.13 and 2.17 (at least for armhf). Some comments about it in
https://github.com/NixOS/nixpkgs/issues/234#issuecomment-11764352
As there says, without this, gmp (after glibc built) tests crashed
- segfault.
