Passing `-l$NIX_BUILD_CORES` improperly limits the overall system load.
For a build machine which is configured to run `$B` builds where each
build gets `total cores / B` cores (`$C`), passing `-l $C` to make will
improperly limit the load to `$C` instead of `$B * $C`.
This effect becomes quite pronounced on machines with 80 cores, with
40 simultaneous builds and a cores limit of 2. On a machine with this
configuration, Nix will run 40 builds and make will limit the overall
system load to approximately 2. A build machine with this many cores
can happily run with a load approaching 80.
A non-solution is to oversubscribe the machine, by picking a larger
`$C`. However, there is no way to divide the number of cores in a way
which fairly subdivides the available cores when `$B` is greater than
1.
There has been exploration of passing a jobserver in to the sandbox,
or sharing a jobserver between all the builds. This is one option, but
relatively complicated and only supports make. Lots of other software
uses its own implementation of `-j` and doesn't support either `-l` or
the Make jobserver.
For the case of an interactive user machine, the user should limit
overall system load using `$B`, `$C`, and optionally systemd's
cpu/network/io limiting features.
Making this change should significantly improve the utilization of our
build farm, and improve the throughput of Hydra.
Before the change separate-debug-info.sh did the stripping itself.
This scheme has a few problems:
1. Stripping happens only on ELF files. *.a and *.o files are skipped.
Derivations have to do it manually. Usually incorrectly
as they don't run $RANLIB (true for `glibc` and `musl`).
2. Stripping happens on all paths. Ideally only `stripDebugList` paths
should be considered.
3. Host strip is called on Target files.
This change offloads stripping logic to strip hook. This strips more
files for `glibc` and `musl`. Now we can remove most $STRIP calls
from individual derivations.
Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
Linking via `-lpthread` (or `-pthread`) is not needed anymore since
`glibc-2.34` since all the functionality is part of `libc.so.6` and
`libpthread.so.6` only exists for backwards-compatibility.
However, e.g. `gcc` (`libgomp` to be precise) expects a `libpthread.so`
to link against, otherwise the configure script will fail. As already
stated in the glibc release-notes itself, it is to expect that a lot
more applications will have issues with this, so I decided to re-add
`libpthread.so` as well.
For `librt.so.1`, the same thing is needed to make sure that Perl still
compiles:
/nix/store/d6y5r7m93x14bmgn2p75fannz39jz66f-binutils-2.35.1/bin/ld: cannot find -lrt
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:490: ../../lib/auto/Time/HiRes/HiRes.so] Error 1
make[1]: Leaving directory '/build/perl-5.34.0/dist/Time-HiRes'
glibc's buildsystem uses its own executables to generate locales.
This does not work for cross-compilation so instead we use localedef
from buildPackages.
The hack of using `crossConfig` to enforce stricter handling of
dependencies is replaced with a dedicated `strictDeps` for that purpose.
(Experience has shown that my punning was a terrible idea that made more
difficult and embarrising to teach teach.)
Now that is is clear, a few packages now use `strictDeps`, to fix
various bugs:
- bintools-wrapper and cc-wrapper
Now it's not an actual archive but a linker script, and the absolute
paths in there were broken due to moving *.a into $static.
Let's fix this up in all *.a in case there are more in future.
This reverts commit 1daf2e26d2, reversing
changes made to c0c50dfcb7.
It seems this is what has been causing all the reliability problems
on Hydra. I'm currently unable to find why it happens, so I'm forced
to revert the update for now. Discussion: #22874.
Enables previously manually disabled stackprotector and stackguard
randomization.
From https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=511811:
If glibc is built with the --enable-stackguard-randomization option,
each application gets a random canary value (at runtime) from /dev/urandom.
If --enable-stackguard-randomization is absent, applications get a static
canary value of "0xff0a0000". This is very unfortunate, because the
attacker may be able to bypass the stack protection mechanism, by placing
those 4 bytes in the canary word, before the actual canary check is
performed (for example in memcpy-based buffer overflows).
