nixpkgs/pkgs/development/libraries/glibc/default.nix
Adam Joseph 7553d0fe29 stdenv: Nix-driven bootstrap of gcc
#### Summary

By default, when you type `make`, GCC will compile itself three
times.  This PR inhibits that behavior by configuring GCC with
`--disable-bootstrap`, and reimplements the triple-rebuild using
Nix rather than `make`/`sh`.

 #### Immediate Benefits

- Allow `gcc11` and `gcc12` on `aarch64` (without needing new
  `bootstrapFiles`)
- Faster stdenv rebuilds: the third compilation of gcc
  (i.e. stageCompare) is no longer a `drvInput` of the final stdenv.
  This allows Nix to build stageCompare in parallel with the rest of
  nixpkgs instead of in series.
- No more copying `libgcc_s` out of the bootstrap-files or other
  derivations
- No more Frankenstein compiler: the final gcc and the libraries it
  links against (mpfr, mpc, isl, glibc) are all built by the same
  compiler (xgcc) instead of a mixture of the bootstrapFiles'
  compiler and xgcc.
- No more [static lib{mpfr,mpc,gmp,isl}.a hack]
- Many other small `stdenv` hacks eliminated
- `gcc` and `clang` share the same codepath for more of `cc-wrapper`.

 #### Future Benefits

- This should allow using a [foreign] `bootstrap-files` so long as
  `hostPlatform.canExecute bootstrapFiles`.
- This should allow each of the libraries that ship with `gcc`
  (lib{backtrace, atomic, cc1, decnumber, ffi, gomp, iberty,
  offloadatomic, quadmath, sanitizer, ssp, stdc++-v3, vtv}) to be
  built in separate (one-liner) derivations which `inherit src;`
  from `gcc`, much like https://github.com/NixOS/nixpkgs/pull/132343

 #### Incorporates

- https://github.com/NixOS/nixpkgs/pull/210004
- https://github.com/NixOS/nixpkgs/pull/36948 (unreverted)
- https://github.com/NixOS/nixpkgs/pull/210325
- https://github.com/NixOS/nixpkgs/pull/210118
- https://github.com/NixOS/nixpkgs/pull/210132
- https://github.com/NixOS/nixpkgs/pull/210109
- https://github.com/NixOS/nixpkgs/pull/213909
- https://github.com/NixOS/nixpkgs/pull/216136
- https://github.com/NixOS/nixpkgs/pull/216237
- https://github.com/NixOS/nixpkgs/pull/210019
- https://github.com/NixOS/nixpkgs/pull/216232
- https://github.com/NixOS/nixpkgs/pull/216016
- https://github.com/NixOS/nixpkgs/pull/217977
- https://github.com/NixOS/nixpkgs/pull/217995

 #### Closes

- Closes #108305
- Closes #108111
- Closes #201254
- Closes #208412

 #### Credits

This project was made possible by three important insights, none of
which were mine:

1. @ericson2314 was the first to advocate for this change, and
   probably the first to appreciate its advantages.  Nix-driven
   (external) bootstrap is "cross by default".

2. @trofi has figured out a lot about how to get gcc to not mix up
   the copy of `libstdc++` that it depends on with the copy that it
   builds, by moving the `bootstrapFiles`' `libstdc++` into a
   [versioned directory].  This allows a Nix-driven bootstrap of gcc
   without the final gcc would still having references to the
   `bootstrapFiles`.

3. Using the undocumented variable [`user-defined-trusted-dirs`]
   when building glibc.  When glibc `dlopen()`s `libgcc_s.so`, it
   uses a completely different and totally special set of rules for
   finding `libgcc_s.so`.  This trick is the only way we can put
   `libgcc_s.so` in its own separate outpath without creating
   circular dependencies or dependencies on the bootstrapFiles.  I
   would never have guessed to use this (or that it existed!) if it
   were not for a [comment in guix] which @Mic92 [mentioned].

My own role in this PR was basically: being available to go on a
coding binge at an opportune moment, so we wouldn't waste a
[crisis].

[aarch64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662822938
[amd64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662825857
[nonexistent sysroot]: https://github.com/NixOS/nixpkgs/pull/210004
[versioned directory]: https://github.com/NixOS/nixpkgs/pull/209054
[`user-defined-trusted-dirs`]: https://sourceware.org/legacy-ml/libc-help/2013-11/msg00026.html
[comment in guix]: 5e4ec82181/gnu/packages/gcc.scm (L253)
[mentioned]: https://github.com/NixOS/nixpkgs/pull/210112#issuecomment-1379608483
[crisis]: https://github.com/NixOS/nixpkgs/issues/108305
[foreign]: https://github.com/NixOS/nixpkgs/pull/170857#issuecomment-1170558348
[static lib{mpfr,mpc,gmp,isl}.a hack]: 2f1948af9c/pkgs/stdenv/linux/default.nix (L380)
2023-04-02 13:49:41 -07:00

169 lines
6.5 KiB
Nix

{ lib, stdenv, callPackage
, withLinuxHeaders ? true
, profilingLibraries ? false
, withGd ? false
, withLibcrypt? false
, buildPackages
}:
let
gdCflags = [
"-Wno-error=stringop-truncation"
"-Wno-error=missing-attributes"
"-Wno-error=array-bounds"
];
in
(callPackage ./common.nix { inherit stdenv; } {
inherit withLinuxHeaders withGd profilingLibraries withLibcrypt;
pname = "glibc" + lib.optionalString withGd "-gd";
}).overrideAttrs(previousAttrs: {
# Note:
# Things you write here override, and do not add to,
# the values in `common.nix`.
# (For example, if you define `patches = [...]` here, it will
# override the patches in `common.nix` -- so instead you should
# write `patches = (previousAttrs.patches or []) ++ [ ... ]`.
NIX_NO_SELF_RPATH = true;
postConfigure = ''
# Hack: get rid of the `-static' flag set by the bootstrap stdenv.
# This has to be done *after* `configure' because it builds some
# test binaries.
export NIX_CFLAGS_LINK=
export NIX_LDFLAGS_BEFORE=
export NIX_DONT_SET_RPATH=1
unset CFLAGS
# Apparently --bindir is not respected.
makeFlagsArray+=("bindir=$bin/bin" "sbindir=$bin/sbin" "rootsbindir=$bin/sbin")
'' + lib.optionalString stdenv.buildPlatform.isDarwin ''
# ld-wrapper will otherwise attempt to inject CoreFoundation into ld-linux's RUNPATH
export NIX_COREFOUNDATION_RPATH=
'';
# The pie, stackprotector and fortify hardening flags are autodetected by
# glibc and enabled by default if supported. Setting it for every gcc
# invocation does not work.
hardeningDisable = [ "fortify" "pie" "stackprotector" ];
env = (previousAttrs.env or { }) // {
NIX_CFLAGS_COMPILE = (previousAttrs.env.NIX_CFLAGS_COMPILE or "") + lib.concatStringsSep " "
(builtins.concatLists [
(lib.optionals withGd gdCflags)
# Fix -Werror build failure when building glibc with musl with GCC >= 8, see:
# https://github.com/NixOS/nixpkgs/pull/68244#issuecomment-544307798
(lib.optional stdenv.hostPlatform.isMusl "-Wno-error=attribute-alias")
(lib.optionals ((stdenv.hostPlatform != stdenv.buildPlatform) || stdenv.hostPlatform.isMusl) [
# Ignore "error: '__EI___errno_location' specifies less restrictive attributes than its target '__errno_location'"
# New warning as of GCC 9
# Same for musl: https://github.com/NixOS/nixpkgs/issues/78805
"-Wno-error=missing-attributes"
])
]);
};
# glibc needs to `dlopen()` `libgcc_s.so` but does not link
# against it. Furthermore, glibc doesn't use the ordinary
# `dlopen()` call to do this; instead it uses one which ignores
# most paths:
#
# https://sourceware.org/legacy-ml/libc-help/2013-11/msg00026.html
#
# In order to get it to not ignore `libgcc_s.so`, we have to add its path to
# `user-defined-trusted-dirs`:
#
# https://sourceware.org/git/?p=glibc.git;a=blob;f=elf/Makefile;h=b509b3eada1fb77bf81e2a0ca5740b94ad185764#l1355
#
# Conveniently, this will also inform Nix of the fact that glibc depends on
# gcc.libgcc, since the path will be embedded in the resulting binary.
#
makeFlags =
(previousAttrs.makeFlags or [])
++ lib.optionals (stdenv.cc.cc?libgcc) [
"user-defined-trusted-dirs=${stdenv.cc.cc.libgcc}/lib"
];
postInstall = (if stdenv.hostPlatform == stdenv.buildPlatform then ''
echo SUPPORTED-LOCALES=C.UTF-8/UTF-8 > ../glibc-2*/localedata/SUPPORTED
make -j''${NIX_BUILD_CORES:-1} localedata/install-locales
'' else lib.optionalString stdenv.buildPlatform.isLinux ''
# This is based on http://www.linuxfromscratch.org/lfs/view/development/chapter06/glibc.html
# Instead of using their patch to build a build-native localedef,
# we simply use the one from buildPackages
pushd ../glibc-2*/localedata
export I18NPATH=$PWD GCONV_PATH=$PWD/../iconvdata
mkdir -p $NIX_BUILD_TOP/${buildPackages.glibc}/lib/locale
${lib.getBin buildPackages.glibc}/bin/localedef \
--alias-file=../intl/locale.alias \
-i locales/C \
-f charmaps/UTF-8 \
--prefix $NIX_BUILD_TOP \
${if stdenv.hostPlatform.parsed.cpu.significantByte.name == "littleEndian" then
"--little-endian"
else
"--big-endian"} \
C.UTF-8
cp -r $NIX_BUILD_TOP/${buildPackages.glibc}/lib/locale $out/lib
popd
'') + ''
test -f $out/etc/ld.so.cache && rm $out/etc/ld.so.cache
if test -n "$linuxHeaders"; then
# Include the Linux kernel headers in Glibc, except the `scsi'
# subdirectory, which Glibc provides itself.
(cd $dev/include && \
ln -sv $(ls -d $linuxHeaders/include/* | grep -v scsi\$) .)
fi
# Fix for NIXOS-54 (ldd not working on x86_64). Make a symlink
# "lib64" to "lib".
if test -n "$is64bit"; then
ln -s lib $out/lib64
fi
# Get rid of more unnecessary stuff.
rm -rf $out/var $bin/bin/sln
# Backwards-compatibility to fix e.g.
# "configure: error: Pthreads are required to build libgomp" during `gcc`-build
# because it's not actually needed anymore to link against `pthreads` since
# it's now part of `libc.so.6` itself, but the gcc build breaks if
# this doesn't work.
ln -sf $out/lib/libpthread.so.0 $out/lib/libpthread.so
ln -sf $out/lib/librt.so.1 $out/lib/librt.so
ln -sf $out/lib/libdl.so.2 $out/lib/libdl.so
ln -sf $out/lib/libutil.so.1 $out/lib/libutil.so
touch $out/lib/libpthread.a
# Put libraries for static linking in a separate output. Note
# that libc_nonshared.a and libpthread_nonshared.a are required
# for dynamically-linked applications.
mkdir -p $static/lib
mv $out/lib/*.a $static/lib
mv $static/lib/lib*_nonshared.a $out/lib
# Some of *.a files are linker scripts where moving broke the paths.
sed "/^GROUP/s|$out/lib/lib|$static/lib/lib|g" \
-i "$static"/lib/*.a
# Work around a Nix bug: hard links across outputs cause a build failure.
cp $bin/bin/getconf $bin/bin/getconf_
mv $bin/bin/getconf_ $bin/bin/getconf
'';
separateDebugInfo = true;
passthru =
(previousAttrs.passthru or {})
// lib.optionalAttrs (stdenv.cc.cc?libgcc) {
inherit (stdenv.cc.cc) libgcc;
};
meta = (previousAttrs.meta or {}) // { description = "The GNU C Library"; };
})