Commit Graph

414 Commits

Author SHA1 Message Date
Sergei Trofimovich
4d4f5e2db1 glibcLocales: enable parallel building 2023-06-20 09:38:33 +01:00
Maximilian Bosch
1f72072bdd
Merge pull request #238027 from amjoseph-nixpkgs/pr/glibc/parallel
glibcLocales: use more than one core to build
2023-06-17 14:55:40 +02:00
Adam Joseph
7306386eb3 glibcInfo: use makeFlags instead of buildPhase
This commit causes glibc/info.nix to use the standard builder's
`buildPhase` so things like `preBuild`, `postBuild`,
`enableParallelBuilding`, etc work correctly.
2023-06-15 20:14:29 -07:00
Adam Joseph
238b793373 glibc: allow users of glibc/common.nix to override makeFlags
This commit allows to include `makeFlags` in a glibc derivation
without clobbering the flags from `common.nix`
2023-06-15 20:14:29 -07:00
Adam Joseph
b243596eb7 glibcLocales: use more than one core to build
This commit massively reduces the build latency for glibcLocales by
allowing it to build in parallel.  This requires passing
`-j$NIX_BUILD_CORES` via the glibc-specific make variable
`PARALLELMFLAGS`.

This commit also fixes a preexisting bug where the glibcLocales
package would ignore `preBuild` and `postBuild`.
2023-06-15 19:36:23 -07:00
Adam Joseph
cc22c861e8 glibc: allow users of glibc/common.nix to override makeFlags
This commit allows to include `makeFlags` in a glibc derivation
without clobbering the flags from `common.nix`
2023-06-15 19:36:18 -07:00
Linus Heckemann
4d649f2b63 glibc: split getent into its own output
Many dependents only require getent and not all the locale generation
tools that are included in the `bin` output. This can save some
closure size!
2023-06-12 10:26:52 +02:00
Adam Joseph
0e9ef0a07d cc-wrapper: when merging gcc32 and gcc64, merge libgcc as well
Our gcc_multi and glibc_multi expressions merge together a
32-bit-targeted and 64-bit-targeted gcc.  However they do not thread
through the passthru.libgcc from these merged gccs.

This commit corrects that.

It also extends passthru.libgcc to allow a *list* rather than just a
single outpath.

Resolves part of #221891 (at least getting it back to the error
message it gave before).
2023-05-09 00:16:24 -07:00
Adam Joseph
c5a4cc8396 glibc: suppress warning about IEEE-standard long double 2023-04-05 15:54:26 -07:00
Vladimír Čunát
4eaca2b138
Merge #188492: glibc: 2.35-224 -> 2.37-8
...into staging
2023-04-03 20:29:07 +02:00
Adam Joseph
7553d0fe29 stdenv: Nix-driven bootstrap of gcc
#### Summary

By default, when you type `make`, GCC will compile itself three
times.  This PR inhibits that behavior by configuring GCC with
`--disable-bootstrap`, and reimplements the triple-rebuild using
Nix rather than `make`/`sh`.

 #### Immediate Benefits

- Allow `gcc11` and `gcc12` on `aarch64` (without needing new
  `bootstrapFiles`)
- Faster stdenv rebuilds: the third compilation of gcc
  (i.e. stageCompare) is no longer a `drvInput` of the final stdenv.
  This allows Nix to build stageCompare in parallel with the rest of
  nixpkgs instead of in series.
- No more copying `libgcc_s` out of the bootstrap-files or other
  derivations
- No more Frankenstein compiler: the final gcc and the libraries it
  links against (mpfr, mpc, isl, glibc) are all built by the same
  compiler (xgcc) instead of a mixture of the bootstrapFiles'
  compiler and xgcc.
- No more [static lib{mpfr,mpc,gmp,isl}.a hack]
- Many other small `stdenv` hacks eliminated
- `gcc` and `clang` share the same codepath for more of `cc-wrapper`.

 #### Future Benefits

- This should allow using a [foreign] `bootstrap-files` so long as
  `hostPlatform.canExecute bootstrapFiles`.
- This should allow each of the libraries that ship with `gcc`
  (lib{backtrace, atomic, cc1, decnumber, ffi, gomp, iberty,
  offloadatomic, quadmath, sanitizer, ssp, stdc++-v3, vtv}) to be
  built in separate (one-liner) derivations which `inherit src;`
  from `gcc`, much like https://github.com/NixOS/nixpkgs/pull/132343

 #### Incorporates

- https://github.com/NixOS/nixpkgs/pull/210004
- https://github.com/NixOS/nixpkgs/pull/36948 (unreverted)
- https://github.com/NixOS/nixpkgs/pull/210325
- https://github.com/NixOS/nixpkgs/pull/210118
- https://github.com/NixOS/nixpkgs/pull/210132
- https://github.com/NixOS/nixpkgs/pull/210109
- https://github.com/NixOS/nixpkgs/pull/213909
- https://github.com/NixOS/nixpkgs/pull/216136
- https://github.com/NixOS/nixpkgs/pull/216237
- https://github.com/NixOS/nixpkgs/pull/210019
- https://github.com/NixOS/nixpkgs/pull/216232
- https://github.com/NixOS/nixpkgs/pull/216016
- https://github.com/NixOS/nixpkgs/pull/217977
- https://github.com/NixOS/nixpkgs/pull/217995

 #### Closes

- Closes #108305
- Closes #108111
- Closes #201254
- Closes #208412

 #### Credits

This project was made possible by three important insights, none of
which were mine:

1. @ericson2314 was the first to advocate for this change, and
   probably the first to appreciate its advantages.  Nix-driven
   (external) bootstrap is "cross by default".

2. @trofi has figured out a lot about how to get gcc to not mix up
   the copy of `libstdc++` that it depends on with the copy that it
   builds, by moving the `bootstrapFiles`' `libstdc++` into a
   [versioned directory].  This allows a Nix-driven bootstrap of gcc
   without the final gcc would still having references to the
   `bootstrapFiles`.

3. Using the undocumented variable [`user-defined-trusted-dirs`]
   when building glibc.  When glibc `dlopen()`s `libgcc_s.so`, it
   uses a completely different and totally special set of rules for
   finding `libgcc_s.so`.  This trick is the only way we can put
   `libgcc_s.so` in its own separate outpath without creating
   circular dependencies or dependencies on the bootstrapFiles.  I
   would never have guessed to use this (or that it existed!) if it
   were not for a [comment in guix] which @Mic92 [mentioned].

My own role in this PR was basically: being available to go on a
coding binge at an opportune moment, so we wouldn't waste a
[crisis].

[aarch64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662822938
[amd64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662825857
[nonexistent sysroot]: https://github.com/NixOS/nixpkgs/pull/210004
[versioned directory]: https://github.com/NixOS/nixpkgs/pull/209054
[`user-defined-trusted-dirs`]: https://sourceware.org/legacy-ml/libc-help/2013-11/msg00026.html
[comment in guix]: 5e4ec82181/gnu/packages/gcc.scm (L253)
[mentioned]: https://github.com/NixOS/nixpkgs/pull/210112#issuecomment-1379608483
[crisis]: https://github.com/NixOS/nixpkgs/issues/108305
[foreign]: https://github.com/NixOS/nixpkgs/pull/170857#issuecomment-1170558348
[static lib{mpfr,mpc,gmp,isl}.a hack]: 2f1948af9c/pkgs/stdenv/linux/default.nix (L380)
2023-04-02 13:49:41 -07:00
Maximilian Bosch
047f379d38
glibc: use patch from ArchLinux to re-enable DT_HASH 2023-03-29 09:50:29 +02:00
Maximilian Bosch
b42ee8b817
glibc: 2.35-224 -> 2.37-8
Announcement: https://sourceware.org/pipermail/libc-alpha/2022-August/141193.html
Announcement: https://sourceware.org/pipermail/libc-alpha/2023-February/145190.html
2023-03-29 09:50:24 +02:00
Artturin
6b2a05e190 treewide: manual fixups for
treewide: use toString on list NIX_CFLAGS_COMPILE
treewide: move NIX_CFLAGS_COMPILE to the env attrset
2023-02-22 21:23:04 +02:00
Artturin
f9fdf2d402 treewide: move NIX_CFLAGS_COMPILE to the env attrset
with structuredAttrs lists will be bash arrays which cannot be exported
which will be a issue with some patches and some wrappers like cc-wrapper

this makes it clearer that NIX_CFLAGS_COMPILE must be a string as lists
in env cause a eval failure
2023-02-22 21:23:04 +02:00
Felix Buehler
cdb39a86e0 treewide: use optionalString 2023-02-13 21:52:34 +01:00
Sergei Trofimovich
6728277e19 libc: wipe out all references from copied libgcc_s.so.1
Without the change a copy from freshly built `gcc` still retains a
reference even after `patchelf --remove-rpath` because `patchelf` does
not cleanup dynamic sprintgs section.

The change stubs the reference out and fixes build on bootstraps where
`gcc` is built before `glibc`.
2023-01-14 18:23:56 +00:00
Sergei Trofimovich
76f5618e1e glibc: copy libgcc_s.so from .lib output if it exists
Otherwise copy it from the default output. The difference is visible
when we build `glibc` with:

- `bootstrapTools` `gcc`: ${stdenv.cc.cc.out}/lib/libgcc_s.so.1 is used
- `nixpkgs` `gcc`: ${stdenv.cc.cc.lib}/lib/libgcc_s.so.1 is used

Noticed when experimented with multiple `gcc` rebuilds in bootstrap.

While at it killing `RUNPATH` reference to bootstrap `glibc`.
2023-01-14 11:58:31 +00:00
github-actions[bot]
4712ed9439
Merge master into staging-next 2023-01-14 00:02:26 +00:00
Adam Joseph
97c335216d glibc: allow overriding common.nix 2023-01-13 19:16:30 +02:00
Vladimír Čunát
4835c7a74c
glibc, python3Packages.twisted: resolve temporary workaround
It certainly seems better to patch the twisted test than glibc.
2023-01-04 20:28:54 +01:00
Robert Hensing
16f5747575
Merge pull request #175649 from Artturin/opt-in-structured-attrs
stdenv: support opt-in __structuredAttrs
2022-12-10 21:12:43 +01:00
Artturin
c01f509e44 treewide: source .attrs in builders
if theres a source $stdenv then this is needed

for structuredAttrs
2022-12-08 21:09:02 +02:00
Artturin
adc8900df1 treewide: fix some core package structuredAttrs 2022-12-08 21:05:28 +02:00
Vladimír Čunát
5dffcba8fe
glibc: revert one patch from those added in parent commit
For now.  It causes issues in python3Packages.twisted testsuite,
so maybe it's buggy.  The tests might be buggy instead, but so far
we've had no report of the issues that are to be fixed by the patch.
https://github.com/NixOS/nixpkgs/pull/201805#issuecomment-1342735635
2022-12-08 14:34:53 +01:00
Vladimír Čunát
ec7567ba19
glibc: 2.35-163 -> 2.35-224 2022-12-06 15:51:01 +01:00
Dmitry Kalinkin
68e63f0ee3
Merge pull request #202827 from vcunat/p/glibc-kernel-versions
glibc: bump the minimum kernel version
2022-12-03 18:30:33 -05:00
Vladimír Čunát
68b44c86c2
glibc: bump the minimum kernel version
I don't think anyone sane is really using kernel < 3.10 nowadays.
(At least in a use case with glibc from nixpkgs.)

As another suspect with old kernels, Ubuntu seems to start at 3.13 now.
2022-11-25 11:38:48 +01:00
Sergei Trofimovich
ca95ee0977 glibc: backport make-4.4 fix
Not updating the whole snapshot as unrelated patches break some of
timezone-related tests:
    https://github.com/NixOS/nixpkgs/pull/201805#issuecomment-1320917345

Let's unblock `make-4.4` first.
2022-11-19 16:49:53 +00:00
Victor Fuentes
40ade45200
glib-locales: store SUPPORTED locales file 2022-11-02 13:59:53 -04:00
Jan Tojnar
457f28f6f8 Merge branch 'master' into staging-next
; Conflicts:
;	pkgs/development/tools/codespell/default.nix

codespell 2.2.2 switched to pyproject & setuptools_scm:
https://github.com/codespell-project/codespell/pull/2523
2022-10-19 05:24:28 +02:00
Zhaofeng Li
22b6046192 glibc: Don't inject CoreFoundation RUNPATH on Darwin
Ref: #137877
2022-10-18 07:24:18 -04:00
Martin Weinelt
ff30c899d8
glibc: make crypt support optional
The libcrypt library is going to be replaced with libxcrypt in packages
that require it.
2022-10-09 18:07:53 +02:00
Graham Christensen
c2b898da76 treewide: drop -l$NIX_BUILD_CORES
Passing `-l$NIX_BUILD_CORES` improperly limits the overall system load.

For a build machine which is configured to run `$B` builds where each
build gets `total cores / B` cores (`$C`), passing `-l $C` to make will
improperly limit the load to `$C` instead of `$B * $C`.

This effect becomes quite pronounced on machines with 80 cores, with
40 simultaneous builds and a cores limit of 2. On a machine with this
configuration, Nix will run 40 builds and make will limit the overall
system load to approximately 2. A build machine with this many cores
can happily run with a load approaching 80.

A non-solution is to oversubscribe the machine, by picking a larger
`$C`. However, there is no way to divide the number of cores in a way
which fairly subdivides the available cores when `$B` is greater than
1.

There has been exploration of passing a jobserver in to the sandbox,
or sharing a jobserver between all the builds. This is one option, but
relatively complicated and only supports make. Lots of other software
uses its own implementation of `-j` and doesn't support either `-l` or
the Make jobserver.

For the case of an interactive user machine, the user should limit
overall system load using `$B`, `$C`, and optionally systemd's
cpu/network/io limiting features.

Making this change should significantly improve the utilization of our
build farm, and improve the throughput of Hydra.
2022-09-22 16:01:23 -04:00
Minijackson
81c37edce4
glibcLocales: follow host platform endianness 2022-09-05 12:57:53 +02:00
github-actions[bot]
4ad33a5c7c
Merge master into staging-next 2022-08-18 06:01:24 +00:00
Sergei Trofimovich
198a940c61 glibc: add a few TODOs aroung libgcc_s.so hack
Make it clearer why the hack is still there.
2022-08-18 06:36:13 +01:00
Sergei Trofimovich
b3b672d5a1 setup-hooks/separate-debug-info.sh: don't inhibit strip hook
Before the change separate-debug-info.sh did the stripping itself.
This scheme has a few problems:
1. Stripping happens only on ELF files. *.a and *.o files are skipped.
   Derivations have to do it manually. Usually incorrectly
   as they don't run $RANLIB (true for `glibc` and `musl`).
2. Stripping happens on all paths. Ideally only `stripDebugList` paths
   should be considered.
3. Host strip is called on Target files.

This change offloads stripping logic to strip hook. This strips more
files for `glibc` and `musl`. Now we can remove most $STRIP calls
from individual derivations.

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
2022-08-07 12:49:37 +01:00
Jörg Thalheim
5de6b3ecd7
Merge pull request #182281 from helsinki-systems/feat/glibc-sec
glibc: improve configure options
2022-07-27 22:09:44 +01:00
Bernardo Meurer
272fc53af1 glibc: 2.34-210 -> 2.35-163 2022-07-22 22:31:14 -07:00
ajs124
1487fabf60 glibc: enable Intel CET on x86 2022-07-21 00:32:03 +02:00
ajs124
8f3c8aee8a glibc: explicitly enable stack-protector
should be detected automatically
2022-07-21 00:32:03 +02:00
ajs124
34b92568d1 glibc: remove obsolete configure option
see https://sourceware.org/bugzilla/show_bug.cgi?id=27872
2022-07-21 00:32:03 +02:00
github-actions[bot]
1767ba1baa
Merge master into staging-next 2022-05-29 18:01:10 +00:00
Artturin
513b7f1010 glibc_multi: match output ordering of glibc
glibc has an exception in that 'out' is the default output instead of 'bin'

it should be matched here for consistency
2022-05-29 19:54:32 +03:00
sternenseemann
dd0b96be19
Merge pull request #173893 from trofi/fix-glibc-for-gnat6
glibc: apply pending PR29162 to unbreak gnat6
2022-05-23 20:17:48 +02:00
Sergei Trofimovich
3c211fb591 glibc: apply pending PR29162 to unbreak gnat6
commit e938c0274 "Don't add access size hints to fortifiable functions"
converted a few '__attr_access ((...))' into '__fortified_attr_access (...)'
calls.

But one of conversions had double parentheses of '__fortified_attr_access (...)'.

Noticed as a gnat6 build failure:

    /<<NIX>>-glibc-2.34-210-dev/include/bits/string_fortified.h:110:50: error: macro "__fortified_attr_access" requires 3 arguments, but only 1 given

The change fixes parentheses.
2022-05-23 18:40:49 +01:00
Artturin
0f9ee45a6d glibc: enable strictDeps
verified by building and diffing 'glibc' and 'glibcInfo'
2022-05-22 16:40:40 +03:00
Maximilian Bosch
5a3e803bf3
glibc: 2.34-115 -> 2.34-210 2022-05-06 10:28:53 +02:00
Sergei Trofimovich
bf990cc3cc glibc: unconditionally disable pie
glibc already has to be careful not to create extra dynamic relocations
in ld.so. For that it enables -fPIC/-fPIE selectively.
2022-04-18 16:06:50 +01:00