In preparation for the deprecation of `stdenv.isX`.
These shorthands are not conducive to cross-compilation because they
hide the platforms.
Darwin might get cross-compilation for which the continued usage of `stdenv.isDarwin` will get in the way
One example of why this is bad and especially affects compiler packages
https://www.github.com/NixOS/nixpkgs/pull/343059
There are too many files to go through manually but a treewide should
get users thinking when they see a `hostPlatform.isX` in a place where it
doesn't make sense.
```
fd --type f "\.nix" | xargs sd --fixed-strings "stdenv.is" "stdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "stdenv'.is" "stdenv'.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "clangStdenv.is" "clangStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "gccStdenv.is" "gccStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "stdenvNoCC.is" "stdenvNoCC.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "inherit (stdenv) is" "inherit (stdenv.hostPlatform) is"
fd --type f "\.nix" | xargs sd --fixed-strings "buildStdenv.is" "buildStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "effectiveStdenv.is" "effectiveStdenv.hostPlatform.is"
fd --type f "\.nix" | xargs sd --fixed-strings "originalStdenv.is" "originalStdenv.hostPlatform.is"
```
While preparing this change, I read the git blame on all of the files I
touched. I saw a working lifetime of building this system which we use
every day and love dearly and keep maintained ourselves. I saw commits
from a 14 year range between 2003 to 2017!! I could not be more thankful
for Eelco's work on building large parts of the foundation of nixpkgs
that all of us rely on now.
However, the end date of that range of the files I looked at the blame
on was 2017. I did not see surviving code from any newer date than that.
Looking at the Git logs, Eelco has been working on other things, and
that's totally fine.
However, it means that our maintenance metadata is out of date on a lot
of packages, and *that*'s the reason I am submitting this change. There
are a lot of packages that don't have anyone with their name on them to
be pinged if they need attention, even if they have had recent activity
(although it is never clear if recent activity was just someone fixing
it because ZHF or because the package actually matters to them).
There are a lot of packages with storied history that maybe don't need
to be in the set anymore at all since they have not been touched in
years; or maybe they are simply finished.
Empty maintainer lists should be a sign that we need to figure out who
maintains it or potentially remove it if it has rotted, and allowing the
maintainer list to be empty if it is already not maintained is part of a
healthy repository ecology.
Either way, I would like to have the maintenance metadata not mislead
anyone into sending Eelco emails about packages he doesn't, in practice,
work on anymore. I have not removed his name from everything; there are
some things that he is the upstream for or has worked on more recently,
for instance, like Nix, which I have left alone.
this appears to have been added to glibc because of the number
of packages in some distributions that were built with CET enabled
before a CET enabled machine was available to test for breakage
with.
we don't have that problem to such an extent and users of hardened
systems will likely want to enable this by default.
...such as APFS.
The problem here is that a change inside the 2.39 patchset creates a
file `ADVISORIES` and also contains a directory `advisories/` and both
are considered the same file on APFS.
Reproducible with `nix-build -A pkgsCross.gnu64.stdenv.cc` on a
`aarch64-darwin` host.
Hence, excluding the file from the patchset as it is non-functional anyways.
Closes#303193
The nixpkgs-unstable channel's programs.sqlite was used to identify
packages producing exactly one binary, and these automatically added
to their package definitions wherever possible.
Announcement: https://lists.gnu.org/archive/html/info-gnu/2024-01/msg00017.html
This release seems relatively harmless in terms of potential fallout.
Most notably is the removal of `crypt(3)` in favor of libxcrypt which
we've done already and compatibility from ISO C2X.
Also decided to drop the old *.gz approach in favor of inlining the
patch with the changes from the release branch directly: it's relatively
small in contrast to certain lockfiles in this repo and having a textual
version makes reviews & diffs easier. See also
https://github.com/NixOS/nixpkgs/pull/258972#discussion_r1454962456 for
more context.
`glibc-2.39` improved `--enable-cet` detection on unsupported platforms
and `./configure` will start failing as:
glibc> configure: error: "CET is only supported on x86_64 or x32"
This is an alternative resolution of the problem identified in
https://github.com/NixOS/nixpkgs/pull/259964
which stated that "glibc depends on buildPackages.glibc for locale
things. buildPackages.glibc depended on buildPackages.libgcc,
which, since it's GCC, depends on the target's bintools, which
depend on the target's glibc, which, again, depends on
buildPackages.glibc, causing an infinute recursion when evaluating
buildPackages.glibc when glibc hasn't come from stdenv (e.g. on
musl)."
The fact that we use pkgsBuildHost.glibc instead of
pkgsBuildBuild.glibc to generate the locales has always been a gross
hack. If we simply remove the gross hack the circularity goes away.
When specifying the `builder` attribute in `stdenv.mkDerivation`, this
will be effectively transformed into
builtins.derivation {
builder = stdenv.shell;
args = [ "-e" builder ];
}
This also means that `default-builder.sh` is never sourced and as a
result it's not guaranteed that `$NIX_ATTRS_SH_FILE` is set to a correct
location[1].
Also, we need to source `.attrs.sh` to source `$stdenv`. So, the
following is done now:
* If `$NIX_ATTRS_SH_FILE` points to a correct location, then use it.
Directly using `.attrs.sh` is problematic for `nix-shell(1)` usage
(see previous commit for more context), so prefer the environment
variable if possible.
* Otherwise, if `.attrs.sh` exists, then use it. See [1] for when this
can happen.
* If neither applies, it can be assumed that `__structuredAttrs` is
turned off and thus nothing needs to be done.
[1] It's possible that it doesn't exist at all - in case of Nix 2.3 or
it can point to a wrong location on older Nix versions with a bug in
`__structuredAttrs`.
Announcement: https://sourceware.org/pipermail/libc-alpha/2023-July/150524.html
So far this looks surprisingly good, I managed to build the stdenv
on `aarch64-linux` and got up to building `zfs` and `nix` on `x86_64-linux`.
The patchset is still empty because the latest commit on the release branch is
the one the 2.38 tag points to. I added an empty file though to keep
things consistent.
Also applied the new version of the DT_HASH fix from ArchLinux[1]. This
one's a way easier version than before because it doesn't contain the
autoconf changes, but only hardcodes the desired ld flags. It was
already confirmed that this patch is sufficient to fix the underlying
problem[2].
[1] e54d98e2d1 (7b1bfda0391ff4c2662e04a5e193c37e233a0738)
[2] https://github.com/ValveSoftware/Proton/issues/6051#issuecomment-1666055553
On some platforms (e.g. loongarch64-linux), libutil.so.1 is not built in
glibc (see glibc login/Makefile), only a dummy libutil.a is provided.
Do not symlink libutil.so but copy libutil.a to glibc.out if
libutil.so.1 does not exist.
### Summary
This PR completely and finally solves the gcc<->glibc circular
`buildInputs` problem, for cross compilation. The same technique
can be applied to native builds in the future.
Closes#213453
### Motivation
Prior to this PR, we had the following circular `buildInputs` problem:
1. gcc has glibc in its `buildInputs`
- a compiled copy of glibc must be present before building gcc;
if it isn't, gcc cripples itself (`inhibit_libc`) and refuses
to build libgcc_s.so
2. glibc has libgcc_s.so in its `buildInputs`
- glibc `dlopen()`s libgcc_s.so in order to implement POSIX
thread cancellation. For security reasons `glibc` requires
that the path to `libgcc_s.so` is [hardwired] into `glibc` at
compile time, so it's technically not a true dynamic link -- it
just pretends to be one.
3. libgcc_s.so is built in the same derivation as gcc
- libgcc_s.so is built as part of the gcc build process
We must cut one of these three links in the loop.
### Previous Attempts
Previously https://github.com/NixOS/nixpkgs/pull/238154 had
attempted to cut link (1) by building `gcc` without `glibc`, and
using the `libgcc_s` which emerges from that build. Unfortunately
this just doesn't work. GCC's configure script extracts quite a lot
of information from the glibc headers (which are a build artifact --
you can't just copy them out of the source tarball) and various
`./configure`-driven linking attempts. If `glibc` isn't around at
build time you wind up with a `libgcc_s.so` that is missing various
unwinder features (see https://github.com/NixOS/nixpkgs/issues/213453
for the most problematic one).
Musl "cuts" link (2), or rather never creates it in the first place.
["Cancellation cleanup handling in musl has no relationship to C++
exceptions and unwinding... glibc implements cancellation as an
exception"](https://wiki.musl-libc.org/functional-differences-from-glibc.html#Thread-cancellation).
IMHO Musl made the smarter decision here. It is incredibly rare to
find a codebase that uses both POSIX thread cancellation *and* C++
exceptions. I have never seen a codebase that uses both *and*
expects them to be aware of each other, and I would be astonished if
one existed. Glibc paid an immense cost in complexity for something
nobody has ever used.
### Changes Made
This PR cuts link (3): instead of building libgcc_s.so as part of
gcc, we build it separately from gcc. Now there is a strict acyclic
graph of `buildInputs`:
```
gccWithoutTargetLibc
|
+--->glibc-nolibgcc
| |
| v
+--->libgcc
| |
| v
+--->glibc
| |
| v
+--->gcc
```
In other words, there's a simple linear `buildInputs` chain
`glibc-nolibgcc` `->` `libgcc` `->` `glibc` `->` `gcc` where all
four packages are compiled by (and therefore have as a
`(native)BuildInput`) `gccWithoutTargetLibc`.
`gccWithoutTargetLibc` and `glibc-nolibgcc` are strictly
bootstrapping artifacts; nothing else has them as a `buildInput` and
they shouldn't appear in the closure of any final deployment
packages. `glibc-nolibgcc` lacks `libgcc_s.so`, so it will segfault
if you try to use it with POSIX thread cancellation. Fortunately
all we need from it is (a) its headers (`lib.getDev`) and (b) to use
it in the `./configure` script for `libgcc`.
When translated over to the native bootstrap, `xgcc` takes the place
of `gccWithoutTargetLibc`, and the "first `glibc`" (we build two of
them) takes the place of `glibc-nolibgcc`. At that point our native
and cross bootstrap have the same overall architecture, and it
becomes possible to merge them (at last!)
[213453]: https://github.com/NixOS/nixpkgs/issues/213453
[238154]: https://github.com/NixOS/nixpkgs/pull/238154
[hardwired]: 7553d0fe29/pkgs/development/libraries/glibc/default.nix (L69-L88)
The way `nixpkgs` runs parallel `localedef` instances shares
`--prefix=/build` flag. As a result `localedef` processes
non-deterministically extend the file with new locales (hopefully
without data corruption):
https://github.com/NixOS/nixpkgs/issues/245360
Co-authored-by: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com>
This commit causes glibc/info.nix to use the standard builder's
`buildPhase` so things like `preBuild`, `postBuild`,
`enableParallelBuilding`, etc work correctly.
This commit massively reduces the build latency for glibcLocales by
allowing it to build in parallel. This requires passing
`-j$NIX_BUILD_CORES` via the glibc-specific make variable
`PARALLELMFLAGS`.
This commit also fixes a preexisting bug where the glibcLocales
package would ignore `preBuild` and `postBuild`.
Our gcc_multi and glibc_multi expressions merge together a
32-bit-targeted and 64-bit-targeted gcc. However they do not thread
through the passthru.libgcc from these merged gccs.
This commit corrects that.
It also extends passthru.libgcc to allow a *list* rather than just a
single outpath.
Resolves part of #221891 (at least getting it back to the error
message it gave before).
