libiconv-darwin depends on Meson, which (indirectly) depends on
libiconv. When libiconv-darwin is set as libiconv, it will cause an
infinite recursion. Avoid the infinite recursion by using libiconvReal
in stage 1. Every stage after that can use libiconv-darwin.
The cc and bintools wrapper contained ad hoc bootstrapping logic for
expand-response-params (which was callPackage-ed in a let binding). This
lead to the strange situation that the bootstrapping logic related to
expand-response-params is split between the wrapper derivations (where
it is duplicated) and the actual stdenv bootstrapping.
To clean this up, the wrappers simply should take expand-response-params
as an ordinary input: They need an adjacent expand-response-params (i.e.
one that runs on their host platform), but don't care about the how.
Providing this is only problematic during stdenv bootstrapping where we
have to pull it from the previous stage at times.
We don't need to artificially make sure that we can execute the wrapper
scripts on the build platform by using stdenv's shell (which comes from
buildPackages) since our cross infrastructure will get us the wrapper
from buildPackages. The upside of this change is that cross-compiled
wrappers (e.g. pkgsCross.aarch64-multiplatform.gcc) will actually work
when executed!
For bootstrapping this is also not a problem, since we have a long
build->build platform chain so runtimeShell is just as good as
stdenvNoCC.shell. We do fall back to old ways, though, by explicitly
using the bootstrap-tools shell in stage2, so the adjacent bash is only
used from stage4 onwards. This is unnecessary in principle (I'll try
removing this hack in the future), but ensures this change causes zero
rebuilds.
Setting the SDK root by default allows `overrideSDK` to correctly set
the SDK version when using a different SDK. It also allows the correct
SDK version to be set when using an older deployment target. Not setting
the correct SDK version can result in unexpected behavior at runtime.
Examples:
* Automatic dark mode switching requires linking against an SDK version
of 10.14 or newer. With the current behavior, the only way to do this
is by using a 10.14+ deployment target even when the application
supports older platforms when build with a newer SDK.
* MetalD3D checks that the system version is at least 14.0. The API it
uses returns a compatibility version when the the SDK is older than
11.0, which causes it to display an error and terminate the
application even when even when its requirements are all met.
- update the hashes and tools needed to extract the bootstrap-tools archive
- unify the x64 and aarch64 unpack process
- unpin libxml for python-minimal
The Darwin bootstrap currently requires curl, but it is not strictly
required. The bootstrap requires it for two things:
* Fetchers; and
* As a transitive dependency of llvm-manpages (via Sphinx).
For the fetchers, the bootstrap curl can be used. For hatch-vcs, the
dependency, its tests can be disabled. Doing this allows curl to be
dropped from the Darwin stdenv bootstrap.
- merge libcxxabi into libcxx for LLVM 12, 13, 14, 15, 16, 17, and git.
- remove the link time workaround `-lc++ -lc++abi` from 58 packages as it is no longer required.
- fixes https://github.com/NixOS/nixpkgs/issues/166205
- provides alternative fixes for. https://github.com/NixOS/nixpkgs/issues/269548https://github.com/NixOS/nix/issues/9640
- pkgsCross.x86_64-freebsd builds work again
This change can be represented in 3 stages
1. merge libcxxabi into libcxx -- files: pkgs/development/compilers/llvm/[12, git]/{libcxx, libcxxabi}
2. update stdenv to account for merge -- files: stdenv.{adapters, cc.wrapper, darwin}
3. remove all references to libcxxabi outside of llvm (about 58 packages modified)
### merging libcxxabi into libcxx
- take the union of the libcxxabi and libcxx cmake flags
- eliminate the libcxx-headers-only package - it was only needed to break libcxx <-> libcxxabi circular dependency
- libcxx.cxxabi is removed. external cxxabi (freebsd) will symlink headers / libs into libcxx.
- darwin will re-export the libcxxabi symbols into libcxx so linking `-lc++` is sufficient.
- linux/freebsd `libc++.so` is a linker script `LINK(libc++.so.1, -lc++abi)` making `-lc++` sufficient.
- libcxx/default.nix [12, 17] are identical except for patches and `LIBCXX_ADDITIONAL_LIBRARIES` (only used in 16+)
- git/libcxx/defaul.nix does not link with -nostdlib when useLLVM is true so flag is removed. this is not much different than before as libcxxabi used -nostdlib where libcxx did not, so libc was linked in anyway.
### stdenv changes
- darwin bootstrap, remove references to libcxxabi and cxxabi
- cc-wrapper: remove c++ link workaround when libcxx.cxxabi doesn't exist (still exists for LLVM pre 12)
- adapter: update overrideLibcxx to account for a pkgs.stdenv that only has libcxx
### 58 package updates
- remove `NIX_LDFLAGS = "-l${stdenv.cc.libcxx.cxxabi.libName}` as no longer needed
- swift, nodejs_v8 remove libcxxabi references in the clang override
https://github.com/NixOS/nixpkgs/pull/292043
This patch switches the CoreFoundation on x86_64-darwin from the open
source swift-corelibs-foundation (CF) to the system CoreFoundation.
This change was motivated by failures building packages for the current
staging-next cycle #263535 due to an apparent incompatibility with the
rpath-based approach to choosing CF or CoreFoundation and macOS 14. This
error often manifests as a crash with an Illegal Instruction.
For example, building aws-sdk-cpp for building Nix will fail this way.
https://hydra.nixos.org/build/239459417/nixlog/1
Application Specific Information:
CF objects must have a non-zero isa
Error Formulating Crash Report:
PC register does not match crashing frame (0x0 vs 0x7FF8094DD640)
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 CoreFoundation 0x7ff8094dd640 CF_IS_OBJC.cold.1 + 14
1 CoreFoundation 0x7ff8094501d0 CF_IS_OBJC + 60
2 CoreFoundation 0x7ff8093155e8 CFRelease + 40
3 ??? 0x10c7a2c61 s_aws_secure_transport_ctx_destroy + 65
4 ??? 0x10c87ba32 aws_ref_count_release + 34
5 ??? 0x10c7b7adb aws_tls_connection_options_clean_up + 27
6 ??? 0x10c596db4 Aws::Crt::Io::TlsConnectionOptions::~TlsConnectionOptions() + 20
7 ??? 0x10c2d249c Aws::CleanupCrt() + 92
8 ??? 0x10c2d1ff0 Aws::ShutdownAPI(Aws::SDKOptions const&) + 64
9 ??? 0x102d9bc6f main + 335
10 dyld 0x202f333a6 start + 1942
According to a [post][1] on the Apple developer forums, hardening was
added to CoreFoundation, and this particular message occurs when you
attempt to release an object it does not recognize as a valid CF object.
(Thank you to @lilyinstarlight for finding this post).
When I switched aws-sdk-cpp to link against CoreFoundation instead of
CF, the error went away. Somehow both libraries were being used. To
prevent dependent packages from linking the wrong CoreFoundation, it
would need to be added as a propagated build input.
Note that there are other issues related to mixing CF and CoreFoundation
frameworks. #264503 fixes an issue with abseil-cpp where it propagates
CF, causing issues when using a different SDK version. Mixing versions
can also cause crashes with Python when a shared object is loaded that
is linked to the “wrong” CoreFoundation.
`NIX_COREFOUNDATION_RPATH` is supposed to make sure the right
CoreFoundation is being used, but it does not appear to be enough on
macOS 14 (presumably due to the hardening). While it is possible to
propagate CoreFoundation manually, the cleaner solution is to make it
the default. CF remains available as `darwin.swift-corelibs-foundation`.
[1]: https://developer.apple.com/forums/thread/739355
curl needs to link against several frameworks, but building the
frameworks (directly or indirectly) depends on curl via fetchurl and
fetchFromGitHub. Break the infinite recursion by building the SDKs’
dependencies in the last stage of the stdenv bootstrap using the prior
stage’s fetchurl and fetchFromGitHub.
To work around intermitent build failures with clang 16, the stdenv
attempted to pass arguments on the command-line on newer versions of
macOS. Unfortunately, the larger `ARG_MAX` is still not large enough to
build qtwebengine. This commit reverts the `NIX_CC_NO_RESPONSE_FILE`
logic in the stdenv. The changes to cc-wrapper in #245282 are needed for
clang 16 to prevent the above-mentioned build failures.
This fixes pyicu (and any other package that uses `icu-config` instead
of the CMake or some other module to get the build flags).
What happened here is the bootstrap disables `patchShebangs` to avoid
propagating the bootstrap tools to the final stdenv (due to `sh` and
`bash` being on the `PATH` from the bootstrap tools). Because of that,
the `#!/bin/sh` line in `icu-config` was not updated, causing it to
invoke the system bash on Darwin. While that is undesirable in its own
right, when the system bash is invoked as `sh`, `echo -n` will print
`-n`, resulting in the breakage see in https://github.com/NixOS/nixpkgs/pull/241951#issuecomment-1627604354.
The fix is to build bash earlier in the bootstrap while making sure it
is picked up over the one in the bootstrap tools. That allows
`patchShebangs` to be enabled during the bootstrap. Any package with
scripts that is included in the final stdenv should now have its
scripts’ shebang lines properly patched.
swift-corelibs uses libcurl to implement `NSURLSession` in Foundation
via the symbols exported by CF. Foundation is not build on Darwin, and
these symbols are not exported by the system CoreFoundation.
By not linking against libcurl, this breaks a cycle between CF and
libcurl. That should allow libcurl to drop the patch disabling
linking against the SystemConfiguration and restore NAT64 support.
Unfortunately, the Darwin stdenv bootstrap still needs to build
dependencies that use `fetchFromGitHub`. While it can drop curl from the
final stdenv, it still needs to use it during the stdenv bootstrap.
In preparation for bumping the LLVM used by Darwin, this change
refactors and reworks the stdenv build process. When it made sense,
existing behaviors were kept to avoid causing any unwanted breakage.
However, there are some differences. The reasoning and differences are
discussed below.
- Improved cycle times - Working on the Darwin stdenv was a tedious
process because `allowedRequisites` determined what was allowed
between stages. If you made a mistake, you might have to wait a
considerable amount of time for the build to fail. Using assertions
makes many errors fail at evaluation time and makes moving things
around safer and easier to do.
- Decoupling from bootstrap tools - The stdenv build process builds as
much as it can in the early stages to remove the requirement that the
bootstrap tools need bumped in order to bump the stdenv itself. This
should lower the barrier to updates and make it easier to bump in the
future. It also allows changes to be made without requiring additional
tools be added to the bootstrap tools.
- Patterned after the Linux stdenv - I tried to follow the patterns
established in the Linux stdenv with adaptations made to Darwin’s
needs. My hope is this makes the Darwin stdenv more approable for
non-Darwin developers who made need to interact with it. It also
allowed some of the hacks to be removed.
- Documentation - Comments were added explaining what was happening and
why things were being done. This is particular important for some
stages that might not be obvious (such as the sysctl stage).
- Cleanup - Converting the intermediate `allowedRequisites` to
assertions revealed that many packages were being referenced that no
longer exist or have been renamed. Removing them reduces clutter and
should help make the stdenv bootstrap process be more understandable.
cctools-llvm is a replacement for cctools that replaces as much of cctools with equivalents from LLVM that it can reasonably do. This was motivated by wanting to reduce dependencies on cctools, which are updated infrequently by upstream.
To provide a motivating example, the version of `strip` included in cctools cannot properly strip the archives in compiler-rt in LLVM 15. Paths are left to bootstrap tools, resulting in failed requisites checks in the final stdenv build. Since `strip` needs replaced, the opportunity was taken to replace other provided they are functional replacements.
Note: This has to be done in cctools (or some equivalent) because some derivations (noteably LLVM) use the bintools of the stdenv directly instead of going through the wrapper.
The following tools from LLVM are not used in this derivation:
* LLD - not fully compatible with ld64 yet and potentially too big of a change;
* libtool - not a drop-in replacement yet because it does not support linker passthrough, which is needed by xcbuild;
* lipo - crashes when running the LLVM test suite;
* install_name_tool - fails when trying to build swift-corefoundation; and.
* randlib - not completely a drop-in replacement, so leaving it out for now.
If other incompatabilities are found, the tools can be reverted or made conditional. For example, cctools `strip` is preferred on older versions of LLVM (which lack the compiler-rt issue) or when cctools itself is a new enough version because `llvm-strip` on LLVM 11 produces files that older verions of `codesign_allocate` cannot process correctly.
One final caveat/note: Some tools are not duplicated or linked from cctools-port. The names of the tools and which ones were linked was determined based on what is provided upstream in Xcode and is installed on macOS system.
The stdenv wouldn't build with it, as
compiler-rt-libc-11.1.0/lib/darwin/libclang_rt.*_osx.a
retained reference to SDKs (which we forbid for final stdenv).
Assigned authorship to Trofi; I just bisected and added condition.
https://github.com/NixOS/nixpkgs/pull/224669#issuecomment-1518225496
A few potentially disruptive changes:
- binutils does not embed ${binutils-unwrapped}/lib as a default library
search path anymore. This will cause link failures for -lbfd -lopcodes
users that did not declare their dependency on those libraries. They
will need to add `libbfd` and `libopcodes` attributes to build inputs.
- `libbfd` and `libopcodes` attributes now just reference
`binutils-unwrapped.{dev,lib}` pair of attributes without patching
`binutils` build system.
We don't patch build system anymore and use multiple outputs out of
existing `binutils` build. That makes the result more maintainable: no
need to handle ever growing list of dependencied of `libbfd`. This time
new addition was `libsframe`.
To accomodate `out` / `lib` output split I had to remove `lib` -> `bin`
backreference by removing legacy lookup path for plugins.
I also did not enable `zstd` just yet as `nixpkgs` version of `zstd`
package pulls in `cmake` into bootstrap sequence.
Changes: https://lists.gnu.org/archive/html/info-gnu/2023-01/msg00003.html
FreeBSD doesn't use LLVM's cxxabi implementation, for backwards
compatibility reasons. Software expects the libcxxrt API when
building on FreeBSD. This fixes the build of
pkgsCross.x86_64-freebsd.boost.
Stdenv on aarch64-darwin pulls in (bootstrap-stage4) objc4, unlike
x86_64. However derivations that otherwise depend on objc4 would use a
a different objc4 derivation on top of the final stdenv.
Because this library defines an LLVM module, having multiple instances
of it in the import path will interfere with builds.
libtool's libtool.m4 script assumes that `file` is available, and can
be found at `/usr/bin/file` (this path is hardwired). Furthermore,
the script with this assumption is vendored into the ./configure
scripts of an enormous number of packages. Without this commit, you
will frequently see errors like this during the configurePhase with
the sandbox enabled:
./configure: line 9595: /usr/bin/file: command not found
Due mostly to luck, this error does not affect native compiles on
nixpkgs' two most popular platforms, x86_64-linux and aarch64-linux.
However it will cause incorrect linker flag detection and a failure to
generate shared libraries for sandboxed cross-builds to a x86_64-linux
host as well as any sandboxed build (cross or native) for the following
hosts: x86_64-freebsd, *-hpux, *-irix, mips64*-linux, powerpc*-linux,
s390x-linux, s390x-tpf, sparc-linux, and *-solaris.
This commit fixes the problem by adding an extra line to fixLibtool()
in pkgs/stdenv/generic/setup.sh. This extra line will scan the
unpacked source code for executable files named "configure" which
contain the following text:
'GNU Libtool is free software; you can redistribute it and/or modify'
This text is taken to be an indicator of a vendored libtool.m4. When
it is found, the configure script containing it is subjected to `sed
-i s_/usr/bin/file_file_` which replaces all occurrences of
`/usr/bin/file` with `file`.
Additionally, the `file` package is now considered to be part of
`stdenv`. It has been added to `common-path.nix` so that the `file`
binary will be found in the `$PATH` of every build, except for the
bootstrap-tools and the first few stages of stdenv boostrapping.
Verified no regressions under:
nix-build --arg pkgs 'import ./. {}' ./lib/tests/release.nix
This commit allows the following commands to complete, which should
enable Hydra to produce bootstrap-files for mips64el:
nix-build \
--option sandbox true \
--option sandbox-fallback false \
pkgs/top-level/release-cross.nix \
-A bootstrapTools.mips64el-linux-gnuabi64.build
nix-build \
--option sandbox true \
--option sandbox-fallback false \
. \
-A pkgsCross.mips64el-linux-gnuabi64.nix_2_4
The `curlMinimal` is to be used throughout the early bootstrap
stages. The final stage will allow the new references of the `curl`.
Fixes: 29526bc2 ('curl: IDN support requires libidn2 package')