Without the change `runPhase` fails on tarballs like
`diffoscope-269` that contain single top-level `-269` root as:
diffoscope> unpacking source archive /nix/store/p620nidkm73vrp0z6kk5krmrm4vg7bxd-diffoscope-269.tar.bz2
diffoscope> source root is -269
diffoscope> setting SOURCE_DATE_EPOCH to timestamp 1717143039 of file ./-269/tests/utils/versions.py
diffoscope> chmod: invalid mode: ‘-269’
diffoscope> Try 'chmod --help' for more information.
Currently `diffoscope-269` has a `sourceRoot = "./-269";` workaround to
bypass the failure.
Most Linux distributions are enabling this these days and it does
protect against real world vulnerabilities as demonstrated by
CVE-2018-16864 and CVE-2018-16865.
Fix#53753.
Information on llvm version support gleaned from
6609892a2d68e07da3e5092507a730
Information on gcc version support a lot harder to gather,
but both 32bit and 64bit arm do appear to be supported
based on the test suite.
A second take at eb28e5e72e, which was reverted for the extra logging
during the internals of `nix-shell -p`. This commit does the same
logging, but to $NIX_LOG_FD instead, which is echoed during any normal
build, but not during the internals of `nix-shell -p`.
[1]: eb28e5e72e
libiconv-darwin depends on Meson, which (indirectly) depends on
libiconv. When libiconv-darwin is set as libiconv, it will cause an
infinite recursion. Avoid the infinite recursion by using libiconvReal
in stage 1. Every stage after that can use libiconv-darwin.
The cc and bintools wrapper contained ad hoc bootstrapping logic for
expand-response-params (which was callPackage-ed in a let binding). This
lead to the strange situation that the bootstrapping logic related to
expand-response-params is split between the wrapper derivations (where
it is duplicated) and the actual stdenv bootstrapping.
To clean this up, the wrappers simply should take expand-response-params
as an ordinary input: They need an adjacent expand-response-params (i.e.
one that runs on their host platform), but don't care about the how.
Providing this is only problematic during stdenv bootstrapping where we
have to pull it from the previous stage at times.
We don't need to artificially make sure that we can execute the wrapper
scripts on the build platform by using stdenv's shell (which comes from
buildPackages) since our cross infrastructure will get us the wrapper
from buildPackages. The upside of this change is that cross-compiled
wrappers (e.g. pkgsCross.aarch64-multiplatform.gcc) will actually work
when executed!
For bootstrapping this is also not a problem, since we have a long
build->build platform chain so runtimeShell is just as good as
stdenvNoCC.shell. We do fall back to old ways, though, by explicitly
using the bootstrap-tools shell in stage2, so the adjacent bash is only
used from stage4 onwards. This is unnecessary in principle (I'll try
removing this hack in the future), but ensures this change causes zero
rebuilds.
As of late, `final: prev: …` for overlays has become more prevalent in
newer code. This is also exhibited in some code (presumably added
recently) in stdenv. This change is not about any merits or demerits of
any naming convention, but rather aims to make the nomenclature in
stdenv bootstrapping consistent to lessen confusion.
I've chosen to stick to `self: super: …` convention because:
1. It is more common in the code as it stands.
2. Using `final: prev: …` makes the code more confusing, als it causes
`prev` to be in scope alongside `prevStage`. `prevStage` actually
bears no relation to `prev` even though their naming suggests it,
making it easy to confuse them (the former is the (final) package set
of the previous stage while the latter is just the `prev`/`super` of
the overlay “chaining” on a completely fresh package set, i.e. `prev`
doesn't even relate to the previous stage's `overrides` argument).
This change also corrects a naming error in stdenv/native which had no
effect, as the variables were unused.
This fixes using inputDerivation on derivations that are fixed-output.
Previously:
```
nix-repl> drv = runCommand "huh" { outputHash = "sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU="; outputHashAlgo = "sha256"; outputHashType = "flat"; } "touch $out"
nix-repl> drv.inputDerivation
«derivation /nix/store/d8mjs6cmmvsr1fv7psm6imis5pmh9bcs-huh.drv»
nix-repl> :b drv.inputDerivation
error: fixed output derivation 'huh' is not allowed to refer to other store paths.
You may need to use the 'unsafeDiscardReferences' derivation attribute, see the manual for more details.
```
Fixes: https://github.com/NixOS/nixpkgs/issues/304209
Nixpkgs tries to print a helpful message when it blocks unfree packages,
but the suggestion is subtly broken. The predicate only matches on the
package's name, but the suggestion includes the full name-version pair.
Fixed by formatting the message with the same function as the predicate.
This issue arises because check-meta defines its own local getName with
semantics divergent from lib.getName. The former includes the version,
the latter does not.
Example Before:
Alternatively you can configure a predicate to allow specific packages:
{ nixpkgs.config.allowUnfreePredicate = pkg: builtins.elem (lib.getName pkg) [
"obsidian-1.5.12"
];
}
Example After:
Alternatively you can configure a predicate to allow specific packages:
{ nixpkgs.config.allowUnfreePredicate = pkg: builtins.elem (lib.getName pkg) [
"obsidian"
];
}
Fixes#303116
checkValidity has the responsibility to check if a derivation's attributes are valid.
Previously it also had the overloaded task of creating a subset of meta attributes:
- unfree
- broken
- unsupported
- insecure
Not only is this overloading strange, these attributes were only ever consumed by `commonMeta`.
This change makes checkValidity _only_ check for validity, and removes the creation of any meta attributes from `checkValidity` and moves them to `commonMeta`.
This is technically a breaking change but I don't expect any external nixpkgs consumers to rely on these implementation details.
Setting the SDK root by default allows `overrideSDK` to correctly set
the SDK version when using a different SDK. It also allows the correct
SDK version to be set when using an older deployment target. Not setting
the correct SDK version can result in unexpected behavior at runtime.
Examples:
* Automatic dark mode switching requires linking against an SDK version
of 10.14 or newer. With the current behavior, the only way to do this
is by using a 10.14+ deployment target even when the application
supports older platforms when build with a newer SDK.
* MetalD3D checks that the system version is at least 14.0. The API it
uses returns a compatibility version when the the SDK is older than
11.0, which causes it to display an error and terminate the
application even when even when its requirements are all met.
This is effectively a rewrite of `overrideSDK`. It was required because
`wrapGAppsHook` propagates `depsTargetTarget` with the expectation that
it will effectively be `buildInputs` when the hook is itself used as a
`nativeBuildInput`. This propagates Gtk, which itself propagates the
default Dariwn SDK, making it effectively impossible to override the SDK
when a package depends on Gtk and uses `wrapGAppsHook`.
This rewrite implements the following improvements:
* Cross-compilation should be supported correctly (untested);
* Supports public and private frameworks;
* Supports SDK `libs`;
* Remaps instead of replacing extra (native) build inputs in the stdenv;
* Updates any Darwin framework references in `nix-support`; and
* It updates `xcodebuild` regardless of which input its in.
The implementation avoids recursion for performance reasons. Instead, it
enumerates transitive dependencies and walks the list from the leaf
packages backwards to the parent packages.
- update the hashes and tools needed to extract the bootstrap-tools archive
- unify the x64 and aarch64 unpack process
- unpin libxml for python-minimal
In the code examples for using `allowUnfreePredicate` and
`allowNonSourcePredicate`, we use `lib.getName` (which removes
the version) and not the local `getName` (which keeps the version).
`finalAttrs` is never the first argument. This should have been
called `prevAttrs` all along.
It confused me for a bit, because the callback _must not_ be strict
in `finalAttrs` (the first of _two_ parameters), or it will
recurse infinitely while trying to figure out what the attrNames are.
- unpin LLVM11. fix discrepancy with freshBootstrapTools and the tools
built on hydra. pinning the stdenv for the hydra build doesn't pin the
tools as the included packages are able to change.
- remove unused LLVM tools & libs which reduces the uncompressed and
compressed file sizes by more than 1/2. compressed tarball is now 40M
and uncompressed is around 200M
- add @loader_path/. to dylibs that reference other libs in the archive.
this is needed for libraries with re-exports.
- validate shared objects with @rpath references contain the reference
in lib
- add a test to verify that the @loader_path/ works for libc++ as it
re-exports libc++abi
The Darwin bootstrap currently requires curl, but it is not strictly
required. The bootstrap requires it for two things:
* Fetchers; and
* As a transitive dependency of llvm-manpages (via Sphinx).
For the fetchers, the bootstrap curl can be used. For hatch-vcs, the
dependency, its tests can be disabled. Doing this allows curl to be
dropped from the Darwin stdenv bootstrap.
These two commits make for a cleaner commit history and git blame than
https://github.com/NixOS/nixpkgs/pull/295105, where this refactor
was developed.
See its commit messages for details and design choices, esp. up to
and including 37f76fd4c3.
- merge libcxxabi into libcxx for LLVM 12, 13, 14, 15, 16, 17, and git.
- remove the link time workaround `-lc++ -lc++abi` from 58 packages as it is no longer required.
- fixes https://github.com/NixOS/nixpkgs/issues/166205
- provides alternative fixes for. https://github.com/NixOS/nixpkgs/issues/269548https://github.com/NixOS/nix/issues/9640
- pkgsCross.x86_64-freebsd builds work again
This change can be represented in 3 stages
1. merge libcxxabi into libcxx -- files: pkgs/development/compilers/llvm/[12, git]/{libcxx, libcxxabi}
2. update stdenv to account for merge -- files: stdenv.{adapters, cc.wrapper, darwin}
3. remove all references to libcxxabi outside of llvm (about 58 packages modified)
### merging libcxxabi into libcxx
- take the union of the libcxxabi and libcxx cmake flags
- eliminate the libcxx-headers-only package - it was only needed to break libcxx <-> libcxxabi circular dependency
- libcxx.cxxabi is removed. external cxxabi (freebsd) will symlink headers / libs into libcxx.
- darwin will re-export the libcxxabi symbols into libcxx so linking `-lc++` is sufficient.
- linux/freebsd `libc++.so` is a linker script `LINK(libc++.so.1, -lc++abi)` making `-lc++` sufficient.
- libcxx/default.nix [12, 17] are identical except for patches and `LIBCXX_ADDITIONAL_LIBRARIES` (only used in 16+)
- git/libcxx/defaul.nix does not link with -nostdlib when useLLVM is true so flag is removed. this is not much different than before as libcxxabi used -nostdlib where libcxx did not, so libc was linked in anyway.
### stdenv changes
- darwin bootstrap, remove references to libcxxabi and cxxabi
- cc-wrapper: remove c++ link workaround when libcxx.cxxabi doesn't exist (still exists for LLVM pre 12)
- adapter: update overrideLibcxx to account for a pkgs.stdenv that only has libcxx
### 58 package updates
- remove `NIX_LDFLAGS = "-l${stdenv.cc.libcxx.cxxabi.libName}` as no longer needed
- swift, nodejs_v8 remove libcxxabi references in the clang override
https://github.com/NixOS/nixpkgs/pull/292043
this equates to -ftrivial-auto-var-init=pattern
clang has removed support for -ftrivial-auto-var-init=zero and
are unlikely to re-add it, so use -ftrivial-auto-var-init=pattern
on both compilers if only to make behaviour more consistent
between the two.
add to pkgsExtraHardening's defaultHardeningFlags.
This is a small simplification of the control flow surrounding these cases. It should make it more obvious when each case happens, and also explicitly defines the current behaviour of --replace.
Example with `clangUseLLVM` which is the default when using `useLLVM`
```nix
config.replaceCrossStdenv = { buildPackages, baseStdenv }:
if baseStdenv.targetPlatform.useLLVM or false
then (buildPackages.stdenvAdapters.overrideCC baseStdenv buildPackages.llvmPackages_16.clangUseLLVM)
else baseStdenv;
```
The conditional necessary, otherwise the other sets(such as `pkgsCross.aarch64-multiplatform.llvmPackages`)
without `useLLVM` will use the stdenv without the necessary conditions to avoid infinite
recursion because of [targetLlvmLibraries](644b234e1c/pkgs/development/compilers/llvm/16/default.nix (L208))
usage.
[`replaceStdenv` is not used when cross-compiling](d77bda728d/pkgs/stdenv/cross/default.nix (L12-L13))
`replaceStdenv` uses an additional stage to replace the stdenv to avoid
infinite recursion and other issues but that should not be necessary for cross.
The aarch64 musl bootstrap tools are woefully outdated. Just getting
them to the point of being able to build new versions of themselves
required a number of hacks[1] that can be reverted once we have new
bootstrap tools, and before that it stdenv didn't even build for the
preceding three years.
[1]: https://github.com/NixOS/nixpkgs/pull/169764
So, following the script established by previous bootstrap tools
updates:
Files came from this Hydra build:
https://hydra.nixos.org/build/246470544
…which used nixpkgs revision dd5621df6d
to instantiate:
/nix/store/g480ass2vjmakaq03z7k2j95xnxh206a-stdenv-bootstrap-tools.drv
…and then built:
/nix/store/95lm0y33dayag4542s8bi83s31bw68dr-stdenv-bootstrap-tools
I downloaded these files from Hydra and prefetched them into the Nix
store with the following commands:
STOREPATH=95lm0y33dayag4542s8bi83s31bw68dr-stdenv-bootstrap-tools
OPTIONS="--option binary-caches https://cache.nixos.org --option trusted-public-keys cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
nix --extra-experimental-features nix-command store prefetch-file \
file://$(nix --extra-experimental-features nix-command store add-file --name bootstrap-tools.tar.xz $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/bootstrap-tools.tar.xz)
nix --extra-experimental-features nix-command store prefetch-file --executable \
file://$(nix --extra-experimental-features nix-command store add-path --name busybox $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/busybox)
These commands produced the following output:
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
Downloaded 'file:///nix/store/fm8ys5bb737j82xgyiciyzpcwmzyi9b2-bootstrap-tools.tar.xz' to '/nix/store/jml0gh0q2rnc9sgr87irz0jfbl0fq545-fm8ys5bb737j82xgyiciyzpcwmzyi9b2-bootstrap-tools.tar.xz' (hash 'sha256-ZY9IMOmx1VOn6uoFDpdJbTnPX59TEkrVCzWNtjQ8/QE=').
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
Downloaded 'file:///nix/store/rn9fx55mw2s0qg90zdjxa9xn11nlmjpg-busybox' to '/nix/store/9qylz8gqll63pprwkwlyfs9g4zilak2m-rn9fx55mw2s0qg90zdjxa9xn11nlmjpg-busybox' (hash 'sha256-WuOaun7U5enbOy8SuuCo6G1fbGwsO16jhy/oM8K0lAs=').
I used the hashes from the output above to create the fetchurl
invocation which is part of this commit.
I then started the bootstrap with the following command:
nix --extra-experimental-features nix-command build -L -f . --arg localSystem '(import ./lib).systems.examples.aarch64-multiplatform-musl' hello
As @lovesegfault requested, here are the the sha256sums of all the
on-server components for extra verification:
sha256sum /nix/store/${STOREPATH}/on-server/*
…which produced the following output:
658f4830e9b1d553a7eaea050e97496d39cf5f9f53124ad50b358db6343cfd01 /nix/store/95lm0y33dayag4542s8bi83s31bw68dr-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
20cdfecb084ddb6b6b958f2b78fd2cc1d9641632f81ec7d5a48fae0a963ad0fa /nix/store/95lm0y33dayag4542s8bi83s31bw68dr-stdenv-bootstrap-tools/on-server/busybox
This PR updates the bootstrap tarballs for riscv64-linux with new Hydra-generated ones.
Fixes#275848 (bootstrap assembler too old to build gcc 13).
I'll be following the script used in #151399, #168199, #183487, and #188334.
Files came from [this](https://hydra.nixos.org/build/246376732#tabs-summary) Hydra build, which used nixpkgs revision 160cedc144 to instantiate:
```
/nix/store/cpiajh4l83b08pynwiwkpxj53d78pcxr-stdenv-bootstrap-tools-riscv64-unknown-linux-gnu.drv
```
and then built:
```
/nix/store/8a92pj40awdw585mcb9dvm4nyb03k3q3-stdenv-bootstrap-tools-riscv64-unknown-linux-gnu
```
I downloaded these files from Hydra and prefetched them into the nix store with the following commands:
```
STOREPATH=8a92pj40awdw585mcb9dvm4nyb03k3q3-stdenv-bootstrap-tools-riscv64-unknown-linux-gnu
OPTIONS="--option binary-caches https://cache.nixos.org --option trusted-public-keys cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
nix store prefetch-file \
file://$(nix store add-file --name bootstrap-tools.tar.xz $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/bootstrap-tools.tar.xz)
nix store prefetch-file --executable \
file://$(nix store add-path --name busybox $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/busybox)
```
These commands produced the following output:
```
Downloaded 'file:///nix/store/xs74wcxq3qc12plfc70fds8inbndhcfm-bootstrap-tools.tar.xz' to '/nix/store/3fal4gikp92013kac6rdmfbrch2s859b-xs74wcxq3qc12plfc70fds8inbndhcfm-bootstrap-tools.tar.xz' (hash 'sha256-0LxRd7fdafQezNJ+N2tuOfm0KEwgfRSts5fhP0e0r0s=').
Downloaded 'file:///nix/store/9ndpna6jrlac4y9fappdjm0sxx0g2bja-busybox' to '/nix/store/kb7wyy30y1gxcmdajljr26kxxac606qa-9ndpna6jrlac4y9fappdjm0sxx0g2bja-busybox' (hash 'sha256-OGO96QUzs2n5pGipn/V87AxzUY9OWKZl417nE8HdZIE=').
```
I used the hashes from the output above to create the `fetchurl` invocation which is part of this commit.
I then started the bootstrap with the following command:
```
nix build -L -f . --arg localSystem '(import ./lib).systems.examples.riscv64' hello
```
As @lovesegfault requested, here are the the `sha256sum`s of all the `on-server` components for extra verification:
```
sha256sum /nix/store/${STOREPATH}/on-server/*
```
which produced the following output:
```
d0bc5177b7dd69f41eccd27e376b6e39f9b4284c207d14adb397e13f47b4af4b /nix/store/8a92pj40awdw585mcb9dvm4nyb03k3q3-stdenv-bootstrap-tools-riscv64-unknown-linux-gnu/on-server/bootstrap-tools.tar.xz
65f9433abb598f63c932d33351b14f686551512b1cece1e64c2d0e76aa0ec52e /nix/store/8a92pj40awdw585mcb9dvm4nyb03k3q3-stdenv-bootstrap-tools-riscv64-unknown-linux-gnu/on-server/busybox
```
stdenv.cc may throw, e.g. in the case of pkgsCross.ghcjs where we must
not force it for the purpose of attribute accessing (`or` doesn't
implicitly tryEval…).
Regression introduced in 1a5bd697ad.
Before the change `pkgsLLVM` attributes were failing to pull in
`compiler-rt` on `x86_64-linux`:
$ nix build --no-link -f. pkgsLLVM.asciidoc-full
error:
error: attribute 'llvmPackages_13' missing
at pkgs/stdenv/adapters.nix:86:32:
85| inherit libcxx;
86| extraPackages = [ cxxabi pkgs.pkgsTargetTarget."llvmPackages_${lib.versions.major llvmLibcxxVersion}".compiler-rt ];
| ^
87| });
It happens because `pkgs.pkgsTargetTarget` are always empty for
cross-packages like `pkgsLLVM.`, `pkgsCross.*.` or
`--arg crossSystem '...'`.
Before the change `pkgsMusl.adobe-reader` was failing the interpreter:
$ nix-instantiate --eval --strict --expr 'with import ./. {}; builtins.tryEval pkgsMusl.adobe-reader'
error:
error: evaluation aborted with the following error message: 'unsupported platform for the pure Linux stdenv'
After the change `pkgsMusl.adobe-reader` returns catchable excation:
$ nix-instantiate --eval --strict --expr 'with import ./. {}; builtins.tryEval pkgsMusl.adobe-reader'
{ success = false; value = false; }
Noticed when was exploring `nixpkgs` for uncatchable evaluation errors.
Ideally those should only happen when there is a code bug in the use
site. In this case it's just a package with incompatible constraints for
`musl`.
Changed uncatchable `abort` to `throw`.
Aka `checkMeta` goes brrr.
Using the module system type checking works OK & generates good error messages.
The performance of using it however is terrible because of the value merging it does being very allocation heavy.
By implementing a very minimal type checker we can drastically improve the performance when nixpkgs is evaluated with `checkMeta = true`.
Without the change bootstrapTools build fails as:
https://cache.nixos.org/log/g5wyq9xqshan6m3kl21bjn1z88hx48rh-stdenv-bootstrap-tools.drv
error: install_name_tool: changing install names or rpaths can't be redone for: /nix/store/0hxg356h7lnl2hck23wrdpbib3cckx41-stdenv-bootstrap-tools/bin/tac (for architecture x86_64) because larger updated load commands do not fit (the program must be relinked, and you may need to use -headerpad or -headerpad_max_install_names)
this makes it a lot easier to create a modified stdenv with a
different set of defaultHardeningFlags and as a bonus allows us
to inject the correct defaultHardeningFlags into toolchain wrapper
scripts, reducing repetition.
while most hardening flags are arguably more of a compiler thing,
it works better to put them in bintools-wrapper because cc-wrapper
can easily refer to bintools but not vice-versa.
mkDerivation can still easily refer to either when it is constructed.
this also switches fortran-hook.sh to use the same defaults for
NIX_HARDENING_ENABLE as for C. previously NIX_HARDENING_ENABLE
defaults were apparently used to avoid passing problematic flags
to a fortran compiler, but this falls apart as soon as mkDerivation
sets its own NIX_HARDENING_ENABLE - cc.hardeningUnsupportedFlags
is a more appropriate mechanism for this as it actively filters
out flags from being used by the wrapper, so switch to using that
instead.
this is still an imperfect mechanism because it doesn't handle a
compiler which has both langFortran *and* langC very well - applying
the superset of the two's hardeningUnsupportedFlags to either
compiler's invocation. however this is nothing new - cc-wrapper
already poorly handles a langFortran+langC compiler, applying two
setup hooks that have contradictory options.
If a CMake target has a non-default LINKER_LANGUAGE set, CMake will
manually add the libraries it has detected that language's compiler as
linking implicitly. When it does this, it'll pass -Bstatic and
-Bdynamic options based on the vibes it gets from each such detected
library. This in itself isn't a problem, because the compiler
toolchain, or our wrapper, or something, seems to be smart enough to
ignore -Bdynamic for those libraries. But it does create a problem if
the compiler adds extra libraries to the linker command line after
that final -Bdynamic, because those will be linked dynamically. Since
our compiler is static by default, CMake should reset to -Bstatic
after it's done manually specifying libraries, but CMake didn't
actually know that our compiler is static by default. The fix for
that is to tell it, like so.
Until recently, this problem was difficult to notice, because it would
result binaries that worked, but that were dynamically linked. Since
e08ce498f0 ("cc-wrapper: Account for NIX_LDFLAGS and NIX_CFLAGS_LINK
in linkType"), though, -Wl,-dynamic-linker is no longer mistakenly
passed for executables that are supposed to be static, so they end up
created with a /lib interpreter path, and so don't run at all on
NixOS.
This fixes pkgsStatic.graphite2.
As reported in #241692, since the `llvmPackages` bump the
bootstrap-tools started failing to build due to a mismatch in LLVM
versions used to build certain tools.
By overlaying the imported package set to specify `llvmPackages`, we get
everything built with the expected LLVM version.
Provide a `runPhase` function which wraps the phase running action of
genericBuild. The new function can be used as an interface by `nix
develop`, i.e. `nix develop some#flake --build` may just call `runPhase
build`, which makes its behavior more consistent with `nix build`.
In preparation of fixing https://github.com/NixOS/nix/issues/6202
The 10.12 Libsystem is not located as a sub-attribute of
`darwin.apple_sdk_10_12`. This will be fixed as part of the SDK changes
planned for post-23.11. In the meantime, special case it so the adapter
can be used to change the deployment target.
This was taken from #264091 to use in the interim before that PR lands
(sometime after the release of 23.11). It allows different versions of
clang to link the same libc++, allowing dependencies to be linked when
they are built with a different version of clang than the stdenv.
This patch switches the CoreFoundation on x86_64-darwin from the open
source swift-corelibs-foundation (CF) to the system CoreFoundation.
This change was motivated by failures building packages for the current
staging-next cycle #263535 due to an apparent incompatibility with the
rpath-based approach to choosing CF or CoreFoundation and macOS 14. This
error often manifests as a crash with an Illegal Instruction.
For example, building aws-sdk-cpp for building Nix will fail this way.
https://hydra.nixos.org/build/239459417/nixlog/1
Application Specific Information:
CF objects must have a non-zero isa
Error Formulating Crash Report:
PC register does not match crashing frame (0x0 vs 0x7FF8094DD640)
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 CoreFoundation 0x7ff8094dd640 CF_IS_OBJC.cold.1 + 14
1 CoreFoundation 0x7ff8094501d0 CF_IS_OBJC + 60
2 CoreFoundation 0x7ff8093155e8 CFRelease + 40
3 ??? 0x10c7a2c61 s_aws_secure_transport_ctx_destroy + 65
4 ??? 0x10c87ba32 aws_ref_count_release + 34
5 ??? 0x10c7b7adb aws_tls_connection_options_clean_up + 27
6 ??? 0x10c596db4 Aws::Crt::Io::TlsConnectionOptions::~TlsConnectionOptions() + 20
7 ??? 0x10c2d249c Aws::CleanupCrt() + 92
8 ??? 0x10c2d1ff0 Aws::ShutdownAPI(Aws::SDKOptions const&) + 64
9 ??? 0x102d9bc6f main + 335
10 dyld 0x202f333a6 start + 1942
According to a [post][1] on the Apple developer forums, hardening was
added to CoreFoundation, and this particular message occurs when you
attempt to release an object it does not recognize as a valid CF object.
(Thank you to @lilyinstarlight for finding this post).
When I switched aws-sdk-cpp to link against CoreFoundation instead of
CF, the error went away. Somehow both libraries were being used. To
prevent dependent packages from linking the wrong CoreFoundation, it
would need to be added as a propagated build input.
Note that there are other issues related to mixing CF and CoreFoundation
frameworks. #264503 fixes an issue with abseil-cpp where it propagates
CF, causing issues when using a different SDK version. Mixing versions
can also cause crashes with Python when a shared object is loaded that
is linked to the “wrong” CoreFoundation.
`NIX_COREFOUNDATION_RPATH` is supposed to make sure the right
CoreFoundation is being used, but it does not appear to be enough on
macOS 14 (presumably due to the hardening). While it is possible to
propagate CoreFoundation manually, the cleaner solution is to make it
the default. CF remains available as `darwin.swift-corelibs-foundation`.
[1]: https://developer.apple.com/forums/thread/739355
- These new-cli commands can be used with `-f`, in which case they're
evaluated with pure evaluation disabled.
- Nix 2.4+ is not part of the condition; "flakes" is fully descriptive
and more relatable.
- Don't suggest that it only enables this variable.
- Just don't say too much.
This is a replacement for using `darwin.apple_sdk_<ver>.callPackage`.
Instead of injecting the required packages, it provides a stdenv adapter
that modifies the derivation’s build inputs to use the requested SDK
versions. This modification extends to any build inputs propagated to it
as well. The `callPackage` approach is not deprecated yet, but it is
expected that it will be eventually.
Note that this is an MVP. It should work with most packages, but it only
handles build inputs and also only handles frameworks. Once more SDKs
are added (after #229210 is merged) and the SDK structure is normalized,
it can be extended to handle any package in the SDK namespace.
Cross-compilation may or may not work. Any cross-related issues can be
addressed after #256590 is merged.
While there is no fetcher or builder (in nixpkgs) that takes an `md5` parameter,
for some inscrutable reason the nix interpreter accepts the following:
```nix
fetchurl {
url = "https://www.perdu.com";
hash = "md5-rrdBU2a35b2PM2ZO+n/zGw==";
}
```
Note that neither MD5 nor SHA1 are allowed by the syntax of SRI hashes.
curl needs to link against several frameworks, but building the
frameworks (directly or indirectly) depends on curl via fetchurl and
fetchFromGitHub. Break the infinite recursion by building the SDKs’
dependencies in the last stage of the stdenv bootstrap using the prior
stage’s fetchurl and fetchFromGitHub.
Make both ready for cross with prefixes
Currently
`pkgsCross.aarch64-multiplatform.mold.passthru.tests.{wrapped,adapter}`
fail with
```
Testing running the 'hello' binary which should be linked with 'mold'
Hello, world!
Checking for mold in the '.comment' section
No mention of 'mold' detected in the '.comment' section
The command was:
aarch64-unknown-linux-gnu-readelf -p .comment ...bin/hello
The output was:
String dump of section '.comment':
[ 0] GCC: (GNU) 12.3.0
```
This will allow buliding bootstrap tools for platforms with
non-default libcs, like *-unknown-linux-musl.
This gets rid of limitedSupportSystems/systemsWithAnySupport. There
was no need to use systemsWithAnySupport for supportDarwin, because it
was always equivalent to supportedSystems for that purpose, and the
only other way it was used was for determining which platforms to
build the bootstrap tools for, so we might as well use a more explicit
parameter for that, and then we can change how it works without
affecting the rest of the Hydra jobs.
Not affecting the rest of the Hydra jobs is important, because if we
changed all jobs to use config triples, we'd end up renaming every
Hydra job. That might still be worth thinking about at some point,
but it's unnecessary at this point (and would be a lot of work).
I've checked by running
nix-eval-jobs --force-recurse pkgs/top-level/release.nix
that the actual bootstrap tools derivations are unaffected by this
change, and that the only other jobs that change are ones that depend
on the hash of all of Nixpkgs. Of the other jobset entrypoints that
end up importing pkgs/top-level/release.nix, none used the
limitedSupportedSystems parameter, so they should all be unaffected as
well.
When specifying the `builder` attribute in `stdenv.mkDerivation`, this
will be effectively transformed into
builtins.derivation {
builder = stdenv.shell;
args = [ "-e" builder ];
}
This also means that `default-builder.sh` is never sourced and as a
result it's not guaranteed that `$NIX_ATTRS_SH_FILE` is set to a correct
location[1].
Also, we need to source `.attrs.sh` to source `$stdenv`. So, the
following is done now:
* If `$NIX_ATTRS_SH_FILE` points to a correct location, then use it.
Directly using `.attrs.sh` is problematic for `nix-shell(1)` usage
(see previous commit for more context), so prefer the environment
variable if possible.
* Otherwise, if `.attrs.sh` exists, then use it. See [1] for when this
can happen.
* If neither applies, it can be assumed that `__structuredAttrs` is
turned off and thus nothing needs to be done.
[1] It's possible that it doesn't exist at all - in case of Nix 2.3 or
it can point to a wrong location on older Nix versions with a bug in
`__structuredAttrs`.
Relying on `.attrs.sh` to exist in `$NIX_BUILD_TOP` is problematic
because that's not compatible with how `nix-shell(1)` behaves. It places
`.attrs.{json,sh}` into a temporary directory and makes them accessible via
`$NIX_ATTRS_{SH,JSON}_FILE` in the environment[1]. The sole reason that
`nix-shell(1)` still works with structured-attrs enabled derivations
is that the contents of `.attrs.sh` are sourced into the
shell before sourcing `$stdenv/setup` (if `$stdenv` exists) by `nix-shell`.
However, the assumption that two files called `.attrs.sh` and
`.attrs.json` exist in `$NIX_BUILD_TOP` is wrong in an interactive shell
session and thus an inconsistency between shell debug session and actual
builds which can lead to unexpected problems.
To be precise, we currently have the following problem: an expression
like
with import ./. {};
runCommand "foo" { __structuredAttrs = true; foo.bar = [ 1 2 3 ]; }
''
echo "''${__structuredAttrs@Q}"
touch $out
''
prints `1` in its build-log. However when building interactively in a
`nix-shell`, it doesn't.
Because of that, I'm considering to propose a full deprecation of
`$NIX_BUILD_TOP/.attrs.{json,sh}`. A first step is to only mention the
environment variables, but not the actual paths anymore in Nix's
manual[2]. The second step - this patch - is to fix nixpkgs' stdenv
accordingly.
Please note that we cannot check for `-e "$NIX_ATTRS_JSON_FILE"` because
certain outdated Nix minors (that are still in the range of supported
Nix versions in `nixpkgs`) have a bug where `NIX_ATTRS_JSON_FILE` points
to the wrong file while building[3].
Also, for compatibility with Nix 2.3 which doesn't provide these
environment variables at all we still need to check for the existence of
.attrs.json/.attrs.sh here. As soon as we bump nixpkgs' minver to 2.4,
this can be dropped.
Finally, dropped the check for ATTRS_SH_FILE because that was never
relevant. In nix#4770 the ATTRS_SH_FILE variable was introduced[4] and
in a review iteration prefixed with NIX_[5]. In other words, these
variables were never part of a release and you'd only have this problem
if you'd use a Nix from a git revision of my branch from back then. In
other words, that's dead code.
[1] https://github.com/nixos/nix/pull/4770#issuecomment-834718851
[2] https://github.com/NixOS/nix/pull/9032
[3] https://github.com/NixOS/nix/issues/6736
[4] 3944a120ec
[5] 27ce722638
Without this, you get error messages during the install phase along the
lines of: "file RPATH_CHANGE could not write new RPATH:".
This is unsurprising because the static binaries do not have any dynamic
linker and thus, no runpath to rewrite either.
Tell cmake it doesn't need to do RPATH manipulation by passing
cmakeFlags.
While we're here, I also renamed `finalAttrs` to `args` and fixed the
indentation; this improves consistency with the surrounding code and
eliminates a point of confusion: because it was named `finalAttrs` I
presumed I should be able to influence it with an overrideAttrs setting
dontAddStaticConfigureFlags, but this turns out not to be possible;
adding prevAttrs as well doesn't work because of a limitation of
overrideAttrs whereby it gives an infinite recursion if the set of
attribute keys being returned depends on finalAttrs.
Signed-off-by: Peter Waller <p@pwaller.net>
Fixes `pkgsCross.musl64.llvmPackages_16.clang.cc` on `x86_64-linux`,
which used to fail with `/bin/sh: clang-tblgen: not found`.
Same hack is used in other projects:
https://github.com/search?q=%2FCMAKE_CROSSCOMPILING_EMULATOR.%2B%5C%2Fusr%5C%2Fbin%5C%2Fenv%2F+NOT+is%3Afork&type=code
Comment from 30435a9d0f/build/cmake/HostLinuxToolchain.cmake (L64)
> Required to run host Linux executables during the build itself.
> An example would be https://gitub.com/KhronosGroup/Vulkan-Loader and
> its "asm_offset" program.
>
> NOTE: Alternatives have been tried unsuccessfully, i.e.:
>
> With $(set CMAKE_CROSSCOMPILING_EMULATOR), the build fails because
> the CMake ninja/Make script tries to find the executable in the current
> path, as in:
>
> [3/16] Generating gen_defines.asm
> FAILED: loader/gen_defines.asm
> cd /tmp/cc/build-Vulkan-Loader/loader && asm_offset GAS
> /bin/sh: asm_offset: command not found
> ninja: build stopped: subcommand failed.
>
> With $(set CMAKE_CROSSCOMPILING_EMULATOR ""), the build fails because
> the shell cannot find the "" program as in:
>
> [3/16] Generating gen_defines.asm
> FAILED: loader/gen_defines.asm
> cd /tmp/cc/build-Vulkan-Loader/loader && "" /tmp/cc/build-Vulkan-Loader/loader/asm_offset GAS
> /bin/sh: : command not found
> ninja: build stopped: subcommand failed.
>
> It seems that the root of the problem comes from how the CMake function
> cmCustomCommandGenerator::GetArgc0Location() computes the target
> executable's location. At this point it's unclear whether this is a CMake
> bug or a feature.
Risicle discovered this hack.
Co-authored-by: Robert Scott <code@humanleg.org.uk>
In the default `fixupPhase` the output of `substituteAllStream` is
streamed to setup-hook.
`stdenv.cc.bintools.overrideAttrs { NIX_DEBUG = 6; }`
With `NIX_DEBUG` contains:
```
@expandResponseParams@ -> /nix/store/yl01rd58vp4m8bbhkihpk132cprfmx6f-expand-response-params/bin/expand-response-params
...
```
To work around intermitent build failures with clang 16, the stdenv
attempted to pass arguments on the command-line on newer versions of
macOS. Unfortunately, the larger `ARG_MAX` is still not large enough to
build qtwebengine. This commit reverts the `NIX_CC_NO_RESPONSE_FILE`
logic in the stdenv. The changes to cc-wrapper in #245282 are needed for
clang 16 to prevent the above-mentioned build failures.
This fixes pyicu (and any other package that uses `icu-config` instead
of the CMake or some other module to get the build flags).
What happened here is the bootstrap disables `patchShebangs` to avoid
propagating the bootstrap tools to the final stdenv (due to `sh` and
`bash` being on the `PATH` from the bootstrap tools). Because of that,
the `#!/bin/sh` line in `icu-config` was not updated, causing it to
invoke the system bash on Darwin. While that is undesirable in its own
right, when the system bash is invoked as `sh`, `echo -n` will print
`-n`, resulting in the breakage see in https://github.com/NixOS/nixpkgs/pull/241951#issuecomment-1627604354.
The fix is to build bash earlier in the bootstrap while making sure it
is picked up over the one in the bootstrap tools. That allows
`patchShebangs` to be enabled during the bootstrap. Any package with
scripts that is included in the final stdenv should now have its
scripts’ shebang lines properly patched.
When sandboxing is enabled, the hook tries to run `install_name_tool`
and fails because the system one is inaccessible. Having it use
`targetPrefix` allows it to find and use the cross-install_name_tool.
```
nix-repl> (pkgs.htop.overrideAttrs { pname = "hello-overriden"; }).pname
error:
… while evaluating a branch condition
at /nix/store/phn5cahwacv9wjgalygw62x8l4xbl6x3-source/lib/customisation.nix:86:7:
85| in
86| if builtins.isAttrs result then
| ^
87| result // {
… while calling the 'isAttrs' builtin
at /nix/store/phn5cahwacv9wjgalygw62x8l4xbl6x3-source/lib/customisation.nix:86:10:
85| in
86| if builtins.isAttrs result then
| ^
87| result // {
(stack trace truncated; use '--show-trace' to show the full trace)
error: attempt to call something which is not a function but a set
at /nix/store/phn5cahwacv9wjgalygw62x8l4xbl6x3-source/pkgs/stdenv/generic/make-derivation.nix:58:21:
57| f = self: super:
58| let x = f0 super;
| ^
59| in
```
swift-corelibs uses libcurl to implement `NSURLSession` in Foundation
via the symbols exported by CF. Foundation is not build on Darwin, and
these symbols are not exported by the system CoreFoundation.
By not linking against libcurl, this breaks a cycle between CF and
libcurl. That should allow libcurl to drop the patch disabling
linking against the SystemConfiguration and restore NAT64 support.
Unfortunately, the Darwin stdenv bootstrap still needs to build
dependencies that use `fetchFromGitHub`. While it can drop curl from the
final stdenv, it still needs to use it during the stdenv bootstrap.
In preparation for bumping the LLVM used by Darwin, this change
refactors and reworks the stdenv build process. When it made sense,
existing behaviors were kept to avoid causing any unwanted breakage.
However, there are some differences. The reasoning and differences are
discussed below.
- Improved cycle times - Working on the Darwin stdenv was a tedious
process because `allowedRequisites` determined what was allowed
between stages. If you made a mistake, you might have to wait a
considerable amount of time for the build to fail. Using assertions
makes many errors fail at evaluation time and makes moving things
around safer and easier to do.
- Decoupling from bootstrap tools - The stdenv build process builds as
much as it can in the early stages to remove the requirement that the
bootstrap tools need bumped in order to bump the stdenv itself. This
should lower the barrier to updates and make it easier to bump in the
future. It also allows changes to be made without requiring additional
tools be added to the bootstrap tools.
- Patterned after the Linux stdenv - I tried to follow the patterns
established in the Linux stdenv with adaptations made to Darwin’s
needs. My hope is this makes the Darwin stdenv more approable for
non-Darwin developers who made need to interact with it. It also
allowed some of the hacks to be removed.
- Documentation - Comments were added explaining what was happening and
why things were being done. This is particular important for some
stages that might not be obvious (such as the sysctl stage).
- Cleanup - Converting the intermediate `allowedRequisites` to
assertions revealed that many packages were being referenced that no
longer exist or have been renamed. Removing them reduces clutter and
should help make the stdenv bootstrap process be more understandable.
