Commit Graph

259 Commits

Author SHA1 Message Date
Emily
514b00cf08 clang: skip the -nostdlibinc patch on Darwin 2024-10-24 17:55:27 +01:00
Randy Eckenrode
840a4f7771
darwin.stdenv: propagate bmake from stage 1 (for darwin.locale) 2024-10-11 17:58:38 -04:00
Randy Eckenrode
1d030c0ad7
darwin.stdenv: propagate darwin.locale as a no-CC package 2024-10-11 17:58:21 -04:00
Randy Eckenrode
092b50a92c
python3: fix stdenv bootstrap on x86_64-darwin
The bootstrap tools linker sometimes crashes when trying to link the
sqlite3 tests, which causes the bootstrap Python not to have the sqlite3
module. This causes the freezegun module to fail to build later in the
bootstrap. Using the 11.0 SDK fixes the problem.

Upstream Python supports building with a newer SDK and back-deploying,
so this change should not negatively affect users on pre-11.0 releases.
2024-10-10 16:23:09 -04:00
Randy Eckenrode
9c004ef367
darwin.stdenv: use xarMinimal
Using xarMinimal avoids pulling e2fsprogs and (eventually) macfuse-stubs
into the Darwin stdenv bootstrap.
2024-10-10 16:23:03 -04:00
Randy Eckenrode
76a5ef680e
darwin.stdenv: rework for the new SDK
While it would be nice if this could be split, there are too many
changes as part of the cleanup and improvements, including:

- Refactoring all propagated packages into functions that can be used to
  ensure that packages are propagated only at the expected stages;
- Using a sanity-checking merge function to ensure that packages are
  only propagated by one of the above functions;
- Reducing the number of Python builds during the bootstrap to one;
- Removing the extra sysctl stage;
- Using the LLVM bootstrap to build LLVM, clang, libc++, etc;
- Propagating llvmPackages_<version> in the final stdenv, so that
  packages needing that version specifically don’t have to rebuild it;
- Bootstrapping with the new Darwin SDK; and
- Reducing the overall number of paths build during a bootstrap by ~33%.
2024-10-10 16:23:03 -04:00
github-actions[bot]
b6b2ac4f61
Merge master into staging-next 2024-09-07 00:12:59 +00:00
John Ericson
f43cc75868
Clean up cross bootstrapping, take 2 (#321525) 2024-09-06 14:56:54 -04:00
John Ericson
5f134ec6cf Clean up cross bootstrapping
For a long time, we've had `crossLibcStdenv`, `*Cross` libc attributes,
and `*bsdCross` pre-libc package sets. This was always bad because
having "cross" things is "not declarative": the naming doesn't reflect
what packages *need* but rather how we *provide* something. This is
ugly, and creates needless friction between cross and native building.

Now, almost all of these `*Cross` attributes are gone: just these are
kept:

- Glibc's and Musl's are kept, because those packages are widely used
  and I didn't want to risk changing the native builds of those at this
  time.

- generic `libcCross`, `theadsCross`, and friends, because these relate
  to the convolulted GCC bootstrap which still needs to be redone.

The BSD and obscure Linux or freestnanding libcs have conversely all
been made to use a new `stdenvNoLibc`, which is like the old
`crossLibcStdenv` except:

1. It usable for native and cross alike

2. It named according to what it *is* ("a standard environment without
   libc but with a C compiler"), rather than some non-compositional
   jargon ("the stdenv used for building libc when cross compiling",
   yuck).

I should have done this change long ago, but I was stymied because of
"infinite recursions". The problem was that in too many cases we are
overriding `stdenv` to *remove* things we don't need, and this risks
cyles since those more minimal stdenvs are used to build things in the
more maximal stdenvs.

The solution is to pass `stage.nix` `stdenvNoCC`, so we can override to
*build up* rather than *tear down*. For now, the full `stdenv` is also
passed, so I don't need to change the native bootstraps, but I can see
this changing as we make things more uniform and clean those up.

(adapted from commit 51f1ecaa59)
(adapted from commit 1743662e55)
2024-09-06 10:26:56 -04:00
github-actions[bot]
74cdd9c9b1
Merge master into staging-next 2024-09-05 00:13:11 +00:00
w
d7ec977f86 stdenv: Allow user to supply their bootstrapFiles set of tools 2024-08-30 02:31:18 +00:00
Ivan Trubach
5eee6cf40a xar: 1.6.1 -> 498
This change switches the xar package from unmaintained fork of the
original project to the Apple Open Source tarball. See also
https://repology.org/project/xar/versions

Since the package is essentially rewritten from scratch, we take an
opportunity and move it to pkgs/by-name/xa/xar (formatted with nixfmt).

We also remove Windows from the supported platforms because even before
this change pkgsCross.mingwW64.xar failed with
xar> configure: error: can not detect the size of your system's uid_t type
2024-08-25 18:10:08 +03:00
Randy Eckenrode
2c5001b1db
darwin.stdenv: format with nixfmt-rfc-style 2024-08-11 23:11:04 -04:00
Emily
8a837af302
Merge pull request #326819 from risicle/ris-shadowstack
cc-wrapper: add support for `shadowstack` hardening flag
2024-07-28 19:07:52 +01:00
Randy Eckenrode
d5e2c40823
darwin.stdenv: remove darwin.ICU from the final overlay
Nothing in the stdenv needs it. Keeping it there pulls bootstrap tools into its build environment.
2024-07-22 20:54:22 -04:00
Randy Eckenrode
312e20a312
darwin.stdenv: fix scons Python override
This reduces the number of Python builds in the bootstrap to two: a minimal build and a normal build. Both have LTO disabled, which is required due to missing LLVM LTO libraries. This is necessary to correctly enable LTO builds in Python because it needs `llvm-ar` from `stdenv.cc.cc.libllvm`, which does not exist in the bootstrap.
2024-07-20 21:32:38 -04:00
Robert Scott
b207b6ef74 cc-wrapper: add support for shadowstack hardening flag 2024-07-14 21:25:47 +01:00
Randy Eckenrode
a49dbb2544
darwin.stdenv: switch to top-level cctools and ld64 2024-07-13 17:54:36 -04:00
Randy Eckenrode
9403fdc4a6
darwin.stdenv: consolidate stage 2 into one stage
Separate stages are no longer necessary because CF is not built from source and will not be built in the future.
2024-07-13 17:54:35 -04:00
Randy Eckenrode
05e5d7f73e
darwin.stdenv: clean up GNU binutils ban
- Only propagate binutils-unwrapped; and
- Clarify in the `throw` that it cannot be used in the Darwin bootstrap.
2024-07-13 17:54:35 -04:00
Randy Eckenrode
78da51cdb0
darwin.stdenv: adjust flags for llvm-strip in bootstrap tools 2024-07-13 17:54:35 -04:00
Randy Eckenrode
8559d6466b
darwin.stdenv: always use a response file
The bootstrap tools have been updated, so it is no longer necessary to suppress using a response file in the Darwin stdenv bootstrap.
2024-07-13 17:54:35 -04:00
Randy Eckenrode
65a37e7b14
darwin.stdenv: make sure curl cannot be used
Fetchers can use the `curl` binary from the bootstrap tools. Allowing packages in the Darwin bootstrap to link curl makes any curl update cause a full rebuild on Darwin, which is undesirable.
2024-07-13 17:54:35 -04:00
Tristan Ross
7a95a8948b
Merge pull request #324155 from paparodeo/x64-sdk11-no-rebuilds
treewide: change various flags to allow x64 darwin to default to sdk 11.0 when ready
2024-07-08 17:19:17 -07:00
Reno Dakota
3cb23cec23
treewide: fixes to allow x64 darwin to default to sdk 11
update code to not assume that x64 darwin must use sdk 10.12. After this
change it's possible to build a sdk 11 stdenv on darwin x64
2024-07-02 23:30:01 +00:00
Randy Eckenrode
c6e9b98725
darwin.stdenv: make sure GNU binutils cannot be used
GNU binutils is not preferred on Darwin, and newer versions have issues building. Make it an evaluation error to use it in the Darwin stdenv bootstrap.
2024-06-27 08:10:34 -04:00
Randy Eckenrode
2b21ece48b
darwin.stdenv: update for darwin.binutils cleanup
- Only link binaries that exist for stage 0 cctools and LLVM bintools;
- Drop cctools-llvm in favor of the updated darwin.binutils; and
- Update llvm-manages Python overrides (needed for newer versions of LLVM).
2024-06-27 08:10:34 -04:00
Randy Eckenrode
e1ab7a10d8
darwin.stdenv: run LLVM tests only once 2024-06-27 07:33:47 -04:00
Randy Eckenrode
bb96a168d7
darwin.stdenv: reduce number of times Python is built 2024-06-27 07:33:47 -04:00
Franz Pletz
3db93c351d cc-wrapper: add stack clash protection hardening flag
Most Linux distributions are enabling this these days and it does
protect against real world vulnerabilities as demonstrated by
CVE-2018-16864 and CVE-2018-16865.

Fix #53753.

Information on llvm version support gleaned from
6609892a2d
68e07da3e5
092507a730

Information on gcc version support a lot harder to gather,
but both 32bit and 64bit arm do appear to be supported
based on the test suite.
2024-06-07 20:23:46 +01:00
Randy Eckenrode
c88b49062a
libiconv: use libiconv-darwin 2024-04-27 18:09:53 -04:00
Randy Eckenrode
89c9b73ce7
darwin.stdenv: propagate atf and kyua
Avoid building these packages more than once. Even though they require
linking to dylibs, they’re only used for running tests.
2024-04-27 18:09:53 -04:00
Randy Eckenrode
ad38102a67
darwin.stdenv: avoid an infinite recursion
libiconv-darwin depends on Meson, which (indirectly) depends on
libiconv. When libiconv-darwin is set as libiconv, it will cause an
infinite recursion. Avoid the infinite recursion by using libiconvReal
in stage 1. Every stage after that can use libiconv-darwin.
2024-04-27 18:09:52 -04:00
Vladimír Čunát
8c0c955a09
Merge branch 'master' into staging-next 2024-04-19 07:52:28 +02:00
sternenseemann
7be562d046 wrapCC, wrapBintools: move expand-response-params bootstrapping out
The cc and bintools wrapper contained ad hoc bootstrapping logic for
expand-response-params (which was callPackage-ed in a let binding). This
lead to the strange situation that the bootstrapping logic related to
expand-response-params is split between the wrapper derivations (where
it is duplicated) and the actual stdenv bootstrapping.

To clean this up, the wrappers simply should take expand-response-params
as an ordinary input: They need an adjacent expand-response-params (i.e.
one that runs on their host platform), but don't care about the how.
Providing this is only problematic during stdenv bootstrapping where we
have to pull it from the previous stage at times.
2024-04-18 20:49:13 +02:00
sternenseemann
b2a568906a wrapCC, wrapBintools: use runtimeShell instead of stdenv shell
We don't need to artificially make sure that we can execute the wrapper
scripts on the build platform by using stdenv's shell (which comes from
buildPackages) since our cross infrastructure will get us the wrapper
from buildPackages. The upside of this change is that cross-compiled
wrappers (e.g. pkgsCross.aarch64-multiplatform.gcc) will actually work
when executed!

For bootstrapping this is also not a problem, since we have a long
build->build platform chain so runtimeShell is just as good as
stdenvNoCC.shell. We do fall back to old ways, though, by explicitly
using the bootstrap-tools shell in stage2, so the adjacent bash is only
used from stage4 onwards. This is unnecessary in principle (I'll try
removing this hack in the future), but ensures this change causes zero
rebuilds.
2024-04-18 20:49:13 +02:00
annalee
8e7f7b4fa0 stdenv.darwin: bootstrap darwin using updated tools
- update the hashes and tools needed to extract the bootstrap-tools archive
- unify the x64 and aarch64 unpack process
2024-04-07 18:33:42 +02:00
a-n-n-a-l-e-e
7ad496eb80
Merge pull request #300032 from NixOS/revert-295558-darwin-bootstrap-update
Revert "stdenv.darwin: bootstrap darwin using updated tools"
2024-03-29 11:05:07 -07:00
a-n-n-a-l-e-e
6b44d9d62d
Revert "stdenv.darwin: bootstrap darwin using updated tools" 2024-03-29 09:33:31 -07:00
Randy Eckenrode
71c6ee9295
darwin.stdenv: add sdkRoot to extraNativeBuildInputs
Setting the SDK root by default allows `overrideSDK` to correctly set
the SDK version when using a different SDK. It also allows the correct
SDK version to be set when using an older deployment target. Not setting
the correct SDK version can result in unexpected behavior at runtime.

Examples:

* Automatic dark mode switching requires linking against an SDK version
  of 10.14 or newer. With the current behavior, the only way to do this
  is by using a 10.14+ deployment target even when the application
  supports older platforms when build with a newer SDK.
* MetalD3D checks that the system version is at least 14.0. The API it
  uses returns a compatibility version when the the SDK is older than
  11.0, which causes it to display an error and terminate the
  application even when even when its requirements are all met.
2024-03-29 10:07:50 -04:00
Randy Eckenrode
1f231173a5
darwin.stdenv: drop curl from assertions 2024-03-29 10:00:30 -04:00
Randy Eckenrode
017eda24ae
darwin.stdenv: use wrapBintoolsWith instead of a direct import 2024-03-29 10:00:28 -04:00
annalee
f8ad6442d5
stdenv.darwin: bootstrap darwin using updated tools
- update the hashes and tools needed to extract the bootstrap-tools archive
- unify the x64 and aarch64 unpack process
- unpin libxml for python-minimal
2024-03-22 13:27:27 +00:00
Randy Eckenrode
07451425e7
darwin.stdenv: drop curl from the bootstrap
The Darwin bootstrap currently requires curl, but it is not strictly
required. The bootstrap requires it for two things:

* Fetchers; and
* As a transitive dependency of llvm-manpages (via Sphinx).

For the fetchers, the bootstrap curl can be used. For hatch-vcs, the
dependency, its tests can be disabled. Doing this allows curl to be
dropped from the Darwin stdenv bootstrap.
2024-03-17 16:48:35 -04:00
a-n-n-a-l-e-e
ce789e7e35
llvmPackages_{12,13,14,15,16,17,git}.{libcxx,libcxxabi}: merge libcxxabi into libcxx (#292043)
- merge libcxxabi into libcxx for LLVM 12, 13, 14, 15, 16, 17, and git.
- remove the link time workaround `-lc++ -lc++abi` from 58 packages as it is no longer required.
- fixes https://github.com/NixOS/nixpkgs/issues/166205
- provides alternative fixes for. https://github.com/NixOS/nixpkgs/issues/269548 https://github.com/NixOS/nix/issues/9640
- pkgsCross.x86_64-freebsd builds work again

This change can be represented in 3 stages
1. merge libcxxabi into libcxx -- files: pkgs/development/compilers/llvm/[12, git]/{libcxx, libcxxabi}
2. update stdenv to account for merge -- files: stdenv.{adapters, cc.wrapper, darwin}
3. remove all references to libcxxabi outside of llvm (about 58 packages modified)

### merging libcxxabi into libcxx
- take the union of the libcxxabi and libcxx cmake flags
- eliminate the libcxx-headers-only package - it was only needed to break libcxx <-> libcxxabi circular dependency
- libcxx.cxxabi is removed. external cxxabi (freebsd) will symlink headers / libs into libcxx.
- darwin will re-export the libcxxabi symbols into libcxx so linking `-lc++` is sufficient.
- linux/freebsd `libc++.so` is a linker script `LINK(libc++.so.1, -lc++abi)` making `-lc++` sufficient.
- libcxx/default.nix [12, 17] are identical except for patches and `LIBCXX_ADDITIONAL_LIBRARIES` (only used in 16+)
- git/libcxx/defaul.nix  does not link with -nostdlib when useLLVM is true so flag is removed. this is not much different than before as libcxxabi used -nostdlib where libcxx did not, so libc was linked in anyway.

### stdenv changes
- darwin bootstrap, remove references to libcxxabi and cxxabi
- cc-wrapper: remove c++ link workaround when libcxx.cxxabi doesn't exist (still exists for LLVM pre 12)
- adapter: update overrideLibcxx to account for a pkgs.stdenv that only has libcxx

### 58 package updates
- remove `NIX_LDFLAGS = "-l${stdenv.cc.libcxx.cxxabi.libName}` as no longer needed
- swift, nodejs_v8 remove libcxxabi references in the clang override

https://github.com/NixOS/nixpkgs/pull/292043
2024-03-11 03:53:37 -07:00
github-actions[bot]
2f68b418f1
Merge staging-next into staging 2024-01-30 06:01:22 +00:00
Sergei Trofimovich
43695b8db4 pkgs/stdenv/darwin: move bootstrap files definitions to bootstrap-files/ directory
The change moves definition of bootstrap files slightly closer to
`linux` structure to eventually allow those to update in bulk:
https://github.com/NixOS/nixpkgs/issues/253713
2024-01-28 22:41:09 +00:00
Robert Scott
40868719b0 cc-wrapper: add zerocallusedregs hardening flag
this uses the value `used-gpr` which seems to be a commonly
chosen value for general use
2024-01-20 13:48:33 +00:00
Randy Eckenrode
daa79a1d2d
darwin.stdenv: use CoreFoundation instead of CF
This patch switches the CoreFoundation on x86_64-darwin from the open
source swift-corelibs-foundation (CF) to the system CoreFoundation.

This change was motivated by failures building packages for the current
staging-next cycle #263535 due to an apparent incompatibility with the
rpath-based approach to choosing CF or CoreFoundation and macOS 14. This
error often manifests as a crash with an Illegal Instruction.

For example, building aws-sdk-cpp for building Nix will fail this way.

https://hydra.nixos.org/build/239459417/nixlog/1

    Application Specific Information:
    CF objects must have a non-zero isa

    Error Formulating Crash Report:
    PC register does not match crashing frame (0x0 vs 0x7FF8094DD640)

    Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
    0   CoreFoundation                	    0x7ff8094dd640 CF_IS_OBJC.cold.1 + 14
    1   CoreFoundation                	    0x7ff8094501d0 CF_IS_OBJC + 60
    2   CoreFoundation                	    0x7ff8093155e8 CFRelease + 40
    3   ???                           	       0x10c7a2c61 s_aws_secure_transport_ctx_destroy + 65
    4   ???                           	       0x10c87ba32 aws_ref_count_release + 34
    5   ???                           	       0x10c7b7adb aws_tls_connection_options_clean_up + 27
    6   ???                           	       0x10c596db4 Aws::Crt::Io::TlsConnectionOptions::~TlsConnectionOptions() + 20
    7   ???                           	       0x10c2d249c Aws::CleanupCrt() + 92
    8   ???                           	       0x10c2d1ff0 Aws::ShutdownAPI(Aws::SDKOptions const&) + 64
    9   ???                           	       0x102d9bc6f main + 335
    10  dyld                          	       0x202f333a6 start + 1942

According to a [post][1] on the Apple developer forums, hardening was
added to CoreFoundation, and this particular message occurs when you
attempt to release an object it does not recognize as a valid CF object.
(Thank you to @lilyinstarlight for finding this post).

When I switched aws-sdk-cpp to link against CoreFoundation instead of
CF, the error went away. Somehow both libraries were being used. To
prevent dependent packages from linking the wrong CoreFoundation, it
would need to be added as a propagated build input.

Note that there are other issues related to mixing CF and CoreFoundation
frameworks. #264503 fixes an issue with abseil-cpp where it propagates
CF, causing issues when using a different SDK version. Mixing versions
can also cause crashes with Python when a shared object is loaded that
is linked to the “wrong” CoreFoundation.

`NIX_COREFOUNDATION_RPATH` is supposed to make sure the right
CoreFoundation is being used, but it does not appear to be enough on
macOS 14 (presumably due to the hardening). While it is possible to
propagate CoreFoundation manually, the cleaner solution is to make it
the default. CF remains available as `darwin.swift-corelibs-foundation`.

[1]: https://developer.apple.com/forums/thread/739355
2023-11-02 21:20:55 -04:00
Randy Eckenrode
df14e86d9c
stdenv.darwin: fix infinite recursion after curl update
curl needs to link against several frameworks, but building the
frameworks (directly or indirectly) depends on curl via fetchurl and
fetchFromGitHub. Break the infinite recursion by building the SDKs’
dependencies in the last stage of the stdenv bootstrap using the prior
stage’s fetchurl and fetchFromGitHub.
2023-10-13 18:51:05 -04:00