Commit Graph

2663 Commits

Author SHA1 Message Date
annalee
3cdd8d05f0
stdenv.darwin.make-boostrap-tools: include unpack script in the archive 2024-03-19 17:28:32 +00:00
annalee
b310830359
stdenv.darwin.make-boostrap-tools: update tools and unpin LLVM11
- unpin LLVM11. fix discrepancy with freshBootstrapTools and the tools
  built on hydra. pinning the stdenv for the hydra build doesn't pin the
  tools as the included packages are able to change.

- remove unused LLVM tools & libs which reduces the uncompressed and
  compressed file sizes by more than 1/2. compressed tarball is now 40M
  and uncompressed is around 200M

- add @loader_path/. to dylibs that reference other libs in the archive.
  this is needed for libraries with re-exports.

- validate shared objects with @rpath references contain the reference
  in lib

- add a test to verify that the @loader_path/ works for libc++ as it
  re-exports libc++abi
2024-03-19 17:28:32 +00:00
annalee
88f192b247
stdenv.darwin.make-boostrap-tools: refactor to use getBin, getDev and getLib 2024-03-19 17:28:32 +00:00
annalee
7c4b2a599c
freshBootstrapTools: LLVM11 unpin 2024-03-19 14:54:23 +00:00
binarycat
f9094faf70 stdenv: pass --mode=+w to tar, ensuring extraction is possible
fixes #191137
2024-03-18 20:54:31 -04:00
github-actions[bot]
ac91cf83c1
Merge master into staging-next 2024-03-18 00:02:20 +00:00
Randy Eckenrode
07451425e7
darwin.stdenv: drop curl from the bootstrap
The Darwin bootstrap currently requires curl, but it is not strictly
required. The bootstrap requires it for two things:

* Fetchers; and
* As a transitive dependency of llvm-manpages (via Sphinx).

For the fetchers, the bootstrap curl can be used. For hatch-vcs, the
dependency, its tests can be disabled. Doing this allows curl to be
dropped from the Darwin stdenv bootstrap.
2024-03-17 16:48:35 -04:00
Robert Hensing
05f4b7b46d make-derivation.nix: Fix checkMetaRecursively
Oddly, I can't reproduce the error, but this change will make it
more robust.
See https://github.com/NixOS/nixpkgs/pull/295378#issuecomment-2002094487
2024-03-17 19:28:47 +01:00
github-actions[bot]
c160165e60
Merge master into staging-next 2024-03-14 18:01:26 +00:00
Robert Hensing
ba463e70e3 make-derivation.nix: Float out unsafeDerivationToUntrackedOutpath 2024-03-12 20:15:47 +01:00
Robert Hensing
0ab7b23637 make-derivation.nix: Update inline docs 2024-03-12 19:06:07 +01:00
Robert Hensing
797ad5ae87 make-derivation.nix: Float out new constant removedOrReplacedAttrNames 2024-03-12 18:43:11 +01:00
Robert Hensing
cac23248b1 make-derivation.nix: Float out knownHardeningFlags 2024-03-12 18:41:57 +01:00
Robert Hensing
464d8c4a6c make-derivation.nix: Float inward darwin-specific derivation attributes 2024-03-12 18:41:18 +01:00
Robert Hensing
2fb7255cc0 make-derivation.nix: Evaluate flip 2024-03-12 18:39:19 +01:00
Robert Hensing
90197b6833 make-derivation.nix: Apply map composition law
... after inlining chooseDevOutputs.
2024-03-12 18:37:53 +01:00
Robert Hensing
954d9ce64b make-derivation.nix: Drop duplicate functionality from previous commit
These two commits make for a cleaner commit history and git blame than
https://github.com/NixOS/nixpkgs/pull/295105, where this refactor
was developed.

See its commit messages for details and design choices, esp. up to
and including 37f76fd4c3.
2024-03-12 18:32:03 +01:00
Robert Hensing
8dad51a2e2 make-derivation.nix: Split makeDerivationArgument, mkDerivation with duplicate functionality 2024-03-12 18:16:56 +01:00
github-actions[bot]
06e5eb63d3
Merge staging-next into staging 2024-03-11 18:01:53 +00:00
Robert Hensing
de516f6f13 make-derivation.nix: Return mkDerivation as an attribute 2024-03-11 17:06:02 +01:00
Robert Hensing
ab56d3ac71 make-derivation.nix: Move into let binding 2024-03-11 17:02:59 +01:00
a-n-n-a-l-e-e
ce789e7e35
llvmPackages_{12,13,14,15,16,17,git}.{libcxx,libcxxabi}: merge libcxxabi into libcxx (#292043)
- merge libcxxabi into libcxx for LLVM 12, 13, 14, 15, 16, 17, and git.
- remove the link time workaround `-lc++ -lc++abi` from 58 packages as it is no longer required.
- fixes https://github.com/NixOS/nixpkgs/issues/166205
- provides alternative fixes for. https://github.com/NixOS/nixpkgs/issues/269548 https://github.com/NixOS/nix/issues/9640
- pkgsCross.x86_64-freebsd builds work again

This change can be represented in 3 stages
1. merge libcxxabi into libcxx -- files: pkgs/development/compilers/llvm/[12, git]/{libcxx, libcxxabi}
2. update stdenv to account for merge -- files: stdenv.{adapters, cc.wrapper, darwin}
3. remove all references to libcxxabi outside of llvm (about 58 packages modified)

### merging libcxxabi into libcxx
- take the union of the libcxxabi and libcxx cmake flags
- eliminate the libcxx-headers-only package - it was only needed to break libcxx <-> libcxxabi circular dependency
- libcxx.cxxabi is removed. external cxxabi (freebsd) will symlink headers / libs into libcxx.
- darwin will re-export the libcxxabi symbols into libcxx so linking `-lc++` is sufficient.
- linux/freebsd `libc++.so` is a linker script `LINK(libc++.so.1, -lc++abi)` making `-lc++` sufficient.
- libcxx/default.nix [12, 17] are identical except for patches and `LIBCXX_ADDITIONAL_LIBRARIES` (only used in 16+)
- git/libcxx/defaul.nix  does not link with -nostdlib when useLLVM is true so flag is removed. this is not much different than before as libcxxabi used -nostdlib where libcxx did not, so libc was linked in anyway.

### stdenv changes
- darwin bootstrap, remove references to libcxxabi and cxxabi
- cc-wrapper: remove c++ link workaround when libcxx.cxxabi doesn't exist (still exists for LLVM pre 12)
- adapter: update overrideLibcxx to account for a pkgs.stdenv that only has libcxx

### 58 package updates
- remove `NIX_LDFLAGS = "-l${stdenv.cc.libcxx.cxxabi.libName}` as no longer needed
- swift, nodejs_v8 remove libcxxabi references in the clang override

https://github.com/NixOS/nixpkgs/pull/292043
2024-03-11 03:53:37 -07:00
OPNA2608
5ec7dcd7c5 pkgs/stdenv/linux: init powerpc64-unknown-linux-gnuabielfv2 bootstrap-files
sha256sum of files to be uploaded:

$ sha256sum /nix/store/8frm8kk8gzpv31r289ai5jgkwfikmpm4-stdenv-bootstrap-tools-powerpc64-unknown-linux-gnuabielfv2/on-server/*
0af311476b54b399f3024b92e9c518363acd2b15f713e83d0bb4fb3f8f26d98b  /nix/store/8frm8kk8gzpv31r289ai5jgkwfikmpm4-stdenv-bootstrap-tools-powerpc64-unknown-linux-gnuabielfv2/on-server/bootstrap-tools.tar.xz
529a053a8022e89357aaa608e57aaddaa7c3ded93d633916ddca92bb81e22125  /nix/store/8frm8kk8gzpv31r289ai5jgkwfikmpm4-stdenv-bootstrap-tools-powerpc64-unknown-linux-gnuabielfv2/on-server/busybox

Suggested commands to upload files to 'tarballs.nixos.org':

    $ nix-store --realize /nix/store/8frm8kk8gzpv31r289ai5jgkwfikmpm4-stdenv-bootstrap-tools-powerpc64-unknown-linux-gnuabielfv2
    $ aws s3 cp --recursive --acl public-read /nix/store/8frm8kk8gzpv31r289ai5jgkwfikmpm4-stdenv-bootstrap-tools-powerpc64-unknown-linux-gnuabielfv2/on-server/ s3://nixpkgs-tarballs/stdenv/powerpc64-unknown-linux-gnuabielfv2/57cf2e0b24fb52344cc718913eaed78f389b1319
    $ aws s3 cp --recursive s3://nixpkgs-tarballs/stdenv/powerpc64-unknown-linux-gnuabielfv2/57cf2e0b24fb52344cc718913eaed78f389b1319 ./
    $ sha256sum bootstrap-tools.tar.xz busybox
    $ sha256sum /nix/store/8frm8kk8gzpv31r289ai5jgkwfikmpm4-stdenv-bootstrap-tools-powerpc64-unknown-linux-gnuabielfv2/on-server/*
2024-03-10 12:36:34 +01:00
a-n-n-a-l-e-e
cf1a332500
Revert "stdenv: log build hooks as they run" 2024-03-09 17:32:20 -08:00
github-actions[bot]
c3cf93646a
Merge staging-next into staging 2024-03-09 00:02:40 +00:00
Weijia Wang
172fde7e27
Merge pull request #293257 from OPNA2608/add/make-bootstrap-tools-cross-powerpc64
pkgs/stdenv/linux/make-bootstrap-tools-cross: Add powerpc64 entry
2024-03-08 21:00:53 +01:00
github-actions[bot]
a51832837a
Merge staging-next into staging 2024-03-04 18:01:24 +00:00
github-actions[bot]
fe2479b166
Merge master into staging-next 2024-03-04 18:00:55 +00:00
OPNA2608
5f8522eeab pkgs/stdenv/linux/make-bootstrap-tools-cross: Add powerpc64 entry
So Hydra can cross-compile bootstrapFiles for the platform.
2024-03-04 15:01:29 +01:00
a-n-n-a-l-e-e
d7425f0f21
Merge pull request #288250 from trofi/make-bootstrap-tools-cross-new-target
pkgs/stdenv/linux/make-bootstrap-tools-cross.nix: allow entries for t…
2024-03-04 05:18:39 -08:00
Ryan Lahfa
241ca50541
Merge pull request #290081 from Qyriad/diag/log-hooks
stdenv: log build hooks as they run
2024-03-04 00:17:07 +01:00
Qyriad
ff372db9b4
prev: per-review: reword comment regarding logging of implicit hooks
Addresses the following review comment: https://github.com/NixOS/nixpkgs/pull/290081#discussion_r1503503065

This will be squashed into the previous commit pending review.

Co-authored-by: Philip Taron <philip.taron@gmail.com>
2024-02-26 19:59:01 -07:00
Qyriad
9aef09586e prev: apply fixes from code review
Addresses the following review comments:
- eb28e5e72e (r1501466125)
- eb28e5e72e (r1501466232)

This will be squashed into the previous commit pending review.
2024-02-25 22:19:37 -07:00
Robert Scott
4a91b3e798 cc-wrapper: add trivialautovarinit hardening flag support
this equates to -ftrivial-auto-var-init=pattern

clang has removed support for -ftrivial-auto-var-init=zero and
are unlikely to re-add it, so use -ftrivial-auto-var-init=pattern
on both compilers if only to make behaviour more consistent
between the two.

add to pkgsExtraHardening's defaultHardeningFlags.
2024-02-24 12:00:09 +00:00
Qyriad
eb28e5e72e stdenv: log build hooks as they run 2024-02-17 23:15:09 -07:00
Vladimír Čunát
e81960770a
Merge #288866: pkgs/stdenv/linux: update x86_64-unknown-linux-gnu bootstrap-files
...into staging
2024-02-17 10:38:31 +01:00
Vladimír Čunát
eff11adc57
Merge #286721: stdenv: fix rare tar.xz decompression issues
...into staging
2024-02-17 08:45:01 +01:00
Sergei Trofimovich
a0f80e965c pkgs/stdenv/linux/make-bootstrap-tools-cross.nix: allow entries for to-be uplaoded targets
Newly added targets should be fine to be added as hudra jobs before the
first uplaod happens.

Co-authored-by: Cosima Neidahl <opna2608@protonmail.com>
2024-02-15 09:08:58 +00:00
Isidor Zeuner
11a19109b6 stdenv: disregard xz exit status in order to fix subtle decompression issues
There is a subtle bug with unpacking `tar.xz` archives which seems to happen only on some setups, and sometimes not in a reproducible manner (https://github.com/NixOS/nixpkgs/issues/278130, https://github.com/NixOS/nixpkgs/issues/20950). On the last occurrence, it could be tracked down to `xz` failing from a `SIGPIPE`, which can happen when it's connected to `tar` through a pipe and `tar` exits earlier (see e.g. https://www.linuxquestions.org/questions/slackware-14/%5Bpatch%5D-tar-issuing-a-sigpipe-in-installpkg-4175637923/ or https://bugs.gentoo.org/573642#c5).

Since `tar` should be able by itself to detect whether the archive is complete, I suggest to disregard the exit code from the `xz` invocation, done in this PR.

Fixes  https://github.com/NixOS/nixpkgs/issues/278130 (script tested here: https://github.com/NixOS/nixpkgs/pull/286579)
Probably also fixes https://github.com/NixOS/nixpkgs/issues/20950 (issue not reproduced here, feedback therefore welcome)
2024-02-13 17:00:48 +01:00
Sergei Trofimovich
74c4e93541 pkgs/stdenv/linux: update x86_64-unknown-linux-gnu bootstrap-files
sha256sum of files to be uploaded:

$ sha256sum /nix/store/dw6vr6m5w7ysrdrbs0s5wdgbjmbnr7gx-stdenv-bootstrap-tools/on-server/*
61096bd3cf073e8556054da3a4f86920cc8eca81036580f0d72eb448619b50cd  /nix/store/dw6vr6m5w7ysrdrbs0s5wdgbjmbnr7gx-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
87627883b1517087d1c34fadc340bc8deafb3782eb0c60035271822c64dd3401  /nix/store/dw6vr6m5w7ysrdrbs0s5wdgbjmbnr7gx-stdenv-bootstrap-tools/on-server/busybox

Suggested commands to upload files to 'tarballs.nixos.org':

    $ nix-store --realize /nix/store/dw6vr6m5w7ysrdrbs0s5wdgbjmbnr7gx-stdenv-bootstrap-tools
    $ aws s3 cp --recursive --acl public-read /nix/store/dw6vr6m5w7ysrdrbs0s5wdgbjmbnr7gx-stdenv-bootstrap-tools/on-server/ s3://nixpkgs-tarballs/stdenv/x86_64-unknown-linux-gnu/82b583ba2ba2e5706b35dbe23f31362e62be2a9d
    $ aws s3 cp --recursive s3://nixpkgs-tarballs/stdenv/x86_64-unknown-linux-gnu/82b583ba2ba2e5706b35dbe23f31362e62be2a9d ./
    $ sha256sum bootstrap-tools.tar.xz busybox
    $ sha256sum /nix/store/dw6vr6m5w7ysrdrbs0s5wdgbjmbnr7gx-stdenv-bootstrap-tools/on-server/*
2024-02-13 09:10:00 +00:00
Kait Lam
355ab764b4 stdenv: refactor of --replace-{quiet,warn,fail} logic
This is a small simplification of the control flow surrounding these cases. It should make it more obvious when each case happens, and also explicitly defines the current behaviour of --replace.
2024-02-11 10:48:37 +10:00
Rick van Schijndel
2c98eaf597
Merge pull request #286526 from pbsds/fix-replace-quiet-1707148527
stdenv: fix `substituteStream --replace-quiet` deprecation warning
2024-02-07 21:35:19 +01:00
Bernardo Meurer
d73b17633c
Merge pull request #285906 from trofi/update-x86_64-unknown-linux-musl-bootstrap-files
pkgs/stdenv/linux: update x86_64-unknown-linux-musl bootstrap-files
2024-02-05 14:18:03 -05:00
Peder Bergebakken Sundt
cb4c41f93a stdenv: fix substituteStream --replace-quiet deprecation warning 2024-02-05 17:28:02 +01:00
Sergei Trofimovich
5de19bf709 pkgs/stdenv/linux: update x86_64-unknown-linux-musl bootstrap-files
sha256sum of files to be uploaded:

$ sha256sum /nix/store/b0x0qcbf1gsp50jzw52sbbgdp3jlwcjf-stdenv-bootstrap-tools/on-server/*
b745b6311ed4c2d3f260419c468f543ae5df68fe2e5192995c499819c04eb8e0  /nix/store/b0x0qcbf1gsp50jzw52sbbgdp3jlwcjf-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
f4ae6a814d5053b396af788cd47629385cbfa17df5197ff3c33de62dc1dd9116  /nix/store/b0x0qcbf1gsp50jzw52sbbgdp3jlwcjf-stdenv-bootstrap-tools/on-server/busybox

Suggested commands to upload files to 'tarballs.nixos.org':

    $ nix-store --realize /nix/store/b0x0qcbf1gsp50jzw52sbbgdp3jlwcjf-stdenv-bootstrap-tools
    $ aws s3 cp --recursive --acl public-read /nix/store/b0x0qcbf1gsp50jzw52sbbgdp3jlwcjf-stdenv-bootstrap-tools/on-server/ s3://nixpkgs-tarballs/stdenv/x86_64-unknown-linux-musl/125cefd4cf8f857e5ff1aceaef9230ba578a033d
    $ aws s3 cp --recursive s3://nixpkgs-tarballs/stdenv/x86_64-unknown-linux-musl/125cefd4cf8f857e5ff1aceaef9230ba578a033d ./
    $ sha256sum bootstrap-tools.tar.xz busybox
    $ sha256sum /nix/store/b0x0qcbf1gsp50jzw52sbbgdp3jlwcjf-stdenv-bootstrap-tools/on-server/*
2024-02-02 20:32:44 +00:00
github-actions[bot]
2f68b418f1
Merge staging-next into staging 2024-01-30 06:01:22 +00:00
Bernardo Meurer
f228f378e3
Merge pull request #284542 from trofi/i686-linux-bootstrap-update 2024-01-29 21:21:40 -05:00
Sergei Trofimovich
43695b8db4 pkgs/stdenv/darwin: move bootstrap files definitions to bootstrap-files/ directory
The change moves definition of bootstrap files slightly closer to
`linux` structure to eventually allow those to update in bulk:
https://github.com/NixOS/nixpkgs/issues/253713
2024-01-28 22:41:09 +00:00
Sergei Trofimovich
46dae33dd9 pkgs/stdenv/linux: update i686-unknown-linux-gnu bootstrap-files
sha256sum of files to be uploaded:

$ sha256sum /nix/store/5x6dldhza7if5s6wsicaxa8fbndyixps-stdenv-bootstrap-tools/on-server/*
293021dedf7568932230effb3453a3533e9f5c8f48bbe1fb72e383ade5b3f4df  /nix/store/5x6dldhza7if5s6wsicaxa8fbndyixps-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
d5bffc3231c9f95699ee34118e26bc788e09c608735e5edb2b433057011fddd1  /nix/store/5x6dldhza7if5s6wsicaxa8fbndyixps-stdenv-bootstrap-tools/on-server/busybox

Suggested commands to upload files to 'tarballs.nixos.org':

    $ nix-store --realize /nix/store/5x6dldhza7if5s6wsicaxa8fbndyixps-stdenv-bootstrap-tools
    $ aws s3 cp --recursive --acl public-read /nix/store/5x6dldhza7if5s6wsicaxa8fbndyixps-stdenv-bootstrap-tools/on-server/ s3://nixpkgs-tarballs/stdenv/i686-unknown-linux-gnu/125cefd4cf8f857e5ff1aceaef9230ba578a033d
    $ aws s3 cp --recursive s3://nixpkgs-tarballs/stdenv/i686-unknown-linux-gnu/125cefd4cf8f857e5ff1aceaef9230ba578a033d ./
    $ sha256sum bootstrap-tools.tar.xz busybox
    $ sha256sum /nix/store/5x6dldhza7if5s6wsicaxa8fbndyixps-stdenv-bootstrap-tools/on-server/*
2024-01-28 14:49:11 +00:00
Weijia Wang
4354538139 Merge branch 'staging-next' into staging 2024-01-27 04:17:36 +01:00
Alyssa Ross
ea67e455d3 stdenvBootstrapTools: update aarch64 musl
The aarch64 musl bootstrap tools are woefully outdated.  Just getting
them to the point of being able to build new versions of themselves
required a number of hacks[1] that can be reverted once we have new
bootstrap tools, and before that it stdenv didn't even build for the
preceding three years.

[1]: https://github.com/NixOS/nixpkgs/pull/169764

So, following the script established by previous bootstrap tools
updates:

Files came from this Hydra build:

	https://hydra.nixos.org/build/246470544

…which used nixpkgs revision dd5621df6d
to instantiate:

	/nix/store/g480ass2vjmakaq03z7k2j95xnxh206a-stdenv-bootstrap-tools.drv

…and then built:

	/nix/store/95lm0y33dayag4542s8bi83s31bw68dr-stdenv-bootstrap-tools

I downloaded these files from Hydra and prefetched them into the Nix
store with the following commands:

	STOREPATH=95lm0y33dayag4542s8bi83s31bw68dr-stdenv-bootstrap-tools
	OPTIONS="--option binary-caches https://cache.nixos.org --option trusted-public-keys cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
	nix --extra-experimental-features nix-command store prefetch-file \
	  file://$(nix --extra-experimental-features nix-command store add-file --name bootstrap-tools.tar.xz  $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/bootstrap-tools.tar.xz)
	nix --extra-experimental-features nix-command store prefetch-file --executable \
	  file://$(nix --extra-experimental-features nix-command store add-path --name busybox $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/busybox)

These commands produced the following output:

	warning: you did not specify '--add-root'; the result might be removed by the garbage collector
	Downloaded 'file:///nix/store/fm8ys5bb737j82xgyiciyzpcwmzyi9b2-bootstrap-tools.tar.xz' to '/nix/store/jml0gh0q2rnc9sgr87irz0jfbl0fq545-fm8ys5bb737j82xgyiciyzpcwmzyi9b2-bootstrap-tools.tar.xz' (hash 'sha256-ZY9IMOmx1VOn6uoFDpdJbTnPX59TEkrVCzWNtjQ8/QE=').
	warning: you did not specify '--add-root'; the result might be removed by the garbage collector
	Downloaded 'file:///nix/store/rn9fx55mw2s0qg90zdjxa9xn11nlmjpg-busybox' to '/nix/store/9qylz8gqll63pprwkwlyfs9g4zilak2m-rn9fx55mw2s0qg90zdjxa9xn11nlmjpg-busybox' (hash 'sha256-WuOaun7U5enbOy8SuuCo6G1fbGwsO16jhy/oM8K0lAs=').

I used the hashes from the output above to create the fetchurl
invocation which is part of this commit.

I then started the bootstrap with the following command:

	nix --extra-experimental-features nix-command build -L -f . --arg localSystem '(import ./lib).systems.examples.aarch64-multiplatform-musl' hello

As @lovesegfault requested, here are the the sha256sums of all the
on-server components for extra verification:

	sha256sum /nix/store/${STOREPATH}/on-server/*

…which produced the following output:

	658f4830e9b1d553a7eaea050e97496d39cf5f9f53124ad50b358db6343cfd01  /nix/store/95lm0y33dayag4542s8bi83s31bw68dr-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
	20cdfecb084ddb6b6b958f2b78fd2cc1d9641632f81ec7d5a48fae0a963ad0fa  /nix/store/95lm0y33dayag4542s8bi83s31bw68dr-stdenv-bootstrap-tools/on-server/busybox
2024-01-25 18:13:54 +01:00
github-actions[bot]
7af142cf2c
Merge staging-next into staging 2024-01-22 00:02:41 +00:00
github-actions[bot]
dceddd03df
Merge master into staging-next 2024-01-22 00:02:14 +00:00
Francesco Gazzetta
0d7c62f5c9 pkgs/stdenv/linux: update riscv64 bootstrap-files
This PR updates the bootstrap tarballs for riscv64-linux with new Hydra-generated ones.

Fixes #275848 (bootstrap assembler too old to build gcc 13).

I'll be following the script used in #151399, #168199, #183487, and #188334.

Files came from [this](https://hydra.nixos.org/build/246376732#tabs-summary) Hydra build, which used nixpkgs revision 160cedc144 to instantiate:

```
/nix/store/cpiajh4l83b08pynwiwkpxj53d78pcxr-stdenv-bootstrap-tools-riscv64-unknown-linux-gnu.drv
```

and then built:

```
/nix/store/8a92pj40awdw585mcb9dvm4nyb03k3q3-stdenv-bootstrap-tools-riscv64-unknown-linux-gnu
```

I downloaded these files from Hydra and prefetched them into the nix store with the following commands:

```
STOREPATH=8a92pj40awdw585mcb9dvm4nyb03k3q3-stdenv-bootstrap-tools-riscv64-unknown-linux-gnu
OPTIONS="--option binary-caches https://cache.nixos.org --option trusted-public-keys cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
nix store prefetch-file \
  file://$(nix store add-file --name bootstrap-tools.tar.xz  $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/bootstrap-tools.tar.xz)
nix store prefetch-file --executable \
  file://$(nix store add-path --name busybox $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/busybox)
```

These commands produced the following output:

```
Downloaded 'file:///nix/store/xs74wcxq3qc12plfc70fds8inbndhcfm-bootstrap-tools.tar.xz' to '/nix/store/3fal4gikp92013kac6rdmfbrch2s859b-xs74wcxq3qc12plfc70fds8inbndhcfm-bootstrap-tools.tar.xz' (hash 'sha256-0LxRd7fdafQezNJ+N2tuOfm0KEwgfRSts5fhP0e0r0s=').
Downloaded 'file:///nix/store/9ndpna6jrlac4y9fappdjm0sxx0g2bja-busybox' to '/nix/store/kb7wyy30y1gxcmdajljr26kxxac606qa-9ndpna6jrlac4y9fappdjm0sxx0g2bja-busybox' (hash 'sha256-OGO96QUzs2n5pGipn/V87AxzUY9OWKZl417nE8HdZIE=').
```

I used the hashes from the output above to create the `fetchurl` invocation which is part of this commit.

I then started the bootstrap with the following command:

```
nix build -L -f . --arg localSystem '(import ./lib).systems.examples.riscv64' hello
```

As @lovesegfault requested, here are the the `sha256sum`s of all the `on-server` components for extra verification:

```
sha256sum /nix/store/${STOREPATH}/on-server/*
```

which produced the following output:

```
d0bc5177b7dd69f41eccd27e376b6e39f9b4284c207d14adb397e13f47b4af4b  /nix/store/8a92pj40awdw585mcb9dvm4nyb03k3q3-stdenv-bootstrap-tools-riscv64-unknown-linux-gnu/on-server/bootstrap-tools.tar.xz
65f9433abb598f63c932d33351b14f686551512b1cece1e64c2d0e76aa0ec52e  /nix/store/8a92pj40awdw585mcb9dvm4nyb03k3q3-stdenv-bootstrap-tools-riscv64-unknown-linux-gnu/on-server/busybox
```
2024-01-20 16:27:22 +01:00
Robert Scott
40868719b0 cc-wrapper: add zerocallusedregs hardening flag
this uses the value `used-gpr` which seems to be a commonly
chosen value for general use
2024-01-20 13:48:33 +00:00
Peder Bergebakken Sundt
4c1b74bac7 stdenv: substituteStream: escape echoed pattern in --replace mismatch warning 2024-01-14 22:08:03 +01:00
Peder Bergebakken Sundt
e07a2fab7f stdenv: substituteStream: deprecate --replace in favor of --replace-{fail,warn,quiet} 2024-01-14 22:07:58 +01:00
sternenseemann
f734056102 stdenv: fix evaluation if !stdenv.cc.hasCC
stdenv.cc may throw, e.g. in the case of pkgsCross.ghcjs where we must
not force it for the purpose of attribute accessing (`or` doesn't
implicitly tryEval…).

Regression introduced in 1a5bd697ad.
2024-01-13 12:45:36 +01:00
Someone Serge
e6e279913c
stdenvAdapters.useLibsFrom: init 2024-01-12 17:38:00 +00:00
Weijia Wang
2f9e98ccf3 Merge branch 'master' into staging-next 2024-01-10 23:12:44 +01:00
Weijia Wang
df865c667f
Merge pull request #279463 from trofi/stdenv.adapters-fix-overrideLibcxx-cross
stdenv.adapters: fix `overrideLibcxx` to refer to existing `compiler…
2024-01-10 19:49:00 +01:00
K900
bff9ee8862 Merge remote-tracking branch 'origin/master' into staging-next 2024-01-09 13:03:34 +03:00
Ryan Burns
c324705cc3 treewide: simplify exec format conditionals 2024-01-07 17:43:33 -08:00
Sergei Trofimovich
55249c817f stdenv.adapters: fixx overrideLibcxx to refer to existing compiler-rt
Before the change `pkgsLLVM` attributes were failing to pull in
`compiler-rt` on `x86_64-linux`:

    $ nix build --no-link -f. pkgsLLVM.asciidoc-full
    error:
       error: attribute 'llvmPackages_13' missing

       at pkgs/stdenv/adapters.nix:86:32:

           85|       inherit libcxx;
           86|       extraPackages = [ cxxabi pkgs.pkgsTargetTarget."llvmPackages_${lib.versions.major llvmLibcxxVersion}".compiler-rt ];
             |                                ^
           87|     });

It happens because `pkgs.pkgsTargetTarget` are always empty for
cross-packages like `pkgsLLVM.`, `pkgsCross.*.` or
`--arg crossSystem '...'`.
2024-01-07 22:11:49 +00:00
Weijia Wang
eb5334c27d Merge branch 'master' into staging-next 2024-01-03 03:22:40 +01:00
Robert Hensing
af62e8e2fb
Merge pull request #273935 from adisbladis/lib-meta-typechecks-go-brrr
stdenv/check-meta: Use bespoke type checking
2024-01-02 14:40:50 +01:00
Vladimír Čunát
de84adb292
Merge branch 'master' into staging-next 2023-12-29 14:00:30 +01:00
Sergei Trofimovich
3f3e74bc93 stdenv: fix eval of pkgsMusl packages with platform constraints
Before the change `pkgsMusl.adobe-reader` was failing the interpreter:

    $ nix-instantiate --eval --strict --expr 'with import ./. {}; builtins.tryEval pkgsMusl.adobe-reader'
error:
       error: evaluation aborted with the following error message: 'unsupported platform for the pure Linux stdenv'

After the change `pkgsMusl.adobe-reader` returns catchable excation:

    $ nix-instantiate --eval --strict --expr 'with import ./. {}; builtins.tryEval pkgsMusl.adobe-reader'
    { success = false; value = false; }

Noticed when was exploring `nixpkgs` for uncatchable evaluation errors.
Ideally those should only happen when there is a code bug in the use
site. In this case it's just a package with incompatible constraints for
`musl`.

Changed uncatchable `abort` to `throw`.
2023-12-28 15:29:29 +00:00
adisbladis
4a5f2bd6e8 stdenv/check-meta: Use bespoke type checking
Aka `checkMeta` goes brrr.

Using the module system type checking works OK & generates good error messages.
The performance of using it however is terrible because of the value merging it does being very allocation heavy.

By implementing a very minimal type checker we can drastically improve the performance when nixpkgs is evaluated with `checkMeta = true`.
2023-12-26 15:14:42 +13:00
github-actions[bot]
bae9ba4ab0
Merge staging-next into staging 2023-12-14 00:02:44 +00:00
Sergei Trofimovich
fccb48062f darwin/make-bootstrap-tools.nix: set -headerpad_max_install_names for coreutils
Without the change bootstrapTools build fails as:
https://cache.nixos.org/log/g5wyq9xqshan6m3kl21bjn1z88hx48rh-stdenv-bootstrap-tools.drv

    error: install_name_tool: changing install names or rpaths can't be redone for: /nix/store/0hxg356h7lnl2hck23wrdpbib3cckx41-stdenv-bootstrap-tools/bin/tac (for architecture x86_64) because larger updated load commands do not fit (the program must be relinked, and you may need to use -headerpad or -headerpad_max_install_names)
2023-12-13 17:57:25 +00:00
Martin Weinelt
e8dd1cf066 Merge remote-tracking branch 'origin/staging-next' into staging 2023-12-12 14:13:07 +01:00
adisbladis
1e66093cca
Merge pull request #269546 from adisbladis/stdenv-meta-no-intermediate-alloc
stdenv: Avoid allocating intermediate attrset when checking meta validity
2023-12-12 10:57:44 +13:00
adisbladis
6aa414bcfd
Merge pull request #269782 from adisbladis/stdenv-meta-tolist-license-allocations
stdenv: Avoid some list allocations in check-meta when checking licenses
2023-12-12 10:55:58 +13:00
adisbladis
9a0a097a94 stdenv: Avoid allocating intermediate attrset when checking meta validity
This is a small performance optimization. It should be impercetible to most.

Benchmarks:

- Before
``` json
{
  "cpuTime": 0.2777960002422333,
  "envs": {
    "bytes": 3832648,
    "elements": 189513,
    "number": 144784
  },
  "gc": {
    "heapSize": 402915328,
    "totalBytes": 50229344
  },
  "list": {
    "bytes": 655304,
    "concats": 3249,
    "elements": 81913
  },
  "nrAvoided": 218962,
  "nrFunctionCalls": 127718,
  "nrLookups": 40946,
  "nrOpUpdateValuesCopied": 1563978,
  "nrOpUpdates": 8542,
  "nrPrimOpCalls": 113032,
  "nrThunks": 329605,
  "sets": {
    "bytes": 29774864,
    "elements": 1824537,
    "number": 36392
  },
  "sizes": {
    "Attr": 16,
    "Bindings": 16,
    "Env": 16,
    "Value": 24
  },
  "symbols": {
    "bytes": 235909,
    "number": 24432
  },
  "values": {
    "bytes": 9691392,
    "number": 403808
  }
}
```

- After
```
{
  "cpuTime": 0.2615779936313629,
  "envs": {
    "bytes": 3833832,
    "elements": 189661,
    "number": 144784
  },
  "gc": {
    "heapSize": 402915328,
    "totalBytes": 50212960
  },
  "list": {
    "bytes": 655304,
    "concats": 3249,
    "elements": 81913
  },
  "nrAvoided": 218814,
  "nrFunctionCalls": 127718,
  "nrLookups": 40798,
  "nrOpUpdateValuesCopied": 1563978,
  "nrOpUpdates": 8542,
  "nrPrimOpCalls": 113032,
  "nrThunks": 329457,
  "sets": {
    "bytes": 29765392,
    "elements": 1824093,
    "number": 36244
  },
  "sizes": {
    "Attr": 16,
    "Bindings": 16,
    "Env": 16,
    "Value": 24
  },
  "symbols": {
    "bytes": 235909,
    "number": 24432
  },
  "values": {
    "bytes": 9687840,
    "number": 403660
  }
}
```
2023-12-12 00:08:34 +13:00
adisbladis
3b13bd5c84 stdenv: Avoid some list allocations in check-meta when checking licenses 2023-12-12 00:03:20 +13:00
Robert Scott
dc2247a3b5 stdenvAdapters: add withDefaultHardeningFlags 2023-12-09 16:32:15 +00:00
Robert Scott
1a5bd697ad mkDerivation, bintools-wrapper: move defaultHardeningFlags determination to bintools-wrapper
this makes it a lot easier to create a modified stdenv with a
different set of defaultHardeningFlags and as a bonus allows us
to inject the correct defaultHardeningFlags into toolchain wrapper
scripts, reducing repetition.

while most hardening flags are arguably more of a compiler thing,
it works better to put them in bintools-wrapper because cc-wrapper
can easily refer to bintools but not vice-versa.

mkDerivation can still easily refer to either when it is constructed.

this also switches fortran-hook.sh to use the same defaults for
NIX_HARDENING_ENABLE as for C. previously NIX_HARDENING_ENABLE
defaults were apparently used to avoid passing problematic flags
to a fortran compiler, but this falls apart as soon as mkDerivation
sets its own NIX_HARDENING_ENABLE - cc.hardeningUnsupportedFlags
is a more appropriate mechanism for this as it actively filters
out flags from being used by the wrapper, so switch to using that
instead.

this is still an imperfect mechanism because it doesn't handle a
compiler which has both langFortran *and* langC very well - applying
the superset of the two's hardeningUnsupportedFlags to either
compiler's invocation. however this is nothing new - cc-wrapper
already poorly handles a langFortran+langC compiler, applying two
setup hooks that have contradictory options.
2023-12-09 16:30:45 +00:00
Dmitry Kalinkin
025a278148 Merge remote-tracking branch 'origin/master' into staging
Conflicts:
	pkgs/tools/networking/ofono/default.nix
2023-12-08 12:27:28 -05:00
Martin Weinelt
570af7d909
Merge pull request #263446 from emilylange/stdenv-xz-multithread
stdenv: enable multithreading for `xz` decompression
2023-12-06 15:41:01 +01:00
github-actions[bot]
035a649a67
Merge master into staging-next 2023-12-06 06:01:30 +00:00
Rick van Schijndel
ba75066821
Merge pull request #271362 from pbsds/rm-dead-patches
treewide: remove unreferenced patch files
2023-12-06 03:28:15 +01:00
Vladimír Čunát
305cd929fd
Merge branch 'staging' into staging-next 2023-12-01 07:33:49 +01:00
Peder Bergebakken Sundt
f41aba3739 treewide: remove unreferenced patch files
Found with `fd \\.patch$ pkgs/ -x bash -c 'rg -F "{/}" pkgs/ -q || echo {}'`
2023-12-01 06:11:20 +01:00
Alyssa Ross
0d44b5b773 pkgsStatic.stdenv: fix custom CMake LINKER_LANGUAGE
If a CMake target has a non-default LINKER_LANGUAGE set, CMake will
manually add the libraries it has detected that language's compiler as
linking implicitly.  When it does this, it'll pass -Bstatic and
-Bdynamic options based on the vibes it gets from each such detected
library.  This in itself isn't a problem, because the compiler
toolchain, or our wrapper, or something, seems to be smart enough to
ignore -Bdynamic for those libraries.  But it does create a problem if
the compiler adds extra libraries to the linker command line after
that final -Bdynamic, because those will be linked dynamically.  Since
our compiler is static by default, CMake should reset to -Bstatic
after it's done manually specifying libraries, but CMake didn't
actually know that our compiler is static by default.  The fix for
that is to tell it, like so.

Until recently, this problem was difficult to notice, because it would
result binaries that worked, but that were dynamically linked.  Since
e08ce498f0 ("cc-wrapper: Account for NIX_LDFLAGS and NIX_CFLAGS_LINK
in linkType"), though, -Wl,-dynamic-linker is no longer mistakenly
passed for executables that are supposed to be static, so they end up
created with a /lib interpreter path, and so don't run at all on
NixOS.

This fixes pkgsStatic.graphite2.
2023-11-30 10:46:44 +00:00
github-actions[bot]
e019e7cdb1
Merge staging-next into staging 2023-11-29 12:01:23 +00:00
piegames
cb47f7e108
Merge pull request #267309: check-meta: Improve performance 2023-11-29 12:46:12 +01:00
Artturi
4b27e14f8c
Merge pull request #265738 from Artturin/patchshebangcon
stdenv: run patchShebangs on the configure script when it's a file
2023-11-26 01:28:41 +02:00
Artturin
e68001550d stdenv: run patchShebangs on the configure script when it's a file
if the configure script has a `/usr/bin/env` or some other shebang which
is not in the sandbox then there will be errors such as

`...-stdenv-linux/setup: line 1299: ./configure: cannot execute: required file not found`

There are 250 files which `patchShebangs` `./configure`

https://github.com/search?q=NOT+is%3Afork+lang%3Anix+%2FpatchShebangs+.%5C%2Fconfigure%2F&type=code
2023-11-25 18:16:12 +02:00
github-actions[bot]
991a07725b
Merge staging-next into staging 2023-11-17 12:01:30 +00:00
K900
b40035f472
Merge pull request #267058 from toonn/bootstrap-tools-specify-llvm
freshBootstrapTools: Overlay the package set with the desired LLVM
2023-11-17 14:06:37 +03:00
Adam Joseph
985ff1cb88
Update pkgs/stdenv/darwin/make-bootstrap-tools.nix 2023-11-17 08:26:41 +00:00
Ivan Mincik
3b4b805561 stdenv: consistent phases header
Make phases header consistent for all phases.

`Running phase:` is from an old nix ux doc from 2020

https://github.com/tweag/nix-ux/blob/master/first_steps_with_nix_v2.md

Co-authored-by: Artturin <Artturin@artturin.com>
2023-11-17 03:11:48 +02:00
Artturin
249c6d04b1 check-meta: don't use rec
possible performance improvement
2023-11-14 16:00:42 +02:00
Artturin
0527676bd9 check-meta: don't use with 2023-11-13 23:47:25 +02:00
Artturin
215951e4c8 check-meta: Improve performance
See c3c31aa798
2023-11-13 23:43:03 +02:00
Weijia Wang
74d102ce50 Merge branch 'master' into staging-next 2023-11-12 19:45:20 +01:00
toonn
887d33a089 freshBootstrapTools: Overlay the package set with the desired LLVM
As reported in #241692, since the `llvmPackages` bump the
bootstrap-tools started failing to build due to a mismatch in LLVM
versions used to build certain tools.

By overlaying the imported package set to specify `llvmPackages`, we get
everything built with the expected LLVM version.
2023-11-12 15:08:11 +01:00
Artturin
c3c31aa798 stdenv: Improve performance
| stat                   | before          | after           | Δ               | Δ%      |
|------------------------|-----------------|-----------------|-----------------|---------|
| cpuTime                | 513.67          | 507.77          | ↘ 5.90          | -1.15%  |
| envs-bytes             | 20,682,847,968  | 20,628,961,616  | ↘ 53,886,352    | -0.26%  |
| envs-elements          | 1,054,735,104   | 1,051,395,620   | ↘ 3,339,484     | -0.32%  |
| envs-number            | 765,310,446     | 763,612,291     | ↘ 1,698,155     | -0.22%  |
| gc-heapSize            | 53,439,602,688  | 51,711,545,344  | ↘ 1,728,057,344 | -3.23%  |
| gc-totalBytes          | 113,062,066,672 | 112,139,998,240 | ↘ 922,068,432   | -0.82%  |
| list-bytes             | 3,118,249,784   | 3,118,249,784   | 0               |         |
| list-concats           | 52,834,140      | 52,834,140      | 0               |         |
| list-elements          | 389,781,223     | 389,781,223     | 0               |         |
| nrAvoided              | 968,097,988     | 991,889,795     | ↗ 23,791,807    | 2.46%   |
| nrFunctionCalls        | 697,259,792     | 697,259,792     | 0               |         |
| nrLookups              | 510,257,062     | 338,275,331     | ↘ 171,981,731   | -33.70% |
| nrOpUpdateValuesCopied | 1,446,690,216   | 1,446,690,216   | 0               |         |
| nrOpUpdates            | 68,504,034      | 68,504,034      | 0               |         |
| nrPrimOpCalls          | 429,464,805     | 429,464,805     | 0               |         |
| nrThunks               | 1,009,240,391   | 982,109,100     | ↘ 27,131,291    | -2.69%  |
| sets-bytes             | 33,524,722,928  | 33,524,722,928  | 0               |         |
| sets-elements          | 1,938,309,212   | 1,938,309,212   | 0               |         |
| sets-number            | 156,985,971     | 156,985,971     | 0               |         |
| sizes-Attr             | 16              | 16              | 0               |         |
| sizes-Bindings         | 16              | 16              | 0               |         |
| sizes-Env              | 16              | 16              | 0               |         |
| sizes-Value            | 24              | 24              | 0               |         |
| symbols-bytes          | 2,151,298       | 2,151,298       | 0               |         |
| symbols-number         | 159,707         | 159,707         | 0               |         |
| values-bytes           | 30,218,194,248  | 29,567,043,264  | ↘ 651,150,984   | -2.15%  |
| values-number          | 1,259,091,427   | 1,231,960,136   | ↘ 27,131,291    | -2.15%  |

> Accessing the lexical scope directly should be more efficient, yes, because it changes from a binary search (many lookups) to just two memory accesses
> correction: one short linked list + one array access
> oh and you had to do the lexical scope lookup anyway for lib itself
> so it really does save a binary search at basically no extra cost

- roberth

after seeing the stats

> Oooh nice. I did not consider that more of the maybeThunk optimization becomes effective (nrAvoided). Those lookups also caused allocations!

- roberth

Left `lib.generators` and `lib.strings` alone because they're only used
once.
2023-11-12 10:42:49 +02:00
Randy Eckenrode
940180423a
Merge pull request #265102 from reckenrode/darwin-corefoundation
darwin.stdenv: use CoreFoundation instead of CF
2023-11-11 12:10:10 -05:00