Commit Graph

2523 Commits

Author SHA1 Message Date
github-actions[bot]
9dd2031c08
Merge staging-next into staging 2023-04-06 16:11:47 +00:00
Adam Joseph
3778e07994 stdenv: fix pkgsMusl
I broke `pkgsMusl` with #209870.

Something odd is happening with `xgcc` (the temporary compiler that
should be used only to compile `gcc`, although we are using it to
compile a temporary `patchelf` too) and `libstdc++`.

The temporary fix in this commit is to use `-static-libstdc++` for
the ephemeral `patchelf` built by `xgcc`.  It isn't pretty, but it
appears to work.

Incorporates:

- https://github.com/NixOS/nixpkgs/pull/224945
2023-04-06 03:16:31 -07:00
Adam Joseph
1847b1bc15 stdenv/linux: fix patchelf confusion
The stage before `xgcc` creates the first compiled patchelf
(i.e. not from bootstrapFiles).

The `xgcc` stage was inadvertently switching *back* to using the
patchelf *from* the bootstrapFiles.

The first commit in this PR adds self-checking comments (assertions)
to make it clear where each stage's patchelf comes from.

The second commit fixes the bug, and updates the self-checking
comments.
2023-04-05 22:18:27 -07:00
Adam Joseph
2abf285e2b stdenv/linux: add is{From,BuiltBy} assertions for patchelf 2023-04-05 22:18:22 -07:00
Sergei Trofimovich
c2e14a73e6 xgcc: avoid libc.so mix between gcc and binutils-ld linker plugin
Without the change when I attempt to built `nixpkgs` with weekly
`gcc-13` (it pulls in `flex` as a build input`) I am getting build
failure related to glibc mix caused by glibc loading:

    ...-binutils-patchelfed-ld-2.40/bin/ld: ...-xgcc-13.0.0/libexec/gcc/x86_64-unknown-linux-gnu/13.0.1/liblto_plugin.so:
    error loading plugin: ...-bootstrap-tools/lib/libpthread.so.0: undefined symbol: __libc_vfork, version GLIBC_PRIVATE

The change disables LTO plugin entirely to avoid loading of `glibc` mix.
2023-04-05 07:43:22 +01:00
Bernardo Meurer
f1f6ca8bcd
Merge pull request #209870 from amjoseph-nixpkgs/pr/stdenv/external-gcc-bootstrap 2023-04-03 08:19:03 -07:00
Adam Joseph
96588eb3de gcc: add common/checksum.nix
This commit adds `gcc/common/checksum.nix`, which contains code
common to both gcc11 and gcc12, implementing the `enableChecksum`
feature.

When gcc's built-in bootstrap (`--enable-bootstrap`) is used, gcc
compiles itself three times and compares a hash of the unlinked `.o`
files from the second and third compilation.  The
`enableChecksum=true` parameter performs the same comparison as part
of the `postInstall` phase.

Notably, `enableChecksum=true` can be used with `enableBootstrap=false`.

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
2023-04-02 13:49:53 -07:00
Adam Joseph
fed2300bea unpack-bootstrap-tools.sh: patchelf libgcc_s.so.1 2023-04-02 13:49:53 -07:00
Adam Joseph
d7fe0a5548 make-bootstrap-tools.nix: use a patchelf built with -static-{libgcc,libstdc++}
Our bootstrap-files unpacker has always relied on a lot of unstated
assumptions, one of them being that every library has a DT_NEEDED
for librt.so, so patchelf'ing something into the RUNPATH into
librt.so means that it will be searched for every library load in
all of the bootstrap-files.

Unfortunately that assumption is not true for libgcc.

This causes problems, because patchelf links against libgcc (and
against libstdc++, which links against libgcc).  So we can't use
patchelf on libgcc, because it needs libgcc, so patchelf doesn't
work until libgcc is patchelfed.

The robust solution here is to use static linking for the copy of
patchelf that is shipped with the bootstrap-files.  We don't have to
go all the way to a statically linked libc; just -static-libgcc and
-static-libstdc++ are enough to break the circular dependency.
2023-04-02 13:49:53 -07:00
Adam Joseph
86ca0faff7 make-bootstrap-tools.nix: cp libgcc_s without -d
We do not want to preserve the symlinks from libgcc_s, since they
point to another outpath.  We want to copy from that outpath rather
than link to it.
2023-04-02 13:49:53 -07:00
Adam Joseph
c6bd37a691 make-bootstrap-tools.nix: ship libisl.so
Now that we've dropped the
gcc-links-statically-to-lib{isl,mpfr,mpc,gmp} hack, our gcc needs
libisl.so.  Let's add it to the bootstrap-files.
2023-04-02 13:49:53 -07:00
Adam Joseph
7553d0fe29 stdenv: Nix-driven bootstrap of gcc
#### Summary

By default, when you type `make`, GCC will compile itself three
times.  This PR inhibits that behavior by configuring GCC with
`--disable-bootstrap`, and reimplements the triple-rebuild using
Nix rather than `make`/`sh`.

 #### Immediate Benefits

- Allow `gcc11` and `gcc12` on `aarch64` (without needing new
  `bootstrapFiles`)
- Faster stdenv rebuilds: the third compilation of gcc
  (i.e. stageCompare) is no longer a `drvInput` of the final stdenv.
  This allows Nix to build stageCompare in parallel with the rest of
  nixpkgs instead of in series.
- No more copying `libgcc_s` out of the bootstrap-files or other
  derivations
- No more Frankenstein compiler: the final gcc and the libraries it
  links against (mpfr, mpc, isl, glibc) are all built by the same
  compiler (xgcc) instead of a mixture of the bootstrapFiles'
  compiler and xgcc.
- No more [static lib{mpfr,mpc,gmp,isl}.a hack]
- Many other small `stdenv` hacks eliminated
- `gcc` and `clang` share the same codepath for more of `cc-wrapper`.

 #### Future Benefits

- This should allow using a [foreign] `bootstrap-files` so long as
  `hostPlatform.canExecute bootstrapFiles`.
- This should allow each of the libraries that ship with `gcc`
  (lib{backtrace, atomic, cc1, decnumber, ffi, gomp, iberty,
  offloadatomic, quadmath, sanitizer, ssp, stdc++-v3, vtv}) to be
  built in separate (one-liner) derivations which `inherit src;`
  from `gcc`, much like https://github.com/NixOS/nixpkgs/pull/132343

 #### Incorporates

- https://github.com/NixOS/nixpkgs/pull/210004
- https://github.com/NixOS/nixpkgs/pull/36948 (unreverted)
- https://github.com/NixOS/nixpkgs/pull/210325
- https://github.com/NixOS/nixpkgs/pull/210118
- https://github.com/NixOS/nixpkgs/pull/210132
- https://github.com/NixOS/nixpkgs/pull/210109
- https://github.com/NixOS/nixpkgs/pull/213909
- https://github.com/NixOS/nixpkgs/pull/216136
- https://github.com/NixOS/nixpkgs/pull/216237
- https://github.com/NixOS/nixpkgs/pull/210019
- https://github.com/NixOS/nixpkgs/pull/216232
- https://github.com/NixOS/nixpkgs/pull/216016
- https://github.com/NixOS/nixpkgs/pull/217977
- https://github.com/NixOS/nixpkgs/pull/217995

 #### Closes

- Closes #108305
- Closes #108111
- Closes #201254
- Closes #208412

 #### Credits

This project was made possible by three important insights, none of
which were mine:

1. @ericson2314 was the first to advocate for this change, and
   probably the first to appreciate its advantages.  Nix-driven
   (external) bootstrap is "cross by default".

2. @trofi has figured out a lot about how to get gcc to not mix up
   the copy of `libstdc++` that it depends on with the copy that it
   builds, by moving the `bootstrapFiles`' `libstdc++` into a
   [versioned directory].  This allows a Nix-driven bootstrap of gcc
   without the final gcc would still having references to the
   `bootstrapFiles`.

3. Using the undocumented variable [`user-defined-trusted-dirs`]
   when building glibc.  When glibc `dlopen()`s `libgcc_s.so`, it
   uses a completely different and totally special set of rules for
   finding `libgcc_s.so`.  This trick is the only way we can put
   `libgcc_s.so` in its own separate outpath without creating
   circular dependencies or dependencies on the bootstrapFiles.  I
   would never have guessed to use this (or that it existed!) if it
   were not for a [comment in guix] which @Mic92 [mentioned].

My own role in this PR was basically: being available to go on a
coding binge at an opportune moment, so we wouldn't waste a
[crisis].

[aarch64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662822938
[amd64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662825857
[nonexistent sysroot]: https://github.com/NixOS/nixpkgs/pull/210004
[versioned directory]: https://github.com/NixOS/nixpkgs/pull/209054
[`user-defined-trusted-dirs`]: https://sourceware.org/legacy-ml/libc-help/2013-11/msg00026.html
[comment in guix]: 5e4ec82181/gnu/packages/gcc.scm (L253)
[mentioned]: https://github.com/NixOS/nixpkgs/pull/210112#issuecomment-1379608483
[crisis]: https://github.com/NixOS/nixpkgs/issues/108305
[foreign]: https://github.com/NixOS/nixpkgs/pull/170857#issuecomment-1170558348
[static lib{mpfr,mpc,gmp,isl}.a hack]: 2f1948af9c/pkgs/stdenv/linux/default.nix (L380)
2023-04-02 13:49:41 -07:00
Adam Joseph
70d34589bd stdenv/linux: factor out commonGccOverrides
This commit has no effect on eval.  It simply factors out a common
subexpression.
2023-04-02 03:04:20 -07:00
Adam Joseph
e8b10284f3 linux/default.nix: use mipsel.nix instead of longson.nix for mips32
Right now our bootstrapFiles-selecting algorithm uses the
`loongson2f.nix` bootstrapFiles (which were not built by Hydra).

These bootstrapFiles don't work anymore.  They were added in 2010 by
40405d03ac.

This commit causes mipsel-linux native builds to use the Hydra-built
bootstrap files from this PR instead:

  https://github.com/NixOS/nixpkgs/pull/183487
2023-04-01 23:25:25 -07:00
Elias Naur
88fd6601d8 stdenv: remove the NIX_LIB*_IN_SELF_RPATH environment variables
The NIX_LIB64|32_IN_SELF_RPATH environment variables control whether
to add lib64 and lib32 to rpaths. However, they're set depending
on the build paltform, not the target platform and thus their values
are incorrect for for cross-builds.

On the other hand, setting them according to the build platform introduce
pointless differences in build outputs; see #221350 for details.

This change fixes the issues by boldly removes the NIX_LIB*_IN_SELF_RPATH
facility altogether, in the hope that it is no longer necessary. They
were introduced in 2009, long before nixpkgs had good support for
cross-builds.

Fixes #221350
2023-03-31 10:47:16 -06:00
Vladimír Čunát
12dd95fbb1
Merge branch 'master' into staging-next 2023-03-24 09:07:41 +01:00
Adam Joseph
13507da345 check-meta.nix: fix self-contradictory error messages
See https://github.com/NixOS/nixpkgs/pull/222792#pullrequestreview-1356114111

You can't just `lib.filter _ lib.systems.all` -- that throws away
important information, leading to nixpkgs disagreeing with itself
like this:

```
$ NIXPKGS_ALLOW_BROKEN=1 nix-instantiate . -A pkgsStatic.systemd
error: Package ‘systemd-252.5’ in ... is only supported on ... x86_64-linux but not on requested x86_64-linux, refusing to evaluate.
```

After:

```
$ NIXPKGS_ALLOW_BROKEN=1 nix-instantiate . -A pkgsStatic.systemd
error: Package ‘systemd-252.5’ in ... is not available on the requested hostPlatform:
         hostPlatform.config = "x86_64-unknown-linux-musl"
         package.meta.platforms = [
           "aarch64-linux"
           "armv5tel-linux"
           "armv6l-linux"
           "armv7a-linux"
           "armv7l-linux"
           "i686-linux"
           "m68k-linux"
           "microblaze-linux"
           "microblazeel-linux"
           "mipsel-linux"
           "mips64el-linux"
           "powerpc64-linux"
           "powerpc64le-linux"
           "riscv32-linux"
           "riscv64-linux"
           "s390-linux"
           "s390x-linux"
           "x86_64-linux"
         ]
         package.meta.badPlatforms = [
           {
             isStatic = true;
             parsed = { };
           }
         ]
       , refusing to evaluate.
```
2023-03-24 00:15:57 -07:00
Weijia Wang
b46c97e1dc stdenv: generalise showPlatforms 2023-03-23 23:47:11 +02:00
github-actions[bot]
307b719414
Merge master into staging-next 2023-03-23 18:01:20 +00:00
Weijia Wang
2a7e1e0228 stdenv: fix error with patterned platforms 2023-03-23 17:27:49 +02:00
Sandro
979c6c1fe4
Merge pull request #213780 from SuperSandro2000/check-meta-platform
stdenv: show supported and requested platforms when check meta fails
2023-03-23 15:01:34 +01:00
github-actions[bot]
455127ad5e
Merge master into staging-next 2023-03-16 18:01:20 +00:00
Bernardo Meurer
6e55733359
Merge pull request #219747 from Stunkymonkey/deprecate-isNull 2023-03-16 11:10:22 -03:00
Martin Weinelt
19680e9902
Merge pull request #217568 from trofi/stdenv-parallel-install
stdenv/generic/setup.sh: enable parallel installs for parallel builds
2023-03-15 17:10:19 +00:00
Martin Weinelt
ef91384e6f Merge remote-tracking branch 'origin/master' into staging-next 2023-03-10 13:09:25 +01:00
Bernardo Meurer
e9cf2d1a41
Merge pull request #188334 from amjoseph-nixpkgs/pr/bootstrapFiles/mips64n32 2023-03-09 21:21:15 -05:00
Adam Joseph
49878856e6 https://github.com/NixOS/nixpkgs/pull/188334#issuecomment-1445425412 2023-03-07 02:11:30 -08:00
Felix Buehler
d10e69c86b treewide: deprecate isNull
https://nixos.org/manual/nix/stable/language/builtins.html#builtins-isNull
2023-03-06 22:40:04 +01:00
Martin Weinelt
5aeab34845
Merge pull request #218301 from rrbutani/fix/separate-debuginfo-with-lld
Fix `separate-debug-info` with lld
2023-03-04 00:43:26 +00:00
Weijia Wang
5028ec96c9
Merge pull request #202347 from stephank/feat/bootstrap-aarch64-darwin
stdenvBootstrapTools: native aarch64-darwin build
2023-03-03 20:05:28 +02:00
Vladimír Čunát
763470bbb0
Merge #207135: stdenv: aarch64-linux: gcc9 -> gcc12
...into staging
Also merge the commit that was referenced as the base for the build
of the new bootstrap tools (although others would give the same).
2023-03-01 12:54:52 +01:00
Sergei Trofimovich
69cf5181c3 stdenv/generic/setup.sh: enable parallel installs by default
The primary motivating example is openssl:

Before the change full package build took 1m54s minutes.
After the change full package build takes 59s.

About a 2x speedup.

The difference is visible because openssl builds hundreds of manpages
spawning a perl process per manual in `install` phase. Such a workload
is very easy to parallelize.

Another example would be `autotools`+`libtool` based build system where
install step requires relinking. The more binaries there are to relink
the more gain it will be to do it in parallel.

The change enables parallel installs by default only for buiilds that
already have parallel builds enabled. There is a high chance those build
systems already handle parallelism well but some packages will fail.

Consistently propagated the enableParallelBuilding to:
- cmake (enabled by default, similar to builds)
- ninja (set parallelism explicitly, don't rely on default)
- bmake (enable when requested)
- scons (enable when requested)
- meson (set parallelism explicitly, don't rely on default)
- waf (set parallelism explicitly, don't rely on default)
- qmake-4/5/6 (enable by default, similar to builds)
- xorg (always enable, similar to builds)
2023-02-26 22:02:09 +00:00
Weijia Wang
f05b5d4054 stdenv: aarch64-linux: gcc9 -> gcc12
Hydra job building them: https://hydra.nixos.org/build/208909151

The bootstrap files can be reproduced on the commit 21ec906463, e.g. by:
  cat $(nix-build pkgs/top-level/release.nix -QA stdenvBootstrapTools.aarch64-linux.dist)/nix-support/hydra-build-products
    file tarball /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
    file busybox /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/busybox
and the hashes as well:
  nix hash file /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
    sha256-aJvtsWeuQHbb14BGZ2EiOKzjQn46h3x3duuPEawG0eE=
  nix hash path /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/busybox
    sha256-0MuIeQlBUaeisqoFSu8y+8oB6K4ZG5Lhq8RcS9JqkFQ=

You can check this on any machine, as the builds are on cache.nixos.org
but also you can reproduce the hashes when rebuilt on aarch64-linux HW.
2023-02-26 23:26:30 +02:00
Rahul Butani
d364ee8d13
mkDerivation: do not disable separateDebugInfo on LLVM stdenvs
This was disabled here: b86e62d30d (diff-282a02cc3871874f16401347d8fadc90d59d7ab11f6a99eaa5173c3867e1a160)

h/t to @teh: b86e62d30d (commitcomment-77916294)
for pointing out that the failure that @matthewbauer was
seeing was caused by the `separate-debug-info.sh` `build-id` length
requirement that #146275 will relax

`lld` has had `--build-id` support dating back to LLVM4: https://reviews.llvm.org/D18091

This predates every `llvmPackages_` version currently in nixpkgs (and
certainly every version actually still used in `useLLVM` stdenvs) so
with the previous commit (asking `ld` for sufficiently long SHA1 hashes)
I think we can safely enable `separateDebugInfo` when using LLVM
bintools.
2023-02-25 12:49:40 -06:00
github-actions[bot]
3cdd771820
Merge staging-next into staging 2023-02-23 18:01:49 +00:00
Alyssa Ross
52c286ee5b
Merge remote-tracking branch 'origin/master' into staging-next
Conflicts:
	pkgs/development/libraries/pmdk/default.nix
2023-02-23 13:51:34 +00:00
Artturin
f9fdf2d402 treewide: move NIX_CFLAGS_COMPILE to the env attrset
with structuredAttrs lists will be bash arrays which cannot be exported
which will be a issue with some patches and some wrappers like cc-wrapper

this makes it clearer that NIX_CFLAGS_COMPILE must be a string as lists
in env cause a eval failure
2023-02-22 21:23:04 +02:00
Sergei Trofimovich
3057968396
Merge pull request #210019 from amjoseph-nixpkgs/pr/stdenv/verify-comments
express #208478 as assertions
2023-02-21 19:38:58 +00:00
Adam Joseph
d7aad24531 express #208478 as assertions
PR #208478 added a lot of documentation about which packages were
rebuilt in each stage of the stdenv bootstrap.  However nothing
checks that these comments agree with reality; they can bitrot over
time.  This PR rewrites those comments as assertions, so they cannot
bitrot.

This conversion did expose some ambiguity in our scheme for naming
the stages.   Suppose that `pkgs.stdenv.name=="stdenv-stage4", then
which of these is "the stage4 coreutils"?

```
pkgs.coreutils
pkgs.stdenv.__bootPackages.coreutils
```

The choice is arbitrary, and both choices have confusing corner
cases.  We should revisit this at some point.
2023-02-20 23:20:30 -08:00
Artturi
ee54eb7d21
Merge pull request #216383 from Artturin/bintoolswrappermold 2023-02-17 19:32:06 +02:00
Artturin
299a7bd35e stdenvAdapters: add useMoldLinker 2023-02-17 06:35:24 +02:00
Robert Scott
0eedcfc3f4
Merge pull request #212498 from risicle/ris-fortify3
hardening flags: add `FORTIFY_SOURCE=3` support
2023-02-16 21:19:30 +00:00
github-actions[bot]
c4fe2133de
Merge staging-next into staging 2023-02-15 06:01:44 +00:00
github-actions[bot]
58cfebde3e
Merge staging-next into staging 2023-02-15 00:02:50 +00:00
Felix Buehler
bc3d5934d7 treewide: use lib.optionals 2023-02-14 19:11:59 +01:00
Felix Buehler
cdb39a86e0 treewide: use optionalString 2023-02-13 21:52:34 +01:00
github-actions[bot]
29c2637dda
Merge staging-next into staging 2023-02-13 12:02:05 +00:00
github-actions[bot]
fd8daee7c2
Merge master into staging-next 2023-02-13 12:01:30 +00:00
Weijia Wang
7c026bae62
stdenv: aarch64-linux: update the bootstrap tools (incl. busybox)
Hydra job building them: https://hydra.nixos.org/build/208909151

The bootstrap files can be reproduced on the parent commit, e.g. by:
  cat $(nix-build pkgs/top-level/release.nix -QA stdenvBootstrapTools.aarch64-linux.dist)/nix-support/hydra-build-products
    file tarball /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
    file busybox /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/busybox
and the hashes as well:
  nix hash file /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
    sha256-aJvtsWeuQHbb14BGZ2EiOKzjQn46h3x3duuPEawG0eE=
  nix hash path /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/busybox
    sha256-0MuIeQlBUaeisqoFSu8y+8oB6K4ZG5Lhq8RcS9JqkFQ=

You can check this on any machine, as the builds are on cache.nixos.org
but also you can reproduce the hashes when rebuilt on aarch64-linux HW.
2023-02-12 10:09:35 +01:00
Stéphan Kochen
c3693fbfd5 stdenvBootstrapTools: native aarch64-darwin build 2023-02-11 20:11:55 +01:00
Artturin
84e37a10ec stdenv: allow propagating propagated dependencies separately from the
fixup phase

for makeSetupHook
2023-02-07 21:00:18 +02:00
Robert Hensing
059bd43546 make-derivation.nix: Support inputDerivation on disallowedReferences 2023-02-07 18:58:13 +01:00
github-actions[bot]
99cce0e1f1
Merge staging-next into staging 2023-02-07 06:02:00 +00:00
github-actions[bot]
41e5bd55d5
Merge master into staging-next 2023-02-07 06:01:28 +00:00
figsoda
42d1d60a92
Merge pull request #206773 from SuperSandro2000/cleanup-unused-bindings
treewide: cleanup some unused bindings
2023-02-06 20:07:50 -05:00
Sandro Jäckel
50e0012f9d
treewide: cleanup some unused bindings 2023-02-07 01:36:15 +01:00
Artturi
aa7d2fe7dc
Merge pull request #214937 from hercules-ci/support-NIX_ATTRS-envs 2023-02-06 20:59:10 +02:00
github-actions[bot]
36822caa75
Merge staging-next into staging 2023-02-06 18:02:09 +00:00
github-actions[bot]
f876e1f1e9
Merge master into staging-next 2023-02-06 18:01:37 +00:00
John Ericson
6d0b3086f7
Merge pull request #214304 from obsidiansystems/pkg-config-meta
meta.pkgConfigModules: Init convention
2023-02-06 11:44:29 -05:00
Robert Hensing
afef6588e2 stdenv/setup.sh: Allow NIX_ATTRS_{JSON,SH}_FILE to be set correctly by Nix 2023-02-06 14:17:59 +01:00
github-actions[bot]
dfee1a3150
Merge staging-next into staging 2023-02-05 00:03:15 +00:00
github-actions[bot]
bc833a50cc
Merge master into staging-next 2023-02-05 00:02:31 +00:00
superherointj
2c77d453e3
Merge pull request #213633 from alyssais/mkDerivation-meson-exotic
stdenv.mkDerivation: fix meson for some archs
2023-02-04 18:43:43 -03:00
John Ericson
6e4a1b18d9 meta.pkgConfigModules: Init convention
See docs.

Follow-up work:

- Existing packages should be converted

- `defaultPkgConfigPackages` should assert on `meta.pkgConfigModules`
  and let `tests.pkg-config` alone test the build results.

CC @sternenseemann

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-02-03 09:37:31 -05:00
Artturi
dcc7df7fe6
Merge pull request #211685 from Artturin/splicingstuff1-split 2023-02-03 12:49:08 +02:00
Artturi
fa5f0d6bd0
Merge pull request #211783 from R-VdP/fix_disallowed_references
stdenv: don't include drvs in disallowedRefs as build-time deps.
2023-01-31 20:51:03 +02:00
Sandro Jäckel
0a8e692d29
stdenv: show supported and requested platforms when check meta fails 2023-01-31 14:43:47 +01:00
github-actions[bot]
33d4318fcc
Merge staging-next into staging 2023-01-31 00:03:07 +00:00
github-actions[bot]
dd1ff149da
Merge master into staging-next 2023-01-31 00:02:31 +00:00
Alyssa Ross
b6304bf11d
stdenv.mkDerivation: fix meson for some archs
platform.uname.processor seems to be what we want in many more cases
than what we were using before — it does the right thing for aarch64,
x86_64, riscv32, riscv64, mips, mips64, powerpc, and powerpc64 (the
latter three of which were broken before).

This fixes cross-compilation of systemd for PowerPC/POWER platforms.
2023-01-30 19:42:19 +00:00
R-VdP
51acc6245e
stdenv: don't include drvs in disallowedRefs as build-time deps.
Derivations listed as disallowedReferences or disallowedRequisites,
currently end up as build-time dependencies.
This is problematic since the disallowed derivations will be built by nix as
build-time dependencies, while those derivations might take a very long time
to build, or might not even build successfully on the platform used.
However, in order to scan for disallowed references in the final output,
knowing the out path is sufficient, and the out path can be calculated from
the derivation without needing to build it, saving time and resources.

While the problem is less severe for allowedReferences and allowedRequisites,
since we want the derivation to be built eventually, we would still like to
get the error early and without having to wait while nix builds a derivation
that might not be used (e.g. if we listed the wrong one).
2023-01-30 17:58:05 +01:00
Sergei Trofimovich
b3a97d7eb4
Merge pull request #211910 from tejing1/srcs-name-collision
stdenv: Improve error from `stdenv` when `srcs` is used with colliding directories
2023-01-29 12:04:42 +00:00
Sergei Trofimovich
0ba9da441d
Merge pull request #211126 from trofi/binutils-update
binutils: 2.39 -> 2.40
2023-01-29 09:50:57 +00:00
Jeff Huffman
183939da54
improve error when srcs is used with directories with the same post-hash name 2023-01-29 04:27:24 -05:00
Sergei Trofimovich
e1ef521cff binutils: 2.39 -> 2.40
A few potentially disruptive changes:

- binutils does not embed ${binutils-unwrapped}/lib as a default library
  search path anymore. This will cause link failures for -lbfd -lopcodes
  users that did not declare their dependency on those libraries. They
  will need to add `libbfd` and `libopcodes` attributes to build inputs.

- `libbfd` and `libopcodes` attributes now just reference
  `binutils-unwrapped.{dev,lib}` pair of attributes without patching
  `binutils` build system.

We don't patch build system anymore and use multiple outputs out of
existing `binutils` build. That makes the result more maintainable: no
need to handle ever growing list of dependencied of `libbfd`. This time
new addition was `libsframe`.

To accomodate `out` / `lib` output split I had to remove `lib` -> `bin`
backreference by removing legacy lookup path for plugins.

I also did not enable `zstd` just yet as `nixpkgs` version of `zstd`
package pulls in `cmake` into bootstrap sequence.

Changes: https://lists.gnu.org/archive/html/info-gnu/2023-01/msg00003.html
2023-01-27 23:16:45 +00:00
Adam Joseph
778419b9e6 Revert "lib/meta.nix: platformMatch: allow predicate functions"
This reverts commit b7d097438b.
2023-01-27 01:49:39 -08:00
github-actions[bot]
c47e4b69a3
Merge master into staging-next 2023-01-25 00:02:20 +00:00
Robert Scott
c09e1fa406 gcc: mark hardeningUnsupportedFlags fortify3 for all but gcc 12 2023-01-24 21:52:12 +00:00
Robert Scott
3d453e2aee mkDerivation: add support for fortify3 hardening flag 2023-01-24 21:52:10 +00:00
Alyssa Ross
2ae30c9f45 llvmPackages: use libcxxrt on FreeBSD
FreeBSD doesn't use LLVM's cxxabi implementation, for backwards
compatibility reasons.  Software expects the libcxxrt API when
building on FreeBSD.  This fixes the build of
pkgsCross.x86_64-freebsd.boost.
2023-01-24 21:18:39 +00:00
github-actions[bot]
feb2240b37
Merge master into staging-next 2023-01-22 18:01:03 +00:00
Adam Joseph
b7d097438b lib/meta.nix: platformMatch: allow predicate functions 2023-01-22 00:27:19 -08:00
Guillaume Girol
4ce9749f74 Merge remote-tracking branch 'origin/staging' into staging-nativeCheckInputs 2023-01-21 17:18:51 +01:00
Artturin
d29b1ecb89 make-bootstrap-tools.nix: fix for wrapped gzip 2023-01-21 17:19:38 +02:00
Guillaume Girol
f39abbc350 mkDerivation: introduce native checkInputs and nativeCheckInputs
When strictDeps is set, only nativeCheckInputs are added to PATH and
only checkInputs can be linked against. See https://github.com/NixOS/nixpkgs/issues/161570
2023-01-21 16:42:09 +01:00
Guillaume Girol
90c78aee6c Merge branch 'nativeCheckInputs' into staging-nativeCheckInputs 2023-01-21 12:00:00 +00:00
Sergei Trofimovich
b9b1d958d0 Merge remote-tracking branch 'origin/staging-next' into staging
Conflicts:
	pkgs/development/libraries/qt-6/modules/qtbase.nix
	pkgs/stdenv/linux/make-bootstrap-tools.nix
2023-01-20 21:56:57 +00:00
Alyssa Ross
0ae87d514f treewide: add names to all setup hooks 2023-01-19 15:00:36 +00:00
Andrew Childs
68f1182b65 stdenv: don't clobber useArray and type in {prepend,append}ToVar
Some other packages, for example ruby gems via buildRubyGem, use a
variable called "type" internally, which is overwritten here and
causes failures like:

    failure: $gempkg path unspecified

Fix for changes in 11c3127e38.
2023-01-18 11:59:47 +09:00
Artturin
e525ae1e1e stdenv: disable shellcheck rule SC2068 & SC1091
this is intentional to support both structuredAttrs and non

In pkgs/stdenv/generic/setup.sh line 614:
for pkg in ${depsBuildBuild[@]} ${depsBuildBuildPropagated[@]}; do
           ^------------------^ SC2068 (error): Double quote array expansions to avoid re-splitting elements.

In pkgs/stdenv/generic/setup.sh line 521:
    local varRef="$varVar[$((targetOffset - hostOffset))]"
                  ^-- SC1087 (error): Use braces when expanding arrays, e.g. ${array[idx]} (or ${var}[.. to quiet).
2023-01-16 23:54:14 +02:00
Artturin
f2c27018f0 stdenv: fix SC2242
exit -1 == exit 255 but we don't have a reason to use 255

In pkgs/stdenv/generic/setup.sh line 518:
    (( hostOffset <= targetOffset )) || exit -1
                                             ^-- SC2242 (error): Can only exit with status 0-255. Other data should be wri
tten to stdout/stderr.
2023-01-16 23:03:21 +02:00
sternenseemann
8e35002bc2
make-bootstrap-tools.nix: don't pull in pkgs.glibc in test (#210038)
`builtins.baseNameOf` retains any string context, causing the test
derivation to incorrectly depend on `pkgs.glibc`. All we really want is
to know what the dynamicLinker is called, but we don't need it to be
present in store.

Thanks to Adam Joseph for spotting this.
2023-01-16 19:29:40 +01:00
Artturin
e58785bf41 stdenv: disable shellcheck rule SC2048
we use [*] to support structuredAttrs and non

In pkgs/stdenv/generic/setup.sh line 1542:
    for curPhase in ${phases[*]}; do
                    ^----------^ SC2048 (warning): Use "${array[@]}" (with quotes) to prevent whitespace problems.
2023-01-16 08:23:04 +02:00
Artturin
4db439c599 stdenv: disable shellcheck rules
In pkgs/stdenv/generic/setup.sh line 101:
        source "$hookName"
               ^---------^ SC1090 (warning): ShellCheck can't follow non-constant source. Use a directive to specify location.

In pkgs/stdenv/generic/setup.sh line 166:
            mkdir -p "$out/nix-support"
                      ^--^ SC2154 (warning): out is referenced but not assigned.

In pkgs/stdenv/generic/setup.sh line 407:
PATH=
^--^ SC2123 (warning): PATH is the shell search path. Use another name.

In pkgs/stdenv/generic/setup.sh line 452:
declare -a pkgBuildAccumVars=(pkgsBuildBuild pkgsBuildHost pkgsBuildTarget)
           ^---------------^ SC2034 (warning): pkgBuildAccumVars appears unused. Verify use (or export if used e
xternally).
because pkgBuildAccumVars is used

In pkgs/stdenv/generic/setup.sh line 235:
        nameref="$* ${nameref-}"
        ^-----^ SC2178 (warning): Variable was used as an array but is now assigned a string.
because we theres a useArray conditional
2023-01-16 08:01:38 +02:00
Artturin
0417f953e2 stdenv: fix SC2004 & SC2086
SC2004 (style): $/${} is unnecessary on arithmetic variables.
SC2086 (info): Double quote to prevent globbing and word splitting.
2023-01-16 07:54:19 +02:00
Artturin
0e8263ce73 stdenv: fix SC2223
In pkgs/stdenv/generic/setup.sh line 36:
    : ${outputs:=out}
      ^-------------^ SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
2023-01-16 07:54:19 +02:00
Artturin
856f3a46b2 stdenv: drop remove unnecessary env var
and the associated obsolete functions

support for log nesting was removed in 2017 6669a3b477
2023-01-15 23:56:23 +02:00
Sergei Trofimovich
9c0068c40e
Merge pull request #210752 from trofi/make-bootstrap-tools-libstdcxx-rpath
make-bootstrap-tools: fix test to include libstdc++ -rpath
2023-01-15 07:01:34 +00:00
Robert Hensing
bff126ffbc stdenv.mkDerivation: Make overrideAttrs overridable
(cherry picked from commit 43c8b43f808f48fd5600afcad5503eaeaf6d71b7)
2023-01-15 04:37:14 +02:00
Artturi
f338f1422e
Merge pull request #209371 from Artturin/gziprepro
gzip: make reproducible when GZIP_NO_TIMESTAMPS is set
2023-01-15 00:04:51 +02:00
Artturi
5fae6b5c74
Merge pull request #210109 from amjoseph-nixpkgs/pr/stdenv/build-gettext-only-once 2023-01-14 21:55:22 +02:00
Sergei Trofimovich
497033467f make-bootstrap-tools: fix test to include libstdc++ -rpath
After https://github.com/NixOS/nixpkgs/pull/209054 we started moving
libstdc++.so out of default glibc's paths. This exposed bootstrap
tools build failure as:

    $ nix build --no-link -f ./pkgs/stdenv/linux/make-bootstrap-tools.nix
    ...
    >
    .../bin/bar: error while loading shared libraries: libstdc++.so.6:
       cannot open shared object file: No such file or directory

Note that bootstrap itself did not break. The change only expands
handcrafted `-rpath` entries.
2023-01-14 19:42:50 +00:00
Adam Joseph
58857196f1
stdenv: mark binutils-patchelfed (#209600)
The stdenv bootstrap creates several wrappers around binutils, and
gives them the exact same drvName as the binutils package itself.

These wrappers cost almost nothing to create (they are just file
copies and patchelf runs, not builds), so we should distinguish them
from expensive binutils builds with a unique pname.  This commit
does that.
2023-01-14 10:07:44 +00:00
Sergei Trofimovich
4433093854
Merge pull request #209054 from trofi/stdcxx-out-of-bootstrap-lib
linux/bootstrap-tools: move libstdc++ out of default library search path
2023-01-12 18:27:58 +00:00
Artturin
88f36d2694 gzip: make reproducible when GZIP_NO_TIMESTAMPS is set
the logic can be tested with

```
$ GZIP_NO_TIMESTAMPS=1 && echo "${GZIP_NO_TIMESTAMPS:+-n }"
-n
$ unset GZIP_NO_TIMESTAMPS && echo "${GZIP_NO_TIMESTAMPS:+-n }"

```
2023-01-12 01:30:29 +02:00
github-actions[bot]
b3d2428df4
Merge staging-next into staging 2023-01-11 20:59:01 +00:00
github-actions[bot]
a113c6f06f
Merge master into staging-next 2023-01-11 20:58:29 +00:00
Adam Joseph
098c6b0bec check-meta(hasUnsupportedPlatform): use lib.meta.availableOn
`hasUnsupportedPlatform` was not updated with #37395, so it does not
understand attrsets in `meta.[bad]platforms`.  In particular,
attrsets in `meta.badPlatforms` will "fail open" and be ignored.

Let's use `lib.meta.availableOn` instead of duplicating its logic.

Thanks to @alyssais for [noticing][1].

[1][https://github.com/NixOS/nixpkgs/pull/194148#discussion_r990817610]

Co-authored-by: sternenseemann <sternenseemann@systemli.org>
2023-01-11 19:31:52 +00:00
Adam Joseph
74df5ad72b stdenv: build gettext only once
Right now we build gettext several times during the bootstrap.
Gettext's build process is "embarrassingly serial", so avoiding
rebuilding it speeds things up considerably.
2023-01-10 13:29:02 -08:00
Artturi
386f2bb148
Merge pull request #209255 from Artturin/stdenvupdate1
stdenv: don't fail installPhase on missing makefile
2023-01-10 03:48:17 +02:00
Sergei Trofimovich
8a4e6b7e1b stdenv/darwin: rebuild gawk earlier: in stage4 instead of final stage
`gawk-5.1.1 -> 5.2.1` update (https://github.com/NixOS/nixpkgs/pull/207478)
started failing `stdenv` reference checks as `gawk` now leaks
`bootstrapTools` reference:

    `gawk` -> `gettext` -> `libiconv` -> `bootstrapTools`.

The change rebuild `gawk` in `stage4` to pull rebuilt tools.
2023-01-06 06:48:35 +00:00
Artturin
314b03125f stdenv: don't fail installPhase on missing makefile
otherwise the build just fails with 'make: *** No rule to make target 'install'.  Stop.'

and update buildPhase message

i don't know if the 'makefile may have been created in buildPhase' is
true but i guess it might be possible
2023-01-06 03:01:39 +02:00
github-actions[bot]
789ae2c93d
Merge staging-next into staging 2023-01-04 18:01:36 +00:00
Sergei Trofimovich
b470a6b212 linux/bootstrap-tools: move libstdc++ out of default library search path
This change allows building new gcc during bootstrap without fear of
pulling in outdated libstdc++.so after g++ switched from bootstrapTools
to freshly built g++.

Noticed when tried to add early bootstrap stage to rebuild `gcc` before
`glibc` is fully untangled from `bootstrapTools` as a failure to built
`binutils`:

    ld: dwp.o: in function `__gnu_cxx::new_allocator<gold::Dwp_output_file::Contribution>::allocate(unsigned long, void const*)':
    /nix/store/...-gcc-11.3.0/include/c++/11.3.0/ext/new_allocator.h:116: undefined reference to `std::__throw_bad_array_new_length()'

The change survives existing bootstrap and unblockes early `gcc` bootstrap.
2023-01-04 16:02:22 +00:00
piegames
aa8ead30fd
Merge pull request #175495 from amjoseph-nixpkgs/pr/sourceProvenance/correct-meta-typecheck
check-meta.nix: make non-source consistent with documentation
2023-01-04 15:31:51 +01:00
github-actions[bot]
03e830eb69
Merge staging-next into staging 2023-01-03 00:02:53 +00:00
John Ericson
a6b1de7902
Merge pull request #208478 from trofi/comment-stdenv-bootstrap
stdenv/linux: document some tips in debugging stdenv bootstrap tower
2023-01-02 17:28:43 -05:00
Sergei Trofimovich
062e14b561 stdenv/linux: document some tips in debugging stdenv bootstrap tower
Just a few comments added:

- added a few one-liners to explore which tools are rebuilt at each
  stdenv iteration during bootstrap
- explicitly listed available toolchains and their sources for on each
  bootstrap step: glibc, binutils, gcc, coreutils.
- added mention of static libraries linked into gcc

Co-authored-by: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com>
2023-01-02 21:55:32 +00:00
Adam Joseph
607d59fa9e check-meta.nix: make non-source consistent with documentation
The documentation for `meta.sourceProvenance` in
`doc/stdenv/meta.chapter.md` says: "the `meta.sourceProvenance`
attribute should be a list containing one or more value..."

Let's update check-meta.nix to require that `meta.sourceProvenance` is
a list, as the documentation says, rather than a single element.

Adding two extra keystrokes `[` and `]` when filling out this field is
an insignificant burden for package authors, and being able to assume
that the `meta.sourceProvenance` field is always a list greatly
simplifies any code that acts on the value of this field.

Since `meta.sourceProvenance` was just merged a few hours ago now is
the easiest time to fix this: nobody is using the feature yet.
2023-01-01 18:21:11 -08:00
Jan Tojnar
5810109b42 Merge branch 'staging-next' into staging
- readline6 attribute removed from all-packages.nix in d879125d61
- readline attribute was bumped to readline82 in 50adabdd60
2023-01-02 03:04:32 +01:00
Naïm Favier
4af22aab8e
stdenv/check-meta: do deep type checks
Use a wrapper around `mergeDefinitions` to type-check values deeply, so
that e.g. `maintainers = [ 42 ];` is an error.
2023-01-01 14:10:42 +01:00
github-actions[bot]
dc7ebb0163
Merge staging-next into staging 2022-12-18 18:01:41 +00:00
figsoda
ec8cb34358 treewide: fix typos 2022-12-17 19:39:44 -05:00
Sergei Trofimovich
428107f837 stdenv: set enableParallelBuilding explicitly if enableParallelBuildingByDefault is set
Without the change we don't propagate `enableParallelBuilding = true`
and leave most builds sequential.

Noticed on `mythtv` package which did not specify parallelism and
`config.enableParallelBuildingByDefault = true` had no effect.
2022-12-17 22:23:24 +00:00
Naïm Favier
0b067316d4
stdenv: use intersectAttrs instead of intersectLists
Better complexity.
2022-12-17 12:55:44 +01:00
Naïm Favier
84eebc0fe4
Merge pull request #205944 from ncfavier/structured-attrs-env 2022-12-15 13:27:54 +01:00
Naïm Favier
e14de22618
stdenv: handle env gracefully
Derivations not using `__structuredAttrs` should not attempt to set
environment variables from `env`.

Derivations using `__structuredAttrs` should fail if `env` is not
exportable.
2022-12-15 13:27:11 +01:00
Artturin
630bb71ac5 stdenv: sort defaultNativeBuildInputs alphabetically 2022-12-13 01:12:20 +02:00
Artturin
b3717f6c14 stdenv: remove now unneeded linux conditional 2022-12-13 01:04:29 +02:00
Artturin
7866db71cc stdenv/generic: fix todo 2022-12-13 00:34:48 +02:00
Artturin
11c3127e38 stdenv: detect the type of variable in {prepend,append}ToVar
stdenv: error if using {prepend,append}ToVar on associative array

i don't know how to prepend to associative array
2022-12-10 04:42:36 +02:00
Artturin
89dc806f13 what to do about attrs.env or {} maybe have a empty env attrset always
so no need to use `or`
2022-12-08 21:11:35 +02:00
Artturin
02e3f51d27 darwin: use // for binutils-unwrapped and cctools to preserve the other
attributes
2022-12-08 21:09:02 +02:00
Artturin
bb914d8676 stdenv: export system pname name version for substituteAll
so we don't have to add these to the env attrset
2022-12-08 21:09:02 +02:00
Artturin
734d7df235 allow derivation attributes in env
derivations can be coerced to their output paths
2022-12-08 06:13:19 +02:00
Artturin
8ad0103a34 config.structuredAttrsByDefault: add option 2022-12-08 06:13:19 +02:00
Artturin
1c4820efdd work around a nix bug 2022-12-08 06:13:19 +02:00
Artturin
238a6053c4 stdenv: support opt-in __structuredAttrs
Co-authored-by: Robin Gloster <mail@glob.in>

stdenv: print message if structuredAttrs is enabled

stdenv: add _append

reduces the chance of a user doing it wrong

fix nix develop issue

output hooks don't work yet in nix develop though

making $outputs be the same on non-structuredAttrs and structuredAttrs
is too much trouble.

lets instead make a function that gets the output names

reading environment file '/nix/store/2x7m69a2sm2kh0r6v0q5s9z1dh41m4xf-xz-5.2.5-env-bin'
nix: src/nix/develop.cc:299: std::string Common::makeRcScript(nix::ref<nix::Store>, const BuildEnvironment&, const Path&): Assertion `outputs != buildEnvironment.vars.end()' failed.

use a function to get all output names instead of using $outputs

copy env functionality from https://github.com/NixOS/nixpkgs/pull/76732/commits
2022-12-08 06:13:19 +02:00
Artturi
df3056bfac
Merge pull request #176597 from amjoseph-nixpkgs/pr/remove-redundant-file-buildInput
stdenv/cross: remove now-redundant `file` nativeBuildInput on mingw
2022-11-21 19:22:16 +02:00
github-actions[bot]
77fbd162df
Merge staging-next into staging 2022-11-20 18:01:45 +00:00
Artturin
341e6fd558 splice.nix: start deprecating nativeDrv and crossDrv 2022-11-19 00:04:54 +02:00
Robert Hensing
51518a5fd3 stdenv.tests: Add succeedOnFailure 2022-11-14 19:03:35 +01:00
Sandro Jäckel
9f4143e964
stdenv: fix succeedOnFailure 2022-11-12 19:25:14 +01:00
github-actions[bot]
f52955a521
Merge master into staging-next 2022-10-16 18:01:42 +00:00
piegames
87d738e864
Merge #195120: check-meta.nix: fix checkMetaRecursively option 2022-10-16 15:33:16 +02:00
github-actions[bot]
4f07e7f326
Merge master into staging-next 2022-10-15 06:07:54 +00:00
Robert Scott
692512da6a
Merge pull request #193415 from risicle/ris-nixpkgs-allow-nonsource-fix-1
stdenv/check-meta: fix support for `NIXPKGS_ALLOW_NONSOURCE=1`
2022-10-15 01:25:44 +01:00
arcnmx
466fd1439f check-meta.nix: fix checkMetaRecursively option
In specific cases, combining the `checkMeta` and `checkMetaRecursively`
config options would result in `error: infinite recursion encountered`

fixes #193296
2022-10-13 14:25:57 -07:00
Franz Pletz
3624ac2458
perl: fix build with libxcrypt 2022-10-09 18:07:54 +02:00
Martin Weinelt
253ca4957d Merge remote-tracking branch 'origin/master' into staging-next 2022-10-05 00:44:16 +02:00
Artturin
30cc0cde34 stdenv: complete the deprecation of stdenv.glibc 2022-10-03 20:19:54 +03:00
Robert Scott
d02ac63f4f stdenv/check-meta: fix support for NIXPKGS_ALLOW_NONSOURCE=1 2022-09-28 23:18:33 +01:00
github-actions[bot]
f18d801779
Merge staging-next into staging 2022-09-28 18:05:53 +00:00
aszlig
8f98a6d39b check-meta: Add isHydraChannel
This is needed in order to mark a certain derivation containing a Nix
expression tarball to Hydra so that it is recognised as a channel.

When I first got an evaluation error due to using this meta attribute, I
was under the impression that nobody outside of Vuizvui[1] is using this
feature and that we don't have any occurrence of isHydraChannel in
Nixpkgs.

However, when working around[2] the issue I assumed that it's not
something that should be included in Nixpkgs because we're not using it
there.

It turned out that my assumption was wrong and we *do* use the attribute
in Nixpkgs, namely via releaseTools.channel, which is similar to what
we're doing in Vuizvui.

Since we already include a bunch of undocumented attributes in
metaTypes, it only makes sense to add isHydraChannel as well since it's
actually documented in the Hydra documentation[3].

[1]: https://github.com/openlab-aux/vuizvui
[2]: https://github.com/openlab-aux/vuizvui/commit/e0685e81b3fdc43a272f0
[3]: 53335323ae/doc/manual/src/jobs.md (meta-fields)

Signed-off-by: aszlig <aszlig@nix.build>
2022-09-28 14:12:45 +02:00
github-actions[bot]
e0bea8e017
Merge staging-next into staging 2022-09-26 00:03:58 +00:00
piegames
6762de9a28 check-meta.nix: type checking changes
- Enable metadata checking by default, see https://github.com/NixOS/nixpkgs/pull/25304#issuecomment-298385426
- Check metadata before any other package issues, see https://github.com/NixOS/nixpkgs/issues/191124#issuecomment-1246523976
- Document that type checks only apply to the top level of nested values.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2022-09-25 16:37:15 +02:00
Graham Christensen
c2b898da76 treewide: drop -l$NIX_BUILD_CORES
Passing `-l$NIX_BUILD_CORES` improperly limits the overall system load.

For a build machine which is configured to run `$B` builds where each
build gets `total cores / B` cores (`$C`), passing `-l $C` to make will
improperly limit the load to `$C` instead of `$B * $C`.

This effect becomes quite pronounced on machines with 80 cores, with
40 simultaneous builds and a cores limit of 2. On a machine with this
configuration, Nix will run 40 builds and make will limit the overall
system load to approximately 2. A build machine with this many cores
can happily run with a load approaching 80.

A non-solution is to oversubscribe the machine, by picking a larger
`$C`. However, there is no way to divide the number of cores in a way
which fairly subdivides the available cores when `$B` is greater than
1.

There has been exploration of passing a jobserver in to the sandbox,
or sharing a jobserver between all the builds. This is one option, but
relatively complicated and only supports make. Lots of other software
uses its own implementation of `-j` and doesn't support either `-l` or
the Make jobserver.

For the case of an interactive user machine, the user should limit
overall system load using `$B`, `$C`, and optionally systemd's
cpu/network/io limiting features.

Making this change should significantly improve the utilization of our
build farm, and improve the throughput of Hydra.
2022-09-22 16:01:23 -04:00
Stéphan Kochen
6515b56871
darwin.apple_sdk_11_0: use stdenv objc4
Stdenv on aarch64-darwin pulls in (bootstrap-stage4) objc4, unlike
x86_64. However derivations that otherwise depend on objc4 would use a
a different objc4 derivation on top of the final stdenv.

Because this library defines an LLVM module, having multiple instances
of it in the import path will interfere with builds.
2022-09-20 12:14:49 +02:00
Vladimír Čunát
f08ac2c92b
Merge branch 'master' into staging-next 2022-08-29 08:58:52 +02:00
pennae
abd2e87da1 stdenv: remove deprecated adapters 2022-08-26 16:25:44 +02:00
Adam Joseph
51ba233717 pkgs/stdenv/linux: bootstrap-files for third (and final) mips ABI: n32
This PR provides the Hydra-generated bootstrap tarballs for mips64el-linux-gnuabin32.

With this PR we now have the bootstrap-file for all three little-endian ABIs on mips: n64, n32, and o32.  I do not currently plan to do big-endian mips unless some motivation arises; all mips chips are bi-endian and Debian has dropped big-endian support due to lack of interest.

I'll be following the script used in #151399, #168199, and #183487.

Files came from [this](https://hydra.nixos.org/build/188389586#tabs-summary) Hydra build, which used nixpkgs revision 97d9c84e1d to instantiate:

```
/nix/store/hakn8s85s9011v61r6svp5qy8x1y64fv-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabin32.drv
```

and then built:

```
/nix/store/rjgybpnf3yiqyhvl2n2lx31jf800fii2-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabin32
```

I downloaded these files from Hydra and prefetched them into the nix store with the following commands:

```
STOREPATH=rjgybpnf3yiqyhvl2n2lx31jf800fii2-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabin32
OPTIONS="--option binary-caches https://cache.nixos.org --option trusted-public-keys cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
nix store prefetch-file \
  file://$(nix store add-file --name bootstrap-tools.tar.xz  $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/bootstrap-tools.tar.xz)
nix store prefetch-file --executable \
  file://$(nix store add-path --name busybox $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/busybox)
```

These commands produced the following output:

```
Downloaded 'file:///nix/store/w6zzd2fx2vhmjfcf5h5zc01m0swldpbw-bootstrap-tools.tar.xz' to '/nix/store/6w0f0mqblrghvh6yjwcb4xdqq9x50lbl-w6zzd2fx2vhmjfcf5h5zc01m0swldpbw-bootstrap-tools.tar.xz' (hash 'sha256-LWrpN6su2yNVurUyhZP34OiZyzgh7MfN13fIIbou8KI=').
Downloaded 'file:///nix/store/nqagw1kgdz1zlmqi00qfjrmwqk3g3bgd-busybox' to '/nix/store/i361xhbdhhnvg7zd637xpm63vbl80s0s-nqagw1kgdz1zlmqi00qfjrmwqk3g3bgd-busybox' (hash 'sha256-4N3G1qYA7vitjhsIW17pR6UixIuzrq4vZXa8F0/X4iI=').
```

I used the hashes from the output above to create the `fetchurl` invocation which is part of this commit.

I then started the bootstrap with the following command:

```
nix build -L -f . --arg localSystem '(import ./lib).systems.examples.mips64el-linux-gnuabin32' hello
```

As @lovesegfault requested, here are the the `sha256sum`s of all the `on-server` components for extra verification:

```
sha256sum /nix/store/${STOREPATH}/on-server/*
```

which produced the following output:

```
2d6ae937ab2edb2355bab5328593f7e0e899cb3821ecc7cdd777c821ba2ef0a2  /nix/store/rjgybpnf3yiqyhvl2n2lx31jf800fii2-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabin32/on-server/bootstrap-tools.tar.xz
01633f71135cb9ab1b5ce3ebb67e80cbf288739729bffc1350c1552f6f8df34b  /nix/store/rjgybpnf3yiqyhvl2n2lx31jf800fii2-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabin32/on-server/busybox
```
2022-08-25 16:13:37 -07:00
github-actions[bot]
0b8aa77df1
Merge master into staging-next 2022-08-25 18:01:10 +00:00
Bernardo Meurer
14181ab152
Merge pull request #185182 from amjoseph-nixpkgs/pr/powerpc64le-bootstrapFiles/again
pkgs/stdenv/linux: update powerpc64le bootstrap-files
2022-08-25 10:07:22 -03:00
Matthew Bauer
ec4019f6dc Set inherit_errexit after bash version check
inherit_errexit wasn’t available in bash 3. We have a check to show a
nice error message, but that check is after we set inherit_errexit in
setup.sh. So we can just move this to below the BASH_VERSINFO check.
2022-08-21 23:01:55 -04:00
Sergei Trofimovich
a0219e7fb7
Merge pull request #181536 from trofi/fix-gnumake-guile
gnumake: unconditionally disable guileSupport on bootstrap
2022-08-19 19:22:13 +01:00
Sergei Trofimovich
7f92320f66 make-bootstrap-tools.nix: drop libelf.so from tootstrap tarballs
gcc stopped using libelf in commit 48215350c24 ("re PR lto/46273 (Failed
to bootstrap)") around 2010, before gcc-4.6.0.

Bootstrap tools don't use it either.
2022-08-18 08:23:08 +01:00
Bernardo Meurer
d00f5c5eaa
Merge pull request #183487 from amjoseph-nixpkgs/pr/mipsel/bootstrap-files
stdenv/linux/bootstrap-files: add mipsel-linux (mips32)
2022-08-17 15:14:47 -07:00
Adam Joseph
6122297eab pkgs/stdenv/linux: update powerpc64le bootstrap-files
This PR updates the Hydra-generated bootstrap tarballs for
powerpc64le-linux.  The bootstrap-files referenced prior to this
commit will only bootstrap in a nixpkgs which has
https://github.com/NixOS/nixpkgs/pull/181802.  That PR was closed in
favor of https://github.com/NixOS/nixpkgs/pull/181943, which is a
cleaner solution but which requires regenerating the bootstrap-files.

I'll be following the script established in
https://github.com/NixOS/nixpkgs/pull/151399, which I previously used
in https://github.com/NixOS/nixpkgs/pull/168199.

Files came from [this](https://hydra.nixos.org/build/186237511) Hydra build, which used nixpkgs revision ac43c44478 to instantiate:

```
/nix/store/nhjbza9vlcyhp9zxfz6lwpc3m2ghrpzj-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu.drv
```

and then built:

```
/nix/store/fklpm7fy6cp5wz55w0gd8wakyqvzapjx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu
```

I downloaded these files from Hydra with the following commands:

```
STOREPATH=fklpm7fy6cp5wz55w0gd8wakyqvzapjx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu
OPTIONS="--option binary-caches https://cache.nixos.org --option trusted-public-keys cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
nix store add-file \
  --name bootstrap-tools.tar.xz  \
  $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/bootstrap-tools.tar.xz
nix store add-path \
  --name busybox \
  $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/busybox
```

I then prefetched them into `/nix/store` with:

```
$ nix store prefetch-file --executable file:///nix/store/p9lz8r81zp3a4sl2qq2v4j69syjzryn2-busybox
Downloaded 'file:///nix/store/p9lz8r81zp3a4sl2qq2v4j69syjzryn2-busybox' to '/nix/store/a42qf2kf5hychcsw5sz0pvghy9vli1im-p9lz8r81zp3a4sl2qq2v4j69syjzryn2-busybox' (hash 'sha256-jtPEAsht4AUAG4MLK8xocQSfveUR4ppU1lS4bGI1VN4=').

$ nix store prefetch-file file:///nix/store/y4530zpk7ia4szf5cdi4zpyy5lpjv3iv-bootstrap-tools.tar.xz
Downloaded 'file:///nix/store/y4530zpk7ia4szf5cdi4zpyy5lpjv3iv-bootstrap-tools.tar.xz' to '/nix/store/kgzyq9q08nll28ccqjcbv8angq5hyvdp-y4530zpk7ia4szf5cdi4zpyy5lpjv3iv-bootstrap-tools.tar.xz' (hash 'sha256-MpIDnpZUK3M17qlnuoxfnK0EgxRosm3TMW1WfPZ1+jU=').
```

And started the bootstrap with the following command:

```
nix build -f . -L hello
```

As @lovesegfault requested in #151399, here are the the `sha256sum`s of all the `on-server` components for extra verification:

```
$ sha256sum /nix/store/fklpm7fy6cp5wz55w0gd8wakyqvzapjx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/*
3292039e96542b7335eea967ba8c5f9cad04831468b26dd3316d567cf675fa35  /nix/store/fklpm7fy6cp5wz55w0gd8wakyqvzapjx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/bootstrap-tools.tar.xz
3d078dff7b4087d82442937667c91dace3321493aae4d3a4160d046b7eabcc2c  /nix/store/fklpm7fy6cp5wz55w0gd8wakyqvzapjx-stdenv-bootstrap-tools-powerpc64le-unknown-linux-gnu/on-server/busybox
```
2022-08-04 23:05:31 -07:00
github-actions[bot]
2a54df0946
Merge staging-next into staging 2022-08-03 18:02:18 +00:00
Sandro
0e0cc47192
Merge pull request #181550 from reckenrode/update-darwin-bootstrap-tools 2022-08-03 16:12:31 +02:00
Sandro
09cb5e8416
Merge pull request #183507 from Artturin/stdenvmeson1 2022-08-01 23:06:51 +02:00
github-actions[bot]
e8ce2f4a26
Merge master into staging-next 2022-08-01 18:01:20 +00:00
Artturin
9f309c8792 stdenv: mesonFlags: use canExecute in needs_exe_wrapper
this is more accurate
2022-08-01 20:03:47 +03:00
Robert Hensing
12a060dbbc
Merge pull request #183967 from Ericson2314/small-stdenv-cleanup
stdenv: Two small cleanups
2022-08-01 14:37:42 +02:00
github-actions[bot]
15686bdd94
Merge master into staging-next 2022-07-30 18:01:31 +00:00
Alyssa Ross
5330c0a1af
treewide: use isAarch where appropriate 2022-07-30 16:18:27 +00:00
John Ericson
650380ca0b stdenv: Simplify dispatch
This is a change for `powerpc-linux` but that is ancient and I don't
think it matters. The impure bootstrap that was previously assigned to
it has probably bitrotted anyways.
2022-07-29 18:34:02 -04:00
John Ericson
9d59a4df78 stdenv/common-path.nix: move to stdenv/generic/
Do this since it is part of the generic stdenv/`mkDerivation`
infrastructure, rather than being a bootstrapping strategy.
2022-07-29 18:24:00 -04:00
Sergei Trofimovich
f6ee60dde4 gnumake: unconditionally disable guileSupport on bootstrap
Before the change an attempt to use `gnumake.override { guileSupport = true; }`
caused recursion in bootstrap stages as guileSupport pulls in guile and it's
dependencies.

To restore the bootstrap the change unconditionally sets
`guileSupport = false;` for `gnumake`.

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
2022-07-29 19:34:09 +01:00
Adam Joseph
efb35dff8a stdenv/linux/bootstrap-files: add mipsel-linux (mips32) 2022-07-28 14:23:11 -07:00
github-actions[bot]
9153131664
Merge staging-next into staging 2022-07-27 12:02:00 +00:00
github-actions[bot]
9ed58a6fb6
Merge master into staging-next 2022-07-27 12:01:18 +00:00
Vladimír Čunát
a0e2bdeb84
Merge #179628: make-bootstrap-tools-cross: add mips32el 2022-07-27 13:12:10 +02:00
Artturi
79e8669003
Merge pull request #179603 from Artturin/subplacemultiple 2022-07-24 04:14:09 +03:00
Artturi
111abd87ca
Merge pull request #168590 from amjoseph-nixpkgs/stdenv-disallowedReferences
stdenv: use disallowedRequisites to check forbidden requisites
2022-07-18 02:51:23 +03:00
github-actions[bot]
8df1eb061a
Merge master into staging-next 2022-07-17 00:02:14 +00:00
Bernardo Meurer
5fed5fe917
Merge pull request #180223 from amjoseph-nixpkgs/pr/mips64el/bootstrap-file-hashes
pkgs/stdenv/linux: add mips64el bootstrap-files
2022-07-16 16:41:08 -07:00
Randy Eckenrode
8d1c1184e6
make-bootstrap-tools: add pbzx and tbd tools on x86_64-darwin 2022-07-14 21:07:28 -04:00
Artturin
ca2ca8b194 stdenv: start deprecating non-list mesonFlags
the motivation for this is to simplify stdenv and ease the job of
reviewers due to them needing to tell contributors about the defacto
rule that mesonFlags should be a list of strings
2022-07-06 15:23:20 +03:00
Adam Joseph
73811b372e pkgs/stdenv/linux: add mips64el bootstrap-files
This PR adds Hydra-generated bootstrap tarballs for mips64el-linux.  I'll be following the script established in https://github.com/NixOS/nixpkgs/pull/151399, which I previously used in https://github.com/NixOS/nixpkgs/pull/168199.

Files came from [this](https://hydra.nixos.org/build/182757245) Hydra build, which used nixpkgs revision ef3fe254f3 to instantiate:

```
/nix/store/a2bvv663wjnyhq8m7v84aspsd3sgf9h6-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64.drv
```

and then built:

```
/nix/store/aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64
```

I downloaded these files from Hydra with the following commands:

```
STOREPATH=aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64
OPTIONS="--option binary-caches https://cache.nixos.org"
nix store add-file \
  --name bootstrap-tools.tar.xz  \
  $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/bootstrap-tools.tar.xz
nix store add-path \
  --name busybox \
  $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/busybox
```

I then prefetched them into `/nix/store` with:

```
$ nix store prefetch-file --executable file:///nix/store/aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64/on-server/busybox
Downloaded 'file:///nix/store/aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64/on-server/busybox' to '/nix/store/ai30ss23914syz6j8m95arkwffbbx44k-busybox' (hash 'sha256-sTE58ofjqAqX3Xtq1g9wDxzIe6Vo//GHbicfqJoivDI=').

$ nix store prefetch-file  file:///nix/store/aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64/on-server/bootstrap-tools.tar.xz
Downloaded 'file:///nix/store/aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64/on-server/bootstrap-tools.tar.xz' to '/nix/store/nr6zjrbwbxcxs6brf042zsyqllwbwj9v-bootstrap-tools.tar.xz' (hash 'sha256-tTgjeXpd2YgnfP4JvRuO0bXd2j8GqzBcd57JI3wH9x0=').
```

And started the bootstrap with the following command (the `--arg localSystem` is needed because #161159 has not merged):

```
nix build -f . -L hello --arg localSystem '(import ./lib).systems.examples.mips64el-linux-gnuabi64'
```

As @lovesegfault requested in #151399, here are the the `sha256sum`s of all the `on-server` components for extra verification:

```
$ sha256sum /nix/store/aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64/on-server/*
b53823797a5dd988277cfe09bd1b8ed1b5ddda3f06ab305c779ec9237c07f71d  /nix/store/aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64/on-server/bootstrap-tools.tar.xz
598e05abb69b2c1a0db46585cd2131212077c0937ce2a665daf3811f059ae767  /nix/store/aw3dmsrh22831l83vi3q9apg9qi3x8ms-stdenv-bootstrap-tools-mips64el-unknown-linux-gnuabi64/on-server/busybox
```
2022-07-05 12:16:58 -07:00
Artturin
3173c3b6b6 stdenv: start deprecating non-list cmakeFlags
the motivation for this is to simplify stdenv and ease the job of
reviewers due to them needing to tell contributors about the defacto
rule that cmakeFlags should be a list of strings
2022-07-05 04:05:30 +03:00
Adam Joseph
dc5bdd6852 Revert "stdenv: label the ephemeral coreutils-stage4 package"
This reverts commit 23ea8b35da.
2022-07-02 16:56:08 -07:00
Rick van Schijndel
2fcdf54568
Merge pull request #169378 from amjoseph-nixpkgs/stdenv-makeStaticLibraries-do-what-comment-says
stdenv: make stage3.{gmp,mpfr,mpc,isl} do what the comment says
2022-06-30 07:44:05 +02:00
Adam Joseph
ea32f3e523 make-bootstrap-tools-cross: enable Hydra builds of mips32el bootstrap-files
This one-line commit will cause Hydra to build a bootstrap-files
tarball for 32-bit little-endian MIPS.
2022-06-29 14:48:13 -07:00
Artturin
ba1efa71ae stdenv: substituteInPlace: accept multiple filenames
I don't know if getopt is available everywhere, so I did not use it.

in any case, it can be changed to use getopt in the future if
needed.
2022-06-29 22:03:19 +03:00
github-actions[bot]
5b9ee17283
Merge master into staging-next 2022-06-24 00:02:29 +00:00
Sergei Trofimovich
b0f5f5cc96
Merge pull request #178408 from trofi/add-config.enableParallelBuildingByDefault
config.configurePlatformsByDefault: init option
2022-06-23 21:32:59 +00:00
Rick van Schijndel
adafa1cb42
Merge pull request #178626 from a-m-joseph/pr/168413fallout/mktemp
setup.sh: use six `X` characters in `mktemp` invocation
2022-06-23 06:45:40 +02:00
Rick van Schijndel
f104ffc941
Merge pull request #177789 from a-m-joseph/pr/setup/busyboxfind
setup.sh: use -exec rather than -execdir
2022-06-23 06:40:38 +02:00