Commit Graph

464699 Commits

Author SHA1 Message Date
Adam Joseph
5f57c2e0f9 pkgs/test/stdenv/default.nix: add gcc-stageCompare
This commit adds a derivation `gcc-stageCompare` to
`pkgs/test/stdenv/default.nix`.

It is important to always build this derivation whenever building
`stdenv`!  Because we are using a Nix-driven bootstrap instead of
gcc's built-in `--enable-bootstrap`, the `gcc` derivation no longer
performs the post-self-compilation sanity check.  You must build
this derivation in order to perform that sanity check.

The major benefit of this new approach is that the sanity check
(which involves a third compilation of gcc) can be performed
*concurrently* with all packages that depend on `stdenv`, rather
than serially.  Since `stdenv` has very little derivation-level
parallelism it cannot take advantage of more than one or perhaps two
builders.  If you have three or more builders this commit will
reduce the time-to-rebuild-stdenv by around 20% (one of three gcc
rebuilds is removed from the critical path, and stdenv's build time
is dominated by roughly 3*gcc + 1*binutils + 1*bison-test-suite).

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
2023-04-02 13:49:53 -07:00
Adam Joseph
96588eb3de gcc: add common/checksum.nix
This commit adds `gcc/common/checksum.nix`, which contains code
common to both gcc11 and gcc12, implementing the `enableChecksum`
feature.

When gcc's built-in bootstrap (`--enable-bootstrap`) is used, gcc
compiles itself three times and compares a hash of the unlinked `.o`
files from the second and third compilation.  The
`enableChecksum=true` parameter performs the same comparison as part
of the `postInstall` phase.

Notably, `enableChecksum=true` can be used with `enableBootstrap=false`.

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
2023-04-02 13:49:53 -07:00
Adam Joseph
fed2300bea unpack-bootstrap-tools.sh: patchelf libgcc_s.so.1 2023-04-02 13:49:53 -07:00
Adam Joseph
d7fe0a5548 make-bootstrap-tools.nix: use a patchelf built with -static-{libgcc,libstdc++}
Our bootstrap-files unpacker has always relied on a lot of unstated
assumptions, one of them being that every library has a DT_NEEDED
for librt.so, so patchelf'ing something into the RUNPATH into
librt.so means that it will be searched for every library load in
all of the bootstrap-files.

Unfortunately that assumption is not true for libgcc.

This causes problems, because patchelf links against libgcc (and
against libstdc++, which links against libgcc).  So we can't use
patchelf on libgcc, because it needs libgcc, so patchelf doesn't
work until libgcc is patchelfed.

The robust solution here is to use static linking for the copy of
patchelf that is shipped with the bootstrap-files.  We don't have to
go all the way to a statically linked libc; just -static-libgcc and
-static-libstdc++ are enough to break the circular dependency.
2023-04-02 13:49:53 -07:00
Adam Joseph
86ca0faff7 make-bootstrap-tools.nix: cp libgcc_s without -d
We do not want to preserve the symlinks from libgcc_s, since they
point to another outpath.  We want to copy from that outpath rather
than link to it.
2023-04-02 13:49:53 -07:00
Adam Joseph
c6bd37a691 make-bootstrap-tools.nix: ship libisl.so
Now that we've dropped the
gcc-links-statically-to-lib{isl,mpfr,mpc,gmp} hack, our gcc needs
libisl.so.  Let's add it to the bootstrap-files.
2023-04-02 13:49:53 -07:00
Adam Joseph
7553d0fe29 stdenv: Nix-driven bootstrap of gcc
#### Summary

By default, when you type `make`, GCC will compile itself three
times.  This PR inhibits that behavior by configuring GCC with
`--disable-bootstrap`, and reimplements the triple-rebuild using
Nix rather than `make`/`sh`.

 #### Immediate Benefits

- Allow `gcc11` and `gcc12` on `aarch64` (without needing new
  `bootstrapFiles`)
- Faster stdenv rebuilds: the third compilation of gcc
  (i.e. stageCompare) is no longer a `drvInput` of the final stdenv.
  This allows Nix to build stageCompare in parallel with the rest of
  nixpkgs instead of in series.
- No more copying `libgcc_s` out of the bootstrap-files or other
  derivations
- No more Frankenstein compiler: the final gcc and the libraries it
  links against (mpfr, mpc, isl, glibc) are all built by the same
  compiler (xgcc) instead of a mixture of the bootstrapFiles'
  compiler and xgcc.
- No more [static lib{mpfr,mpc,gmp,isl}.a hack]
- Many other small `stdenv` hacks eliminated
- `gcc` and `clang` share the same codepath for more of `cc-wrapper`.

 #### Future Benefits

- This should allow using a [foreign] `bootstrap-files` so long as
  `hostPlatform.canExecute bootstrapFiles`.
- This should allow each of the libraries that ship with `gcc`
  (lib{backtrace, atomic, cc1, decnumber, ffi, gomp, iberty,
  offloadatomic, quadmath, sanitizer, ssp, stdc++-v3, vtv}) to be
  built in separate (one-liner) derivations which `inherit src;`
  from `gcc`, much like https://github.com/NixOS/nixpkgs/pull/132343

 #### Incorporates

- https://github.com/NixOS/nixpkgs/pull/210004
- https://github.com/NixOS/nixpkgs/pull/36948 (unreverted)
- https://github.com/NixOS/nixpkgs/pull/210325
- https://github.com/NixOS/nixpkgs/pull/210118
- https://github.com/NixOS/nixpkgs/pull/210132
- https://github.com/NixOS/nixpkgs/pull/210109
- https://github.com/NixOS/nixpkgs/pull/213909
- https://github.com/NixOS/nixpkgs/pull/216136
- https://github.com/NixOS/nixpkgs/pull/216237
- https://github.com/NixOS/nixpkgs/pull/210019
- https://github.com/NixOS/nixpkgs/pull/216232
- https://github.com/NixOS/nixpkgs/pull/216016
- https://github.com/NixOS/nixpkgs/pull/217977
- https://github.com/NixOS/nixpkgs/pull/217995

 #### Closes

- Closes #108305
- Closes #108111
- Closes #201254
- Closes #208412

 #### Credits

This project was made possible by three important insights, none of
which were mine:

1. @ericson2314 was the first to advocate for this change, and
   probably the first to appreciate its advantages.  Nix-driven
   (external) bootstrap is "cross by default".

2. @trofi has figured out a lot about how to get gcc to not mix up
   the copy of `libstdc++` that it depends on with the copy that it
   builds, by moving the `bootstrapFiles`' `libstdc++` into a
   [versioned directory].  This allows a Nix-driven bootstrap of gcc
   without the final gcc would still having references to the
   `bootstrapFiles`.

3. Using the undocumented variable [`user-defined-trusted-dirs`]
   when building glibc.  When glibc `dlopen()`s `libgcc_s.so`, it
   uses a completely different and totally special set of rules for
   finding `libgcc_s.so`.  This trick is the only way we can put
   `libgcc_s.so` in its own separate outpath without creating
   circular dependencies or dependencies on the bootstrapFiles.  I
   would never have guessed to use this (or that it existed!) if it
   were not for a [comment in guix] which @Mic92 [mentioned].

My own role in this PR was basically: being available to go on a
coding binge at an opportune moment, so we wouldn't waste a
[crisis].

[aarch64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662822938
[amd64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662825857
[nonexistent sysroot]: https://github.com/NixOS/nixpkgs/pull/210004
[versioned directory]: https://github.com/NixOS/nixpkgs/pull/209054
[`user-defined-trusted-dirs`]: https://sourceware.org/legacy-ml/libc-help/2013-11/msg00026.html
[comment in guix]: 5e4ec82181/gnu/packages/gcc.scm (L253)
[mentioned]: https://github.com/NixOS/nixpkgs/pull/210112#issuecomment-1379608483
[crisis]: https://github.com/NixOS/nixpkgs/issues/108305
[foreign]: https://github.com/NixOS/nixpkgs/pull/170857#issuecomment-1170558348
[static lib{mpfr,mpc,gmp,isl}.a hack]: 2f1948af9c/pkgs/stdenv/linux/default.nix (L380)
2023-04-02 13:49:41 -07:00
Adam Joseph
fdd49f1bcd gcc/{11,12}: use lib.pipe
This commit has no effect on eval.  It simply reorganizes the
`gcc11` and `gcc12` expressions so they apply a list of
`overrideAttrs`.  The list is currently empty.
2023-04-02 03:04:21 -07:00
Adam Joseph
70d34589bd stdenv/linux: factor out commonGccOverrides
This commit has no effect on eval.  It simply factors out a common
subexpression.
2023-04-02 03:04:20 -07:00
Adam Joseph
87225ff54d ccache-links: inherit version to keep cc-wrapper happy
The Nix-driven bootstrap of gcc expects that `stdenv.cc.cc` has a
`version` attribute, except if `stdenv.cc` is the `bootstrapFiles`.

This commit causes `ccache-links`'s `cc-wrapper` to `inherit
version` in order to satisfy that requirement.  Without this commit,
ofborg CI will fail.

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
2023-04-02 03:04:20 -07:00
Adam Joseph
39c7885cd9 cc-wrapper: if isClang, add -L${gccForLibs.libgcc}/lib
When wrapping `clang` and using a `gccForLibs` whose `libgcc` is in
its own output (rather than the `lib` output), this commit will adds
`-L${gccForLibs.libgcc}/lib` to `cc-ldflags`.

If that flag is not added, `firefox` will fail to compile because it
invokes `clang-wrapper` with `-fuse-ld=lld` and passes `-lgcc_s` to
`lld`, but does not tell `lld` where to find `libgcc_s.so`.  In that
situation, firefox will fail to link.
2023-04-02 03:04:20 -07:00
Adam Joseph
410271448c test-defaultPkgConfigPackages.nix: filter out recurseForDerivations
The command at the top of this file fails to evaluate:

```
$ nix-build -A tests.pkg-config.defaultPkgConfigPackages
in job ‘nixpkgs.tests.pkg-config.defaultPkgConfigPackages.tests-combined.x86_64-linux’:
error: pkg-config module `recurseForDerivations` is not defined to be a derivation. Please check the attribute value for `recurseForDerivations` in `pkgs/top-level/pkg-config-packages.nix` in Nixpkgs.
```

This is also causing eval errors on Hydra:

  https://hydra.nixos.org/jobset/nixos/pr-209870-gcc-external-bootstrap#tabs-errors

Let's filter out `recurseForDerivations=true` from the attrset,
since it exists mainly as a flag to signal special handling when
recursing.
2023-04-02 03:04:20 -07:00
Adam Joseph
3b49fb2ab6 trivial-builders/test/references.nix: fix eval
The command

```
nix-build -A tests.trivial-builders.references --show-trace
```

fails eval with

```
in job ‘nixpkgs.tests.trivial-builders.references’:
error: The option `meta.description' does not exist. Definition values:
       - In `makeTest parameters': "Run the Nixpkgs trivial builders tests"
```

because `meta.description` and `meta.license` are not valid for
`nixosTest`s (they are valid for `mkDerivation` of course).

This has been causing Hydra eval failures:

  https://hydra.nixos.org/jobset/nixos/pr-209870-gcc-external-bootstrap#tabs-errors

Let's fix eval by removing these attributes.
2023-04-02 03:04:20 -07:00
github-actions[bot]
867440983f
Merge staging-next into staging 2023-03-15 06:01:43 +00:00
github-actions[bot]
068a7415f8
Merge master into staging-next 2023-03-15 06:01:09 +00:00
Nick Cao
c5887df742
Merge pull request #221239 from marsam/update-1password
_1password: 2.14.0 -> 2.15.0
2023-03-15 13:08:58 +08:00
Nick Cao
2d444979e5
Merge pull request #221254 from r-ryantm/auto-update/konstraint
konstraint: 0.25.1 -> 0.26.0
2023-03-15 13:07:52 +08:00
Nick Cao
705d0991b5
Merge pull request #221255 from r-ryantm/auto-update/cdk-go
cdk-go: 1.5.1 -> 1.5.2
2023-03-15 13:07:30 +08:00
Nick Cao
75f1996001
Merge pull request #221257 from figsoda/argc
argc: 0.13.0 -> 0.14.0
2023-03-15 13:06:50 +08:00
Nick Cao
8b3a559faa
Merge pull request #221262 from r-ryantm/auto-update/spaceship-prompt
spaceship-prompt: 4.13.2 -> 4.13.3
2023-03-15 13:06:25 +08:00
Nick Cao
a2d0d0d19a
Merge pull request #221263 from r-ryantm/auto-update/ginkgo
ginkgo: 2.9.0 -> 2.9.1
2023-03-15 13:05:19 +08:00
Nick Cao
6e6c88c429
Merge pull request #220932 from NickCao/antlr
antlr4_12: init at 4.12.0
2023-03-15 13:04:12 +08:00
R. Ryantm
dc531f9af8 ginkgo: 2.9.0 -> 2.9.1 2023-03-15 04:18:53 +00:00
R. Ryantm
1dc043677c spaceship-prompt: 4.13.2 -> 4.13.3 2023-03-15 04:15:03 +00:00
Chris Hodapp
1a8edfe192
emacs: Add basic tree-sitter support (#219559)
This commit adds basic support for tree-sitter in the emacs build,
such that (if the user opts into tree-sitter support), tree-sitter
will be enabled and binary library files for tree-sitter can be
included in the `lib` directory of packages passed to
`emacsWithPackages`. The libraries will be aggregated and included in
treesit-extra-load-path.

The previous pattern for this in the community was to add tree-sitter
libaries by patching emacs's `RUNPATH` with `patchelf` in a post-fixup
phase. However, this has the substantial drawback that two different
emacs installations with different lists of available tree-sitter
libraries must be entirely separate builds. By supplying the
tree-sitter libraries in the wrapping layer of `emacsWithpackages`, it
becomes possible to share a single, more-cacheable "core emacs".

This support defaults to "on" only in emacs 29 and up, since previous
versions do not support tree-sitter out of the box.
2023-03-15 16:51:29 +13:00
github-actions[bot]
28b44d37a2 terraform-providers.yandex: 0.86.0 → 0.87.0 2023-03-15 13:47:35 +10:00
github-actions[bot]
63fb7c48ae terraform-providers.tencentcloud: 1.79.14 → 1.79.15 2023-03-15 13:47:35 +10:00
github-actions[bot]
b1b21ead4d terraform-providers.talos: 0.1.1 → 0.1.2 2023-03-15 13:47:35 +10:00
github-actions[bot]
15a9d47764 terraform-providers.spotinst: 1.105.0 → 1.106.0 2023-03-15 13:47:35 +10:00
github-actions[bot]
3b6694fee4 terraform-providers.ns1: 2.0.0 → 2.0.2 2023-03-15 13:47:35 +10:00
github-actions[bot]
7643b2b838 terraform-providers.ksyun: 1.3.66 → 1.3.67 2023-03-15 13:47:35 +10:00
github-actions[bot]
7cb823d01d terraform-providers.github: 5.18.0 → 5.18.3 2023-03-15 13:47:35 +10:00
github-actions[bot]
fa4405bd65 terraform-providers.cloudamqp: 1.24.0 → 1.24.1 2023-03-15 13:47:35 +10:00
figsoda
e1a0bbe252 argc: 0.13.0 -> 0.14.0
Diff: https://github.com/sigoden/argc/compare/v0.13.0...v0.14.0

Changelog: https://github.com/sigoden/argc/releases/tag/v0.14.0
2023-03-14 23:06:34 -04:00
R. Ryantm
0f47213252 cdk-go: 1.5.1 -> 1.5.2 2023-03-15 02:20:14 +00:00
R. Ryantm
6838ade916 konstraint: 0.25.1 -> 0.26.0 2023-03-15 02:09:44 +00:00
Anderson Torres
abb2ade261
Merge pull request #221154 from r-ryantm/auto-update/zchunk
zchunk: 1.2.3 -> 1.3.0
2023-03-14 22:20:26 -03:00
Mario Rodas
ec639cdff3
Merge pull request #221176 from marsam/update-plpgsql_check
postgresqlPackages.plpgsql_check: 2.3.0 -> 2.3.3
2023-03-14 20:15:09 -05:00
Martin Weinelt
3b8cf43350
Merge remote-tracking branch 'origin/master' into staging-next
Conflicts:
- pkgs/development/python-modules/imageio/default.nix
- pkgs/development/python-modules/pytest-order/default.nix
2023-03-15 02:01:15 +01:00
superherointj
1c03223e5c
Merge pull request #220543 from vbgl/ocaml-duration-0.2.1
ocamlPackages.duration: 0.2.0 → 0.2.1
2023-03-14 21:25:48 -03:00
superherointj
920b71a22c
Merge pull request #221230 from trofi/SDL2_mixer-fix-timidity
SDL2_mixer: fix path to timidity.cfg
2023-03-14 21:16:51 -03:00
github-actions[bot]
a3b786aa94
Merge staging-next into staging 2023-03-15 00:03:06 +00:00
Mario Rodas
b16f2a7561
Merge pull request #220736 from marsam/update-cloud-nuke
cloud-nuke: 0.21.0 -> 0.27.0
2023-03-14 18:59:03 -05:00
Fabian Affolter
a2963c6b42
Merge pull request #221235 from fabaff/peaqevcore-bump
python310Packages.peaqevcore: 13.0.1 -> 13.1.1
2023-03-15 00:58:06 +01:00
Fabian Affolter
06b187b0ee
Merge pull request #221236 from fabaff/pontos-bump
python310Packages.pontos: 23.3.3 -> 23.3.5
2023-03-15 00:57:54 +01:00
Mario Rodas
a239c7d2ee
Merge pull request #218114 from marsam/update-ruby
ruby_3_1: 3.1.2 -> 3.1.3, ruby_3_2: init at 3.2.1
2023-03-14 18:48:01 -05:00
Weijia Wang
ce289aad52
Merge pull request #220943 from r-ryantm/auto-update/python310Packages.pytest-order
python310Packages.pytest-order: 1.0.1 -> 1.1.0
2023-03-15 01:27:44 +02:00
Fabian Affolter
d03560dc0a python310Packages.pontos: 23.3.3 -> 23.3.5
Diff: https://github.com/greenbone/pontos/compare/refs/tags/v23.3.3...v23.3.5

Changelog: https://github.com/greenbone/pontos/releases/tag/v23.3.5
2023-03-15 00:26:50 +01:00
Weijia Wang
aba12ee4cd
Merge pull request #221232 from wegank/xen-cleanup
xen: cleanup
2023-03-15 01:26:17 +02:00
Fabian Affolter
628036113e python310Packages.peaqevcore: 13.0.1 -> 13.1.1 2023-03-15 00:20:37 +01:00