Our gcc_multi and glibc_multi expressions merge together a
32-bit-targeted and 64-bit-targeted gcc. However they do not thread
through the passthru.libgcc from these merged gccs.
This commit corrects that.
It also extends passthru.libgcc to allow a *list* rather than just a
single outpath.
Resolves part of #221891 (at least getting it back to the error
message it gave before).
While searching for something different I wondered why there is a
trivial-builders.nix file next to the trivial-builders directory where
only tests live. Lets fix that.
A directory full of *.nupkg files is a valid nuget source. We do not need mono
and the Nuget command line tool to create this structure. This has two
advantages:
- Nuget is currently broken due to a kernel bug affecting mono (#229476).
Replacing the mkNugetSource implementation allows affected users on 6.1+
kernels compile .NET core packages again.
- It removes mono from the build closure of .NET core packages. .NET core
builds should not depend on .NET framework tools like mono.
There is no equivalent of the `nuget init` command in .NET core. The closest
command is `dotnet nuget push`, which just copies the *.nupkg files around
anyway, just like this PR does with `cp`.
`nuget init` used to extract the *.nuspec files from the nupkgs, this new
implementation doesn't. .NET core doesn't care, but it makes the license
extraction more difficult. What was previously done with find/grep/xml2 is now
a python script (extract-licenses-from-nupkgs.py).
Members of the [package] table in Cargo.toml can be either subtables, or
values like strings and bools. Python is happy to check for membership
of "workspace" in a string, since Python strings are iterables, but if
the value is a bool, Python will throw an exception.
LLD supports Windows-style linker arguments, but these previously
triggered purity check false positives, because it saw that they
started with a '/' and assumed they were paths.
This tweaks the path detection to allow through certain values that
could be paths, but are much more likely to be LINK.EXE-style flags.
The risk of false negatives here is low — the only things we'd now
fail to catch would be attempts to link with libraries in the root
directory, which doesn't happen in practice.
We also teach the wrapper how to apply its purity checks to library
paths specified with the /LIBPATH: option.
Tested that paths we expect to be rejected (like /lib/libfoo.so) still
are.
v1 lockfiles can contain multiple references to the same version of a
package, and these references can contain different `integrity` values,
such as one having SHA-1 and SHA-512, while another just has SHA-512.
Given that HashMap iteration order isn't defined, this causes
reproducibility issues, as a different integrity value could be chosen
each time.
Thanks to @lilyinstarlight for discovering this issue originally, as well
as the idea for the sorting-based implementation.
According to <https://gcc.gnu.org/legacy-ml/gcc-patches/2015-08/msg00836.html>,
all code is position-independent on Windows. Some compilers
apparently warn for -fPIC on Windows, and clang errors:
> clang-15: error: unsupported option '-fPIC' for target 'x86_64-pc-windows-msvc'
I'm guessing the check was hostPlatform instead of targetPlatform by mistake.
In f8ee061247, the fallback installPhase
if fFetchAttrs.installPhase is not provided, became dynamically computed.
Due to operator precedence this had the side effect of appending to
fFetchAttrs.installPhase if it is provided, breaking custom
installPhases altogether.
rustc supports way more platforms than Linux and Darwin. We might not
be able to build it for every platform at the moment, but that's what
meta.broken is for.
There are other platforms that rustc can produce binaries for, but
can't run on itself, so those are listed in the defaults for
buildRustPackage.
If multiple files with the same build id were found, we silently randomly
overwrote one with the other.
Change the order to make the output deterministic, and emit a
warning when overwriting.
* buildGoModule: don't inherit postBuild hook when building go-modules
This is a slight revert of 5ce647b8bf
(#212800).
Inheriting these hooks in the `.go-modules` derivation can be confusing:
One doesn't expect them to run when generating the fixed output modules
derivation, but only on the main derivation. A `postBuild` hook that
adds some files to $out will cause a very hard to debug issue[1].
This commit adds support for a dedicated `modPostBuild` hook that will
be used only by the derivation building `.go-modules`. Additionally,
`go.section.md` now explains these attributes behavior better.
[1]:
https://discourse.nixos.org/t/cant-update-a-go-package-getting-go-inconsistent-vendoring/27063/6
Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
Some Flutter packages require additional attribute values to be added to buildFlutterApplication, to add things like libraries and environment variables.
To prevent duplication in applications that use the packages, a repository of package overrides is kept. buildFlutterApplication will look for package overrides for each dependency, and apply them by calling overrideAttrs on itself.
This allows thing like `flutterBuildFlags` and `runtimeDependencies` to be overridden with `overrideAttrs`.
Attributes that affect dependency retrieval cannot be overridden.
This uses `dart pub deps` (https://dart.dev/tools/pub/cmd/pub-deps) to retrieve information about project dependencies.
It requires a fetch-dart-deps derivation as input.
This wraps Flutter programs with an appropriate LD_LIBRARY_PATH.
For some reason, the RUNPATH of the executable is not used to load dynamic libraries in dart:ffi with DynamicLibrary.open().
This could alternatively be fixed with patchelf --add-needed, but this would cause all the libraries to be opened immediately,
which is not what application authors expect.
The name of the runtimeDependencies argument was chosen to match autoPatchelfHook, which has a similar feature.
This reduces the size of the executable.
It works well with Nix - if the user includes the version in their bug report, the exact debugging symbols required can easily be found.
88275ca6d6 inadvertently stopped the pubspec.lock from changing, as it copied the file before running pub get.
pub get can modify the pubspec.lock to update it to newer formats (for example by adding hashes and updating URLs, see an example diff below). We do not need the modifications at any later stage, so we can preserve the original file.
boolean_selector:
dependency: transitive
description:
name: boolean_selector
+ sha256: "6cfb5af12253eaf2b368f07bacc5a80d1301a071c73360d746b7f2e32d762c66"
+ url: "https://pub.dev"
- url: "https://pub.dartlang.org"
source: hosted
+ version: "2.1.1"
- version: "2.1.0"
Pub does not perform any Git operations on the cached package directly, instead cloning it through a mirror.
The .git directories are not needed, and are a source of non-determinism.
Co-authored-by: FlafyDev <flafyarazi@gmail.com>
The dependency fixed-output derivation now uses the recursive hash mode to avoid tarballing and copying all the files.
https://github.com/NixOS/nix/issues/6660 was ran into during the development of this change. Input references were found in the Git package cache before nukeReferences was used.
It turns out that the mirrors in the Git package cache do not actually need to be preserved, as the SDK does not use them during the build process. They are therefore deleted in the dependency derivation and re-created as blank repositories in the main derivation.
A description of the Git package cache layout can be found here: c890afa1d6/lib/src/source/git.dart (L339)
Files and directories such as .dart_tool, .flutter-plugins, .packages and .pub-cache/hosted/*/.cache will be happily regenerated by the SDK in an offline environment.
The patches being made to these files and directories to remove non-determinism were flawed: They did not handle cases where files only appeared in one run, or cases where tooling versions had an effect on the output.
While this may be beneficial for packages that ship native binaries, this causes issues due to Flutter packages that have inappropriately uploaded build artifacts (such as those from example subprojects) to pub.dev.
Without this PR, unlike `RUST_LIB_BACKTRACE=1 cargo run` you won't
get line numbers in backtraces from binaries built with:
```
nix build -f Cargo.nix --arg release false
```
This PR fixes that.
In https://github.com/NixOS/nixpkgs/pull/209870 I tried to unify the
treatment of clang and gcc in cc-wrapper as much as possible.
However it appears that I went too far.
Clang requires -isystem flags in order to be able to find gcc's
libstdc++. Gcc does not need these flags. If they are added,
gfortran will get confused:
https://github.com/NixOS/nixpkgs/pull/209870#issuecomment-1500550903
This commit deunifies the chunk of code that adds the -isystem
flags, and explains why this chunk applies only to clang.
When I authored the nix file in
335a9083b0,
`makeSetupHook` didn't know about `passthru` or `meta`.
So I foisted these attributes on the
derivation with `.overrideAttrs`.
Commits ba895a7da8 and
48034046bf enabled
`makeSetupHook` to receive these attributes directly.
It seems advisable to use that instead of `.overrideAttrs`.
The build.rs script shipped with evdev-sys attempts to detect cross
compilation and uses a completely different codepath which does a
`git fetch` inside the build script. This doesn't work in nixpkgs.
This PR adds a `touch libevdev/.git` to trick the `build.rs` into
thinking that it is not necessary to do a `git fetch`.
Thanks to @figsoda for finding this more-elegant solution to the
problem (my original solution needed to patch `build.rs`):
https://github.com/NixOS/nixpkgs/pull/224893#pullrequestreview-1373809617
Tested on:
- [x] `aarch64-linux` (cross from `x86_64-linux`)
Git dependencies with install scripts are built isolated from the main
package, so their development dependencies are required.
To take advantage of this, #206477 is needed.
This splits prefetch-npm-deps into multiple files, as well as making a
few small changes along the way, such as going from a `HashMap` to a `Vec`
as the container for packages, to deduplicate them more efficently.
AFAIK this doesn't have a noticable effect for typical
Linux executables, but will avoid setting the timestamp
inside the file for Windows executables to the build
date.
Without this change, the `--os` and `--arch` switches are disregarded
for operations involving `skopeo inspect` invocations. This means that,
for example, one cannot fetch Linux images while on macOS.
Unlike most other trivial builders, writeShellApplication has a
relatively heavy checkphase that invokes shellcheck. On my system,
a small writeShellApplication takes over 1 second to build.
This PR allows substitutions and stops preferring local builds for
writeShellApplication.
The Nix-driven bootstrap of gcc resulted in some changes to the
structure of the `libgccjit` outpaths, and also added an additional
output (`libgcc`) to `gcc`.
This commit makes the corresponding changes in the `emacs`
derivation in order to not break emacs.
Emacs is the only user of `libgccjit` in nixpkgs at the moment.
When wrapping `clang` and using a `gccForLibs` whose `libgcc` is in
its own output (rather than the `lib` output), this commit will adds
`-L${gccForLibs.libgcc}/lib` to `cc-ldflags`.
If that flag is not added, `firefox` will fail to compile because it
invokes `clang-wrapper` with `-fuse-ld=lld` and passes `-lgcc_s` to
`lld`, but does not tell `lld` where to find `libgcc_s.so`. In that
situation, firefox will fail to link.
The command
```
nix-build -A tests.trivial-builders.references --show-trace
```
fails eval with
```
in job ‘nixpkgs.tests.trivial-builders.references’:
error: The option `meta.description' does not exist. Definition values:
- In `makeTest parameters': "Run the Nixpkgs trivial builders tests"
```
because `meta.description` and `meta.license` are not valid for
`nixosTest`s (they are valid for `mkDerivation` of course).
This has been causing Hydra eval failures:
https://hydra.nixos.org/jobset/nixos/pr-209870-gcc-external-bootstrap#tabs-errors
Let's fix eval by removing these attributes.
Set the executable bit before running the check phase, so that the check
phase can run the script to test its behaviour.
This aligns with what `concatTextFile` is doing.
Also use explicit `if` statements so that we don't silently ignore
`chmod` failures.
When authoring #220966, I missed that the `//` would take effect solely
in the else branch, fixing hledger-check-fancyassertions, but not
tests.writers.bin.haskell.
* Ignore relative `file:` paths.
* Support github codeload URLs with `refs/tags/tag` in addition to just `tag`.
* Support https://github.com/owner/repo/archive/ref.tar.gz URLs for git download.
This fixes multiple entries being returned from getDesktopParam, e.g. in the
case of localized key names: 'Name', 'Name[de]', and makes this function to
match this key exactly instead of a pattern for the same reason.
The reason is that we can not expect the extended logic run on git
dependencies starting from Cargo 1.68 to be reproducible in future
versions, and thus the output hash would not be sufficiently stable.
https://github.com/rust-lang/cargo/pull/11414
Since we grep for 'workspace', it's possible the script ends up running
on a Cargo.toml that has the word 'workspace' in a comment, but does not
actually use workspaces
This allows packages that require several dotnet versions to build (like
BeatSaberModManager) to properly depend on the dotnet-sdk specific deps.
This in turns avoids having to regenerate the deps of those packages
after each dotnet-sdk update.
This also changes nuget-to-nix to accept a file with a list of
exclusions instead of a folder.
Rust 1.64.0 added support for workspace inheritance, which allows
for crates to inherit values such as dependency version constraints or
package metadata information from their workspaces [0].
This works by having workspace members specify a value as a table, with
`workspace` set to true. Thus, supporting this in importCargoLock is as
simple as walking the crate's Cargo.toml, replacing inherited values
with their workspace counterpart.
This is also what a forthcoming Cargo release will do for `cargo vendor` [1],
but we can get ahead of it ;)
[0]: https://blog.rust-lang.org/2022/09/22/Rust-1.64.0.html#cargo-improvements-workspace-inheritance-and-multi-target-builds
[1]: https://github.com/rust-lang/cargo/pull/11414
The command
```
nix-build -A tests.trivial-builders.references --show-trace
```
fails eval with
```
in job ‘nixpkgs.tests.trivial-builders.references’:
error: The option `meta.description' does not exist. Definition values:
- In `makeTest parameters': "Run the Nixpkgs trivial builders tests"
```
because `meta.description` and `meta.license` are not valid for
`nixosTest`s (they are valid for `mkDerivation` of course).
This has been causing Hydra eval failures:
https://hydra.nixos.org/jobset/nixos/pr-209870-gcc-external-bootstrap#tabs-errors
Let's fix eval by removing these attributes.
Without this change $target-cpp is used unwrapped and is missing
standard header search paths among other things).
Example failure:
$ nix build -f. -L pkgsStatic.netbsd.compat
...
> checking how to run the C preprocessor... x86_64-unknown-linux-musl-cpp
> configure: error: in `/build/cvs-export/tools/compat':
> configure: error: C preprocessor "x86_64-unknown-linux-musl-cpp" fails sanity check
> See `config.log' for more details
This is useful to teach `importCargoLock` how to download crates from a
registry other than crates.io. Specifically, we publish our own crates
to an internal registry and this feature lets us pull from it seamlessly.
If RUSTFLAGS is set in the environment, Cargo will ignore rustflags
settings in its TOML configuration. So setting RUSTFLAGS=-g (like
separateDebugInfo does) to generate debug info breaks
dynamically-linked Rust packages on musl. This breakage is visible
for any packages that call into C dynamic libraries. If the binary is
linked directly to a C dynamic library, it will fail to build, and if
it depends on a Rust library which links a C dynamic library, it will
segfault at runtime when it tries to call a function from the C
library. I noticed this because pkgsMusl.crosvm is broken for this
reason, since it sets separateDebugInfo = true.
It shouldn't be possible to end up with broken binaries just by using
RUSTFLAGS to do something innocuous like enable debug info, so I think
that, even though we liked the approach of modiyfing .cargo/config
better at the time, it's become clear that it's too brittle, and we
should bite the bullet and patch the compiler instead when targetting
musl. It does not appear to be necessary to modify the compiler at
all when cross-compiling /from/ dynamically-linked Musl to another
target, so I'm only checking whether the target system is
dynamically-linked Musl when deciding whether to make the modification
to the compiler.
This reverts commit c2eaaae50d
("cargoSetupHook: pass host config flags"), and implements the
compiler patching approach instead.
This commit adds basic support for tree-sitter in the emacs build,
such that (if the user opts into tree-sitter support), tree-sitter
will be enabled and binary library files for tree-sitter can be
included in the `lib` directory of packages passed to
`emacsWithPackages`. The libraries will be aggregated and included in
treesit-extra-load-path.
The previous pattern for this in the community was to add tree-sitter
libaries by patching emacs's `RUNPATH` with `patchelf` in a post-fixup
phase. However, this has the substantial drawback that two different
emacs installations with different lists of available tree-sitter
libraries must be entirely separate builds. By supplying the
tree-sitter libraries in the wrapping layer of `emacsWithpackages`, it
becomes possible to share a single, more-cacheable "core emacs".
This support defaults to "on" only in emacs 29 and up, since previous
versions do not support tree-sitter out of the box.
`fixupPhase` move `$out/man` to `$out/share/man`. So the information of their location in the dune-project file is outdated which breaks dependencies on packages ( `(package foo)`).
Sometimes it's more ergonomic to set up the build environment in
hooks, to add to the default behaviour rather than replacing it. It's
very surprising that the fetcher works fine with a custom unpackPhase,
but not with custom preUnpack or postUnpack.
Packages that use preUnpack or postUnpack and Cargo FODs seem to be
very rare. I searched Nixpkgs for files containing one of
"cargoHash", "cargoDeps", and "cargoSha256", and one of "preUnpack" or
"postUnpack", and only found two such packages:
python3.pkgs.tokenizers and rustdesk. Neither of their Cargo FOD
hashes are affected by this change. So if that's any indication,
we're unlikely to be breaking many out-of-tree hashes with these
changes either.
Currently, separate-debug-info adds a debug output, and the build fail when it is
not created. the output is only created when at least one elf file is
stripped.
As a result, adding separateDebugInfo = true on a lib will break the
static build (unless the lib also contains an executable). In order to
not have to remember to add an exception every time, let's just create
the debug output unconditionally.
(and gitiles)
This allows fetching a patch from servers that return them
base64-encoded, like this:
fetchpatch {
name = "gcc.patch";
url = "f37ae3b1a8^!?format=TEXT";
decode = "base64 -d";
sha256 = "11j1bqz2p8xrfzgfrylgdvmqs45489c4ckl7l0ra1dpfgbqy94a8";
}
Fixes linker errors while building build.rs where it tries to link libiconv but cannot find it.
Rust executable build for Darwin need libiconv, and indeed buildInputs already has this case handled.
So why is another change needed? Suppose we are cross compiling from Darwin (the build platform) to something else, and the package has a build.rs build script.
The build script is built for the build platform (Darwin) and is also a regular Rust executable, needing libiconv, but due to cross compilation (and strict deps) we need an extra nativeBuildInput.
writeShellApplication currently uses the unwrapped (passthru) attribute
which is simply defined as the ShellCheck Haskell package.
Unfortunately the unwrapped version contains everything and the kitchen
sink, while the bin output of the top-level shellcheck package contains
only the static shellcheck executable.
In other words, by using writeShellApplication, currently 3GB of
packages have to be unnecessarily fetched just to run the checkPhase.
$ nix path-info -Sh $(nix build --print-out-paths --no-link nixpkgs#shellcheck.unwrapped)
/nix/store/23x8702b9kqn0r8swah05ky7w5fnh6m2-ShellCheck-0.9.0 3.0G
$ nix path-info -Sh $(nix build --print-out-paths --no-link nixpkgs#shellcheck.bin)
/nix/store/594izb2jz3c57c7hgxfnb6irypnr4575-shellcheck-0.9.0-bin 45.3M
There is no benefit to using shellcheck.unwrapped in this case.
Therefore, replace shellcheck.unwrapped with lib.getExe shellcheck.
This lets you set the version for the derivation produced from
`buildFHSUserEnvChroot` and `buildFHSUserEnvBubblewrap`.
This can help to make it more clear to end-users to see the versions of
the packages they are using.
`cargoDeps` is already passed as `mkDerivation` arguments, and should
not be `passthru`ed again. This fixes the mismatch of `drv.cargoDeps`
and the actual dependency when the original derivation is overriden.
Previously, you had to provide the path to the deps.nix of the package inside
your Nixpkgs checkout as an argument manually. Now it just does that by default
when no argument is passed.
unpackFile doesn't dereference symlinks if cargoDeps is a directory, and
some cargo builds run into permission issues because the files the
symlinks point to are not writable.
v1 lock files (generated by default by Cargo versions 1.40 and below)
use a single table, `metadata`, to store the checksums of packages.
The primary motivation for doing this now is that we're considering
vendoring all Cargo lock files in Nixpkgs, some packages still use it
(e.g. cargo-asm), and adding support for it doesn't increase the
complexity of the function. No matter the outcome of the vendoring
discussion, this is a nice thing to have because Cargo still supports v1
lock files.
NixOS/nixpkgs#146275 has more discussion on this; the abridged version
is that `lld` defaults to using `--build-id=fast` while GNU `ld` defaults
to `--build-id=sha1`. These differ in length and so
`separate-debug-info.sh`, as of this writing, errors on `lld`'s shorter
`--build-id=fast`-generated hashes.
`lld` offers the following `build-id` styles:
- UUID (random; fast but bad for reproducibility)
- fast (xxhash; fast but shorter hashes)
- user provided hexstring
- SHA1
- MD5
GNU `ld` supports the latter three options, `mold` supports all of these
plus SHA256.
UUID is out because it's not reproducible, fast isn't supported by GNU
`ld`
Using a nix provided (sourced from the output base hash) hash as the
`build-id` seems tempting but would require a little extra work
(we have to include some characteristic of the binary being hashed
so that binaries within a derivation still have unique hashes; it
seems easy to get this wrong; i.e. a path based approach would make
two otherwise identical binaries that reside at different paths have
different `build-id` hashes)
That leaves SHA1 and MD5 and GNU `ld` already defaults to the former.
This commit adds `$NIX_BUILD_ID_STYLE` as an escape hatch, in case any
packages have strong opinions about which hash to use.
----
Note that if/when NixOS/nixpkgs#146275 goes through, this change can be
reverted if linker speed is a priority.
The motivation behind this is to alleviate the problem
described in https://github.com/NixOS/nixpkgs/issues/41340.
I'm not sure if this completely fixes the problem, but it
eliminates one more area where we can exceed command line
length limits.
This is essentially the same change as in #112449,
except for `ld-wrapper.sh` instead of `cc-wrapper.sh`.
However, that change alone was not enough; on macOS the
`ld` provided by `darwin.cctools` fails if you use process
substitution to generate the response file, so I put up a
PR to fix that:
https://github.com/tpoechtrager/cctools-port/pull/131
… and I included a patch referencing that fix so that the
new `ld-wrapper` still works on macOS.
with structuredAttrs lists will be bash arrays which cannot be exported
which will be a issue with some patches and some wrappers like cc-wrapper
this makes it clearer that NIX_CFLAGS_COMPILE must be a string as lists
in env cause a eval failure
Before this change, `nix develop` would often result in all the
shell's dependencies being copied to a remote builder, only for the
remote builder to run a trivial derivation build. This change makes
`nix develop` much faster on systems with remote builders configured.
cc-wrapper has essentially two separate codepaths: the `gccForLibs`
codepath, used only by non-gcc (i.e. clang) compilers, and the
"other" codepath.
This PR allows non-clang compilers to opt-in to the `gccForLibs`
codepath (off by default). To allow this, a new parameter
`ccForLibs` is exposed, since it would be extremely confusing for
gcc to be able to use `gccForLibs` but not do so by default.
Fixes "No such file or directory" error when running prefetch-npm-deps
in its default mode (print hashes, for update scripts etc),
in an environment that doesn't have a `nix` binary (for `nix hash`).
otherwise, eval fails when the experimental no-url-literals feature is activated
unquoted urls are discouraged after https://github.com/NixOS/rfcs/pull/45
