Without this PR, unlike `RUST_LIB_BACKTRACE=1 cargo run` you won't
get line numbers in backtraces from binaries built with:
```
nix build -f Cargo.nix --arg release false
```
This PR fixes that.
In https://github.com/NixOS/nixpkgs/pull/209870 I tried to unify the
treatment of clang and gcc in cc-wrapper as much as possible.
However it appears that I went too far.
Clang requires -isystem flags in order to be able to find gcc's
libstdc++. Gcc does not need these flags. If they are added,
gfortran will get confused:
https://github.com/NixOS/nixpkgs/pull/209870#issuecomment-1500550903
This commit deunifies the chunk of code that adds the -isystem
flags, and explains why this chunk applies only to clang.
When I authored the nix file in
335a9083b0,
`makeSetupHook` didn't know about `passthru` or `meta`.
So I foisted these attributes on the
derivation with `.overrideAttrs`.
Commits ba895a7da8 and
48034046bf enabled
`makeSetupHook` to receive these attributes directly.
It seems advisable to use that instead of `.overrideAttrs`.
The build.rs script shipped with evdev-sys attempts to detect cross
compilation and uses a completely different codepath which does a
`git fetch` inside the build script. This doesn't work in nixpkgs.
This PR adds a `touch libevdev/.git` to trick the `build.rs` into
thinking that it is not necessary to do a `git fetch`.
Thanks to @figsoda for finding this more-elegant solution to the
problem (my original solution needed to patch `build.rs`):
https://github.com/NixOS/nixpkgs/pull/224893#pullrequestreview-1373809617
Tested on:
- [x] `aarch64-linux` (cross from `x86_64-linux`)
Git dependencies with install scripts are built isolated from the main
package, so their development dependencies are required.
To take advantage of this, #206477 is needed.
This splits prefetch-npm-deps into multiple files, as well as making a
few small changes along the way, such as going from a `HashMap` to a `Vec`
as the container for packages, to deduplicate them more efficently.
AFAIK this doesn't have a noticable effect for typical
Linux executables, but will avoid setting the timestamp
inside the file for Windows executables to the build
date.
Without this change, the `--os` and `--arch` switches are disregarded
for operations involving `skopeo inspect` invocations. This means that,
for example, one cannot fetch Linux images while on macOS.
Unlike most other trivial builders, writeShellApplication has a
relatively heavy checkphase that invokes shellcheck. On my system,
a small writeShellApplication takes over 1 second to build.
This PR allows substitutions and stops preferring local builds for
writeShellApplication.
The Nix-driven bootstrap of gcc resulted in some changes to the
structure of the `libgccjit` outpaths, and also added an additional
output (`libgcc`) to `gcc`.
This commit makes the corresponding changes in the `emacs`
derivation in order to not break emacs.
Emacs is the only user of `libgccjit` in nixpkgs at the moment.
When wrapping `clang` and using a `gccForLibs` whose `libgcc` is in
its own output (rather than the `lib` output), this commit will adds
`-L${gccForLibs.libgcc}/lib` to `cc-ldflags`.
If that flag is not added, `firefox` will fail to compile because it
invokes `clang-wrapper` with `-fuse-ld=lld` and passes `-lgcc_s` to
`lld`, but does not tell `lld` where to find `libgcc_s.so`. In that
situation, firefox will fail to link.
The command
```
nix-build -A tests.trivial-builders.references --show-trace
```
fails eval with
```
in job ‘nixpkgs.tests.trivial-builders.references’:
error: The option `meta.description' does not exist. Definition values:
- In `makeTest parameters': "Run the Nixpkgs trivial builders tests"
```
because `meta.description` and `meta.license` are not valid for
`nixosTest`s (they are valid for `mkDerivation` of course).
This has been causing Hydra eval failures:
https://hydra.nixos.org/jobset/nixos/pr-209870-gcc-external-bootstrap#tabs-errors
Let's fix eval by removing these attributes.
Set the executable bit before running the check phase, so that the check
phase can run the script to test its behaviour.
This aligns with what `concatTextFile` is doing.
Also use explicit `if` statements so that we don't silently ignore
`chmod` failures.
When authoring #220966, I missed that the `//` would take effect solely
in the else branch, fixing hledger-check-fancyassertions, but not
tests.writers.bin.haskell.
This fixes multiple entries being returned from getDesktopParam, e.g. in the
case of localized key names: 'Name', 'Name[de]', and makes this function to
match this key exactly instead of a pattern for the same reason.
The reason is that we can not expect the extended logic run on git
dependencies starting from Cargo 1.68 to be reproducible in future
versions, and thus the output hash would not be sufficiently stable.
https://github.com/rust-lang/cargo/pull/11414
Since we grep for 'workspace', it's possible the script ends up running
on a Cargo.toml that has the word 'workspace' in a comment, but does not
actually use workspaces
This allows packages that require several dotnet versions to build (like
BeatSaberModManager) to properly depend on the dotnet-sdk specific deps.
This in turns avoids having to regenerate the deps of those packages
after each dotnet-sdk update.
This also changes nuget-to-nix to accept a file with a list of
exclusions instead of a folder.
Rust 1.64.0 added support for workspace inheritance, which allows
for crates to inherit values such as dependency version constraints or
package metadata information from their workspaces [0].
This works by having workspace members specify a value as a table, with
`workspace` set to true. Thus, supporting this in importCargoLock is as
simple as walking the crate's Cargo.toml, replacing inherited values
with their workspace counterpart.
This is also what a forthcoming Cargo release will do for `cargo vendor` [1],
but we can get ahead of it ;)
[0]: https://blog.rust-lang.org/2022/09/22/Rust-1.64.0.html#cargo-improvements-workspace-inheritance-and-multi-target-builds
[1]: https://github.com/rust-lang/cargo/pull/11414
The command
```
nix-build -A tests.trivial-builders.references --show-trace
```
fails eval with
```
in job ‘nixpkgs.tests.trivial-builders.references’:
error: The option `meta.description' does not exist. Definition values:
- In `makeTest parameters': "Run the Nixpkgs trivial builders tests"
```
because `meta.description` and `meta.license` are not valid for
`nixosTest`s (they are valid for `mkDerivation` of course).
This has been causing Hydra eval failures:
https://hydra.nixos.org/jobset/nixos/pr-209870-gcc-external-bootstrap#tabs-errors
Let's fix eval by removing these attributes.
Without this change $target-cpp is used unwrapped and is missing
standard header search paths among other things).
Example failure:
$ nix build -f. -L pkgsStatic.netbsd.compat
...
> checking how to run the C preprocessor... x86_64-unknown-linux-musl-cpp
> configure: error: in `/build/cvs-export/tools/compat':
> configure: error: C preprocessor "x86_64-unknown-linux-musl-cpp" fails sanity check
> See `config.log' for more details
This is useful to teach `importCargoLock` how to download crates from a
registry other than crates.io. Specifically, we publish our own crates
to an internal registry and this feature lets us pull from it seamlessly.
If RUSTFLAGS is set in the environment, Cargo will ignore rustflags
settings in its TOML configuration. So setting RUSTFLAGS=-g (like
separateDebugInfo does) to generate debug info breaks
dynamically-linked Rust packages on musl. This breakage is visible
for any packages that call into C dynamic libraries. If the binary is
linked directly to a C dynamic library, it will fail to build, and if
it depends on a Rust library which links a C dynamic library, it will
segfault at runtime when it tries to call a function from the C
library. I noticed this because pkgsMusl.crosvm is broken for this
reason, since it sets separateDebugInfo = true.
It shouldn't be possible to end up with broken binaries just by using
RUSTFLAGS to do something innocuous like enable debug info, so I think
that, even though we liked the approach of modiyfing .cargo/config
better at the time, it's become clear that it's too brittle, and we
should bite the bullet and patch the compiler instead when targetting
musl. It does not appear to be necessary to modify the compiler at
all when cross-compiling /from/ dynamically-linked Musl to another
target, so I'm only checking whether the target system is
dynamically-linked Musl when deciding whether to make the modification
to the compiler.
This reverts commit c2eaaae50d
("cargoSetupHook: pass host config flags"), and implements the
compiler patching approach instead.
This commit adds basic support for tree-sitter in the emacs build,
such that (if the user opts into tree-sitter support), tree-sitter
will be enabled and binary library files for tree-sitter can be
included in the `lib` directory of packages passed to
`emacsWithPackages`. The libraries will be aggregated and included in
treesit-extra-load-path.
The previous pattern for this in the community was to add tree-sitter
libaries by patching emacs's `RUNPATH` with `patchelf` in a post-fixup
phase. However, this has the substantial drawback that two different
emacs installations with different lists of available tree-sitter
libraries must be entirely separate builds. By supplying the
tree-sitter libraries in the wrapping layer of `emacsWithpackages`, it
becomes possible to share a single, more-cacheable "core emacs".
This support defaults to "on" only in emacs 29 and up, since previous
versions do not support tree-sitter out of the box.
`fixupPhase` move `$out/man` to `$out/share/man`. So the information of their location in the dune-project file is outdated which breaks dependencies on packages ( `(package foo)`).
Sometimes it's more ergonomic to set up the build environment in
hooks, to add to the default behaviour rather than replacing it. It's
very surprising that the fetcher works fine with a custom unpackPhase,
but not with custom preUnpack or postUnpack.
Packages that use preUnpack or postUnpack and Cargo FODs seem to be
very rare. I searched Nixpkgs for files containing one of
"cargoHash", "cargoDeps", and "cargoSha256", and one of "preUnpack" or
"postUnpack", and only found two such packages:
python3.pkgs.tokenizers and rustdesk. Neither of their Cargo FOD
hashes are affected by this change. So if that's any indication,
we're unlikely to be breaking many out-of-tree hashes with these
changes either.
Currently, separate-debug-info adds a debug output, and the build fail when it is
not created. the output is only created when at least one elf file is
stripped.
As a result, adding separateDebugInfo = true on a lib will break the
static build (unless the lib also contains an executable). In order to
not have to remember to add an exception every time, let's just create
the debug output unconditionally.
(and gitiles)
This allows fetching a patch from servers that return them
base64-encoded, like this:
fetchpatch {
name = "gcc.patch";
url = "f37ae3b1a8^!?format=TEXT";
decode = "base64 -d";
sha256 = "11j1bqz2p8xrfzgfrylgdvmqs45489c4ckl7l0ra1dpfgbqy94a8";
}
Fixes linker errors while building build.rs where it tries to link libiconv but cannot find it.
Rust executable build for Darwin need libiconv, and indeed buildInputs already has this case handled.
So why is another change needed? Suppose we are cross compiling from Darwin (the build platform) to something else, and the package has a build.rs build script.
The build script is built for the build platform (Darwin) and is also a regular Rust executable, needing libiconv, but due to cross compilation (and strict deps) we need an extra nativeBuildInput.
writeShellApplication currently uses the unwrapped (passthru) attribute
which is simply defined as the ShellCheck Haskell package.
Unfortunately the unwrapped version contains everything and the kitchen
sink, while the bin output of the top-level shellcheck package contains
only the static shellcheck executable.
In other words, by using writeShellApplication, currently 3GB of
packages have to be unnecessarily fetched just to run the checkPhase.
$ nix path-info -Sh $(nix build --print-out-paths --no-link nixpkgs#shellcheck.unwrapped)
/nix/store/23x8702b9kqn0r8swah05ky7w5fnh6m2-ShellCheck-0.9.0 3.0G
$ nix path-info -Sh $(nix build --print-out-paths --no-link nixpkgs#shellcheck.bin)
/nix/store/594izb2jz3c57c7hgxfnb6irypnr4575-shellcheck-0.9.0-bin 45.3M
There is no benefit to using shellcheck.unwrapped in this case.
Therefore, replace shellcheck.unwrapped with lib.getExe shellcheck.
This lets you set the version for the derivation produced from
`buildFHSUserEnvChroot` and `buildFHSUserEnvBubblewrap`.
This can help to make it more clear to end-users to see the versions of
the packages they are using.
`cargoDeps` is already passed as `mkDerivation` arguments, and should
not be `passthru`ed again. This fixes the mismatch of `drv.cargoDeps`
and the actual dependency when the original derivation is overriden.
Previously, you had to provide the path to the deps.nix of the package inside
your Nixpkgs checkout as an argument manually. Now it just does that by default
when no argument is passed.
unpackFile doesn't dereference symlinks if cargoDeps is a directory, and
some cargo builds run into permission issues because the files the
symlinks point to are not writable.
v1 lock files (generated by default by Cargo versions 1.40 and below)
use a single table, `metadata`, to store the checksums of packages.
The primary motivation for doing this now is that we're considering
vendoring all Cargo lock files in Nixpkgs, some packages still use it
(e.g. cargo-asm), and adding support for it doesn't increase the
complexity of the function. No matter the outcome of the vendoring
discussion, this is a nice thing to have because Cargo still supports v1
lock files.
NixOS/nixpkgs#146275 has more discussion on this; the abridged version
is that `lld` defaults to using `--build-id=fast` while GNU `ld` defaults
to `--build-id=sha1`. These differ in length and so
`separate-debug-info.sh`, as of this writing, errors on `lld`'s shorter
`--build-id=fast`-generated hashes.
`lld` offers the following `build-id` styles:
- UUID (random; fast but bad for reproducibility)
- fast (xxhash; fast but shorter hashes)
- user provided hexstring
- SHA1
- MD5
GNU `ld` supports the latter three options, `mold` supports all of these
plus SHA256.
UUID is out because it's not reproducible, fast isn't supported by GNU
`ld`
Using a nix provided (sourced from the output base hash) hash as the
`build-id` seems tempting but would require a little extra work
(we have to include some characteristic of the binary being hashed
so that binaries within a derivation still have unique hashes; it
seems easy to get this wrong; i.e. a path based approach would make
two otherwise identical binaries that reside at different paths have
different `build-id` hashes)
That leaves SHA1 and MD5 and GNU `ld` already defaults to the former.
This commit adds `$NIX_BUILD_ID_STYLE` as an escape hatch, in case any
packages have strong opinions about which hash to use.
----
Note that if/when NixOS/nixpkgs#146275 goes through, this change can be
reverted if linker speed is a priority.
The motivation behind this is to alleviate the problem
described in https://github.com/NixOS/nixpkgs/issues/41340.
I'm not sure if this completely fixes the problem, but it
eliminates one more area where we can exceed command line
length limits.
This is essentially the same change as in #112449,
except for `ld-wrapper.sh` instead of `cc-wrapper.sh`.
However, that change alone was not enough; on macOS the
`ld` provided by `darwin.cctools` fails if you use process
substitution to generate the response file, so I put up a
PR to fix that:
https://github.com/tpoechtrager/cctools-port/pull/131
… and I included a patch referencing that fix so that the
new `ld-wrapper` still works on macOS.
with structuredAttrs lists will be bash arrays which cannot be exported
which will be a issue with some patches and some wrappers like cc-wrapper
this makes it clearer that NIX_CFLAGS_COMPILE must be a string as lists
in env cause a eval failure
Before this change, `nix develop` would often result in all the
shell's dependencies being copied to a remote builder, only for the
remote builder to run a trivial derivation build. This change makes
`nix develop` much faster on systems with remote builders configured.
cc-wrapper has essentially two separate codepaths: the `gccForLibs`
codepath, used only by non-gcc (i.e. clang) compilers, and the
"other" codepath.
This PR allows non-clang compilers to opt-in to the `gccForLibs`
codepath (off by default). To allow this, a new parameter
`ccForLibs` is exposed, since it would be extremely confusing for
gcc to be able to use `gccForLibs` but not do so by default.
Fixes "No such file or directory" error when running prefetch-npm-deps
in its default mode (print hashes, for update scripts etc),
in an environment that doesn't have a `nix` binary (for `nix hash`).
otherwise, eval fails when the experimental no-url-literals feature is activated
unquoted urls are discouraged after https://github.com/NixOS/rfcs/pull/45
not all linkers have a ld binary in bin
also note the '${ld:-}' which allows users to set the ld path with a env
var
> '${foo:-val}' $foo, or val if unset (or null)
