Rust 1.64.0 added support for workspace inheritance, which allows
for crates to inherit values such as dependency version constraints or
package metadata information from their workspaces [0].
This works by having workspace members specify a value as a table, with
`workspace` set to true. Thus, supporting this in importCargoLock is as
simple as walking the crate's Cargo.toml, replacing inherited values
with their workspace counterpart.
This is also what a forthcoming Cargo release will do for `cargo vendor` [1],
but we can get ahead of it ;)
[0]: https://blog.rust-lang.org/2022/09/22/Rust-1.64.0.html#cargo-improvements-workspace-inheritance-and-multi-target-builds
[1]: https://github.com/rust-lang/cargo/pull/11414
This is useful to teach `importCargoLock` how to download crates from a
registry other than crates.io. Specifically, we publish our own crates
to an internal registry and this feature lets us pull from it seamlessly.
If RUSTFLAGS is set in the environment, Cargo will ignore rustflags
settings in its TOML configuration. So setting RUSTFLAGS=-g (like
separateDebugInfo does) to generate debug info breaks
dynamically-linked Rust packages on musl. This breakage is visible
for any packages that call into C dynamic libraries. If the binary is
linked directly to a C dynamic library, it will fail to build, and if
it depends on a Rust library which links a C dynamic library, it will
segfault at runtime when it tries to call a function from the C
library. I noticed this because pkgsMusl.crosvm is broken for this
reason, since it sets separateDebugInfo = true.
It shouldn't be possible to end up with broken binaries just by using
RUSTFLAGS to do something innocuous like enable debug info, so I think
that, even though we liked the approach of modiyfing .cargo/config
better at the time, it's become clear that it's too brittle, and we
should bite the bullet and patch the compiler instead when targetting
musl. It does not appear to be necessary to modify the compiler at
all when cross-compiling /from/ dynamically-linked Musl to another
target, so I'm only checking whether the target system is
dynamically-linked Musl when deciding whether to make the modification
to the compiler.
This reverts commit c2eaaae50d
("cargoSetupHook: pass host config flags"), and implements the
compiler patching approach instead.
Sometimes it's more ergonomic to set up the build environment in
hooks, to add to the default behaviour rather than replacing it. It's
very surprising that the fetcher works fine with a custom unpackPhase,
but not with custom preUnpack or postUnpack.
Packages that use preUnpack or postUnpack and Cargo FODs seem to be
very rare. I searched Nixpkgs for files containing one of
"cargoHash", "cargoDeps", and "cargoSha256", and one of "preUnpack" or
"postUnpack", and only found two such packages:
python3.pkgs.tokenizers and rustdesk. Neither of their Cargo FOD
hashes are affected by this change. So if that's any indication,
we're unlikely to be breaking many out-of-tree hashes with these
changes either.
Fixes linker errors while building build.rs where it tries to link libiconv but cannot find it.
Rust executable build for Darwin need libiconv, and indeed buildInputs already has this case handled.
So why is another change needed? Suppose we are cross compiling from Darwin (the build platform) to something else, and the package has a build.rs build script.
The build script is built for the build platform (Darwin) and is also a regular Rust executable, needing libiconv, but due to cross compilation (and strict deps) we need an extra nativeBuildInput.
`cargoDeps` is already passed as `mkDerivation` arguments, and should
not be `passthru`ed again. This fixes the mismatch of `drv.cargoDeps`
and the actual dependency when the original derivation is overriden.
unpackFile doesn't dereference symlinks if cargoDeps is a directory, and
some cargo builds run into permission issues because the files the
symlinks point to are not writable.
v1 lock files (generated by default by Cargo versions 1.40 and below)
use a single table, `metadata`, to store the checksums of packages.
The primary motivation for doing this now is that we're considering
vendoring all Cargo lock files in Nixpkgs, some packages still use it
(e.g. cargo-asm), and adding support for it doesn't increase the
complexity of the function. No matter the outcome of the vendoring
discussion, this is a nice thing to have because Cargo still supports v1
lock files.
Used in cases where you need to get the vendor of a target. Such as when
you need to perform dependency resolution outside of Cargo (eg in
Kolloch's crate2nix).
Currently cargo-setup-hook instructs the builder upon cargoSha256 or
cargoHash being out-of-date compared to the Cargo.lock file.
The instructions can be simplified a bit, because nowadays it is fine to
keep a hash empty, instead of filling it with
`0000000000000000000000000000000000000000000000000000`.
Nix nowadays outputs SRI hashes, which should usually be placed in
`cargoHash` instead of `cargoSha256`, but the instructions are still
only referring to `cargoSha256`.
Lastly, the output of Nix doesn't include `got: sha256: ` anymore, as it
now outputs `got: sha256-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=`.
It would be nice to make it clear that the trailing `=` is important as
well, so the full example SRI hash is mentioned.
Fixes#204051. I have tried this on the reproducer stated in the ticket.
```
[nix-develop]$ $(nix-build -I nixpkgs=/home/shana/programming/nixpkgs --no-out-link)/tests/foo
running 1 test
test check_module_name ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
```
This change switches to using GCC 11 by default on aarch64-linux, as well as passing `-lgcc` to the linker, per #201485.
See #201254 and #208412 for wider context on the issue.
This reverts commit b6fc00b8f4.
Rust 1.66.0 contains a fix for libiconv being linked unconditionally on macOS, but this only applies to packages that don't depend on older versions of `libc`.
For now, let's go back to including libiconv in `buildInputs` by default for packages that use `buildRustPackage`. As packages bump their `libc` versions, we can eventually stop including it by default, and manually add it where needed.
There are two problems: first that we end up splitting on spaces in the
loop. Even when that is fixed, we still would split on spaces in the
`export` inside the loop. We need to guard against both.
Fixes#199298
Confirmed that it fixes the case mentioned in the ticket:
```console
[nix-develop]$ $(nix-build -I nixpkgs=/home/shana/programming/nixpkgs Cargo.nix -A rootCrate.build --no-out-link)/bin/nix-rustc-env-escape-repro
Expecting three words, got: first second third
```
I think this is going to cause a rebuild of every Rust package even if
they were unaffected, not much we can do here.
This breaks the builder when a nix-shell or keepBuildTree is used. The
issue occurs because paths to cargo lockfiles are read with NIX_BUILD_TOP,
which is not reliable.
This breaks a nix-shell because NIX_BUILD_TOP simply is not set, causing
an invalid path to be used. This can be worked around using
NIX_BUILD_TOP=$PWD, but that obviously is not great.
This breaks keepBuildTree because it changes the working directory to a
different path than NIX_BUILD_TOP. Since the lockfiles are copied based
on the working directory, but read based on NIX_BUILD_TOP, this causes
the hook to not be able to find them.
This was solved by both reading these files based on the working directory,
using absolute paths to avoid having to traverse back in the directory tree.
Fixes: #138554
Rust binaries are unconditionally linked to libiconv on Darwin (see https://github.com/rust-lang/libc/issues/2870). We already add it as a dependency in `buildRustPackage`, so let's go a step further and propagate it.
`toTargetArch` in `pkgs/build-support/rust/lib/default.nix` is used to
set `CARGO_CFG_TARGET_ARCH`. This environment variable is supposed to
be the `<arch>` portion of an LLVM-style platform name:
```
<arch><sub>-<kernel>-<libc><abi>
```
Note that the pointer-width (the "64" in "x86_64" and "mips64") is
part of `<arch>`, but the endianness (the `_be` in `aarch64_be`) is
*not*.
Unfortunately at the moment nixpkgs' parsed `cpuType` has no way to
query for the three subparts (name, pointer-width, and
subarch/endianness), nor any way to ask for just the first two parts.
For now, this commit simply fixes the problem in the two cases that
matter: `mips64el` and `powerpc64le`, which I believe are the only two
platforms supported by both rust and nixpkgs which have a
"subarchitecture".
Tell rust if we want our binaries linked statically or dynamically.
Otherwise the compiler will always produce statically linked binaries for musl
targets, as this is the default.
We are replicating one mechanism behind `-Z build-std`.
There isn't yet crate2nix support for this, but one can (and I do) add
the missing stdlib deps (for this feature to pick up) with overrides.
With Rust 1.61, it is necessary to link to external static/dynamic libaries
when building the rlib that uses them, rather than when linking the final
binary. In fact, it is no longer necessary to specify the libraries to link
when building the final binary, but the library search path flags must still
be included.
The old logic flow had the structure
if ( … ) {
if ( … ) {
…
} else {
…
}
} else {
…
}
which is quite hard to follow in Nix. Instead we ensure that no if
expression is inside a then branch.
This change is zero rebuild, as no logic was changed.
This parameter is being set to `$NIX_BUILD_CORES` by default. This is a
standard practice but there's a suspicion that this can produce broken
builds. For some details see
https://github.com/cargo2nix/cargo2nix/issues/184 . As a
work-around/test, it'd be good if codegen-units can be set to something
constant, such as `1`. This PR allows it.
Note that the default of `$NIX_BUILD_CORES` is preserved so this MR
causes no change in default behaviour and no rebuilds.
These features are internal-only, have special characters that bash
doesn't support in variable names, and aren't normally given
environment variables by cargo as far as I can tell.
Some crates do not have a Cargo.toml at the top-level, but only in
nested directories. Before this change importCargoLock used to fail with:
error: manifest path `/nix/store/some-store-path/Cargo.toml` does not exist
- `toRustTarget` and friends pulled out from rust tools into rust
library. Since they don't depend on any packages they can be more
widely useable.
- `build-rust-package` gets its own directory
- `fetch-cargo-tarball` gets its own directory
In restricted mode (and therefore with flakes) `builtins.readFile` may not be the result of `builtins.toFile`,
making it impossible to use a generated lockFile (with or without IFD),
and thereby causing evaluation to fail if `system != builtins.currentSystem` on Hydra
so the jobs are not delegated to eligible build machines that support that system.
This is done in a way that avoids rebuilds.
I currently do not have much time to work on nixpkgs. Remove
myself as a maintainer from a bunch of packages to avoid that
people are waiting on me for a review.
near the end of 2019, the default Cargo.lock format was changed to
[[package]]
checksum = ...
This is what importCargoLock assumes. If the crate had not been `cargo
update`'d with a more recent toolchain than the one with the new
format as default, importCargoLock would fail when trying to access
pkg.checksum.
I ran into such a case (shamefully, in my own crate) and it took me a
while to figure out what was going on, so here is an assert with a
more user friendly message and a hint.
According to rustc implementation[1], `-C incremental=no` enables
incremental builds with directory name `no`. This patch removes the
`-C incremental` argument to disable incremental builds.
[1]: ee86f96ba1/compiler/rustc_session/src/options.rs (L918-L919)
This change introduces the cargoLock argument to buildRustPackage,
which can be used in place of cargo{Sha256,Hash} or cargoVendorDir. It
uses the importCargoLock function to build the vendor
directory. Differences compared to cargo{Sha256,Hash}:
- Requires a Cargo.lock file.
- Does not require a Cargo hash.
- Retrieves all dependencies as fixed-output derivations.
This makes buildRustPackage much easier to use as part of a Rust
project, since it does not require updating cargo{Sha256,Hash} for
every change to the lock file.
This function can be used to create an output path that is a cargo
vendor directory. In contrast to e.g. fetchCargoTarball all the
dependent crates are fetched using fixed-output derivations. The
hashes for the fixed-output derivations are gathered from the
Cargo.lock file.
Usage is very simple, e.g.:
importCargoLock {
lockFile = ./Cargo.lock;
}
would use the lockfile from the current directory.
The implementation of this function is based on Eelco Dolstra's
import-cargo:
https://github.com/edolstra/import-cargo/blob/master/flake.nix
Compared to upstream:
- We use fetchgit in place of builtins.fetchGit.
- Sync to current cargo vendoring.
Also begin to start work on cross compilation, though that will have to
be finished later.
The patches are based on the first version of
https://reviews.llvm.org/D99484. It's very annoying to do the
back-porting but the review has uncovered nothing super major so I'm
fine sticking with what I've got.
Beyond making the outputs work, I also strove to re-sync the packages,
as they have been drifting pointlessly apart for some time.
----
Other misc notes, highly incomplete
- lvm-config-native and llvm-config are put in `dev` because they are
tools just for build time.
- Clang no longer has an lld dep. That was introduced in
db29857eb3, but if clang needs help
finding lld when it is used we should just pass it flags / put in the
resource dir. Providing it at build time increases critical path
length for no good reason.
----
A note on `nativeCC`:
`stdenv` takes tools from the previous stage, so:
1. `pkgsBuildBuild`: `(?1, x, x)`
2. `pkgsBuildBuild.stdenv.cc`: `(?0, ?1, x)`
while:
1. `pkgsBuildBuild`: `(?1, x, x)`
2. `pkgsBuildBuild.targetPackages`: `(x, x, ?2)`
3. `pkgsBuildBuild.targetPackages.stdenv.cc`: `(?1, x, x)`
The `checkType` argument of buildRustPackage was not used anymore
since the refactoring of `buildRustPackage` into hooks. This was
an oversight that is fixed by this change.
The check type can also be passed directly to cargoCheckHook using the
`cargoCheckType` environment variable.
API change:
`cargoParallelTestThreads` suggests that this attribute sets the
number of threads used during tests, while it is actually a boolean
option (use 1 thread or NIX_BUILD_CORES threads). In the hook, this
is replaced by a more canonical name `dontUseCargoParallelTests`.
The directory in the tarball of vendored dependencies contains `name`,
which is by default set to `${pname}-${version}`. This adds an
additional attribute to permit setting the name to something of the
user's choosing.
Since `cargoSha256`/`cargoHash` depend on the name of the directory of
vendored dependencies, `cargoDepsName` can be used to e.g. make the
hash invariant to the package version by setting `cargoDepsName =
pname`.
- API change: remove the `target` argument of `buildRustPackage`, the
target should always be in sync with the C/C++ compiler that is used.
- Gathering of binaries has moved from `buildPhase` to `installPhase`,
this simplifies the hook and orders this functionality logically
with the installation logic.
`buildRustPackage` currently accepts `cargoSha256` as a hash for
vendored dependencies. This change adds `cargoHash` which accepts SRI
hashes, setting `outputHashAlgo` to `null`.
The hash mismatch message still uses `cargoSha256` as an example,
which it probably should until we completely switch to SRI hashes.
As @lopsided98 points out in #105305, since the hashes are now target
sensative, and until we find reason to actually care to test what they
are exactly, we are best just normalizing them away in the tests.
