Closes#302291Closes#301536
The following things have changed:
* For 5.7+: ZSWAP compressor uses zstd now.
* For 5.11+: ZRAM compressor uses zstd now.
* For 5.13+: kernel modules are compressed with zstd instead of xz.
* For 5.19+: support zstd-compressed firmware.
The modules-closure functionality needed explicit support for copying
over `.zst` files. Also, the VM image builder used busybox's `insmod`
before which doesn't support zstd. Switched to `kmod` and added xz/zstd
as dependencies for it, similar to how it's done for the actual stage1
in d33e52b253. The use of `kmod` here
doesn't seem to be such a big deal since it's only a build-time
dependency.
Since the tool is exposed more prominently now, we should clear up what
it is and note that it is to be considered unstable, i.e. we may change
it if the necessity arises. (In practice it is probably going to be
fairly stable though, as compiler interfaces tend to be quite stable.)
Should we add a version?
The cc and bintools wrapper contained ad hoc bootstrapping logic for
expand-response-params (which was callPackage-ed in a let binding). This
lead to the strange situation that the bootstrapping logic related to
expand-response-params is split between the wrapper derivations (where
it is duplicated) and the actual stdenv bootstrapping.
To clean this up, the wrappers simply should take expand-response-params
as an ordinary input: They need an adjacent expand-response-params (i.e.
one that runs on their host platform), but don't care about the how.
Providing this is only problematic during stdenv bootstrapping where we
have to pull it from the previous stage at times.
We don't need to artificially make sure that we can execute the wrapper
scripts on the build platform by using stdenv's shell (which comes from
buildPackages) since our cross infrastructure will get us the wrapper
from buildPackages. The upside of this change is that cross-compiled
wrappers (e.g. pkgsCross.aarch64-multiplatform.gcc) will actually work
when executed!
For bootstrapping this is also not a problem, since we have a long
build->build platform chain so runtimeShell is just as good as
stdenvNoCC.shell. We do fall back to old ways, though, by explicitly
using the bootstrap-tools shell in stage2, so the adjacent bash is only
used from stage4 onwards. This is unnecessary in principle (I'll try
removing this hack in the future), but ensures this change causes zero
rebuilds.
To reduce dependencies (mainly nix-prefetch-git and through that git,
git-lfs) when we just need to fixup a lock file, eg when building electron.
This also tries to avoid needless rebuilds when eg. golang is updated.
Also this cleans up and combined the build/installPhase of both tools to
be a lot simpler.
In delicate code like this, it seems unwise to pass something of as
something it isn't for convenience's (?) sake. It causes a slight
possibility for confusion with `buildPackages.stdenv`. However, it
should be possible to eliminate the need for this in a separate change.
`stdenv.is*` defaults to `hostPlatform` which is technically incorrect
here, since what we are gating concerns the target platform only.
This would be a problem in practice if cross compiling from
x86_64-darwin to aarch64-darwin whilst using gcc which is probably
rare enough.
Replace `src = ./.` instances with more explicit source listings.
Otherwise, derivations will be rebuilt on any change to the files
defining them (e.g. formatting via nixfmt-rfc-style).
After enabling a separate binary output for the `Agda` Haskell package,
the new `bin` output measures ~100MiB, compared to the ~4.5GiB before.
Using it in `agdaWithPackages` reduces the closure size of an Agda
installation from ~5GiB to ~3GiB. The remaining space is taken up
mostly by the GHC backend.
With this change, derivations depending on `haskellPackages.Agda`
directly need to pick the right (binary) output. This concerns in
particular `emacsPackages.agda2-mode`.
`lib.getExe pkgsCross.riscv64.buildPackages.pkg-config` should return
`<prefix>/bin/riscv64-unknown-linux-gnu-pkg-config` not `<prefix>/bin/pkg-config`
This was added in d83e9c9573,
but included substitution variables `@blas@` and `@lapack@` that were
never substituted. It's not used anywhere in `nixpkgs`.
Ignore vendorSha256 when vendorHash is specified.
Throw when vendorHash isn't specified:
- "buildGoModule: Expect vendorHash instead of vendorSha256" when
vendorSha256 is specified.
- "buildGoModule: vendorHash is missing" otherwise.
`goModules.outputHashAlgo` is specified as null when vendorHash is not
empty, "sha256" otherwise.
Co-authored-by: zowoq <59103226+zowoq@users.noreply.github.com>
This is convenient for debugging the underlying streamed image used by
`dockerTools.buildLayeredImage`.
Here's an example of how you might use this:
```console
$ nix repl ./.
nix-repl> dockerTools.examples.nginx.passthru.stream
«derivation /nix/store/9zczmlp2kraszx4ssmh6fawnlnsa5a4n-stream-nginx-container.drv»
```
The way we build python environments is subtly broken. A python
environment should be semantically identical to a vanilla Python
installation in, say, /usr/local. The current implementation, however,
differs in two important ways. The first is that it's impossible to use
python packages from the environment in python virtual environments. The
second is that the nix-generated environment appears to be a venv, but
it's not.
This commit changes the way python environments are built:
* When generating wrappers for python executables, we inherit argv[0]
from the wrapper. This causes python to initialize its configuration
in the environment with all the correct paths.
* We remove the sitecustomize.py file from the base python package.
This file was used tweak the python configuration after it was
incorrectly initialized. That's no longer necessary.
The end result is that python environments no longer appear to be venvs,
and behave more like a vanilla python installation. In addition it's
possible to create a venv using an environment and use packages from
both the environment and the venv.
We also renamed `filter` (as a name of a parameter) to `predicate` following the naming suggestion in code review. It's better!
Since it's not part of an attrset, the name can change with no impact to semantics, since it can't be observed with `builtins.functionArgs`.
```
$ nix-repl
Nix 2.21.0
Type :? for help.
nix-repl> f = x: y: z: (x + y + z)
nix-repl> builtins.functionArgs f
{ }
nix-repl> :doc builtins.functionArgs
Synopsis: builtins.functionArgs f
Return a set containing the names of the formal arguments expected by the function f. The value of each attribute is a Boolean denoting whether the corresponding argument has a default value. For instance, functionArgs ({ x, y ?
123}: ...) = { x = false; y = true; }.
"Formal argument" here refers to the attributes pattern-matched by the function. Plain lambdas are not included, e.g. functionArgs (x: ...) = { }.
```
The nixpkgs-unstable channel's programs.sqlite was used to identify
packages producing exactly one binary, and these automatically added
to their package definitions wherever possible.
The `-path` test of `find` does string comparison, not path comparison.
Hence, the format of `everythingFile` needed to be very specific. Now,
it can be denormalized (e.g. it can contain `/./`) and an error is
emitted if the everything file or its interface file can't be removed.
Upstream now provides a library file for the builtin library and ensured
that the existing interface files will be used regardless of whether
--local-interfaces is in effect. Hence, Agda will not try to write to
the Nix store anymore except if the build flags are changed.
Replace writeReferencesToFile with writeClosure.
Make writeClosure accept a list of paths instead of a path.
Re-implement with JSON-based exportReferencesGraph interface provided by
__structuredAttrs = true.
Reword the documentation.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Someone Serge <sergei.kozlukov@aalto.fi>
In NixOS/nixpkgs#290081 it came to attention that autoPatchelfHook is
one of if not the only hook in Nixpkgs that is a multiline string
expression. Almost all hooks are functions, which guard with something
like `if [ -z "${dontDoTheThing-}" ]; then ...` in the function, or
single-line strings which include that guard inline and then call the
real function, e.g. `if [ -z "${dontDoTheThing-} ]; then doTheThing; fi`.
This commit moves autoPatchelfHook to the former, which seems to be the
most common style now.
- Use testers.runNixOSTest instead of testers.nixosTest as nixosTest
has become obsolete.
- Prepare for cross-platform testing.
- Use the testScriptBin passthru'd by the references test package
inside the guest pkgs.
swiftc uses cc-wrapper which sets the -march flag on some systems which
breaks the build. This change adds a flag, disableMarch, to cc-wrapper
which disables using the -march flag.
https://github.com/NixOS/nixpkgs/issues/295322
This assumes that downstream users of `buildNpmPackage` would rather our
own built `node_modules` be copied to the output rather than only the
`bundleDependencies` specified in the `package.json` file.
Having the latter behavior seems unexpected and unintuitive, and would
not work as installing from an `npm pack` is intended to (since doing
that would not do a `rebuild` step on those dependencies and it would
skip reifying a full dependency tree).
- merge libcxxabi into libcxx for LLVM 12, 13, 14, 15, 16, 17, and git.
- remove the link time workaround `-lc++ -lc++abi` from 58 packages as it is no longer required.
- fixes https://github.com/NixOS/nixpkgs/issues/166205
- provides alternative fixes for. https://github.com/NixOS/nixpkgs/issues/269548https://github.com/NixOS/nix/issues/9640
- pkgsCross.x86_64-freebsd builds work again
This change can be represented in 3 stages
1. merge libcxxabi into libcxx -- files: pkgs/development/compilers/llvm/[12, git]/{libcxx, libcxxabi}
2. update stdenv to account for merge -- files: stdenv.{adapters, cc.wrapper, darwin}
3. remove all references to libcxxabi outside of llvm (about 58 packages modified)
### merging libcxxabi into libcxx
- take the union of the libcxxabi and libcxx cmake flags
- eliminate the libcxx-headers-only package - it was only needed to break libcxx <-> libcxxabi circular dependency
- libcxx.cxxabi is removed. external cxxabi (freebsd) will symlink headers / libs into libcxx.
- darwin will re-export the libcxxabi symbols into libcxx so linking `-lc++` is sufficient.
- linux/freebsd `libc++.so` is a linker script `LINK(libc++.so.1, -lc++abi)` making `-lc++` sufficient.
- libcxx/default.nix [12, 17] are identical except for patches and `LIBCXX_ADDITIONAL_LIBRARIES` (only used in 16+)
- git/libcxx/defaul.nix does not link with -nostdlib when useLLVM is true so flag is removed. this is not much different than before as libcxxabi used -nostdlib where libcxx did not, so libc was linked in anyway.
### stdenv changes
- darwin bootstrap, remove references to libcxxabi and cxxabi
- cc-wrapper: remove c++ link workaround when libcxx.cxxabi doesn't exist (still exists for LLVM pre 12)
- adapter: update overrideLibcxx to account for a pkgs.stdenv that only has libcxx
### 58 package updates
- remove `NIX_LDFLAGS = "-l${stdenv.cc.libcxx.cxxabi.libName}` as no longer needed
- swift, nodejs_v8 remove libcxxabi references in the clang override
https://github.com/NixOS/nixpkgs/pull/292043
Add a makeWrapperArgs argument to all script writers under pkgs.writers.
This can be used to set, prefix, or suffix the PATH or other environment variables which improves the ability to generate scripts with reproducible behavior.
Some of the writers (writeBash, writeDash, writeFish, writeNu) previously did not support passing an argument set, for example
```
writeBash "example" "echo hello"
```
In order to add the new capability to these writers as well, their call signature is now overloaded in order to allow the following:
(The old call style from the example above remains intact)
```
writeBash "example"
{ makeWrapperArgs = [ "--prefix" "PATH" ":" "${pkgs.hello}/bin" ]; }
''
hello
''
```
Done as well:
- add tests
- add more docs
- fix some misleading docs
- extend existing docs with more examples
`snapTools.makeSnap` has produced broken snaps since at least Oct 2020,
as indicated by the following issue: https://github.com/NixOS/nixpkgs/issues/100618
No person has shown interest in maintaining it, and given that there is
no fix available, it's assumed that all attempts made to fix that
function have not succeeded.
Given that `snapTools` only contained `makeSnap`, it was removed
completely.
This is an alternative to `fetchNpmDeps` that is notably different in that it uses metadata from `package.json` & `package-lock.json` instead of specifying a fixed-output hash.
Notable features:
- IFD free.
- Only fetches a node dependency once. No massive FODs.
- Support for URL, Git and path dependencies.
- Uses most of the existing `npmHooks`
`importNpmLock` can be used _only_ in the cases where we need to check in a `package-lock.json` in the tree.
Currently this means that we have 13 packages that would be candidates to use this function, though I expect most usage to be in private repositories.
This is upstreaming the builder portion of https://github.com/adisbladis/buildNodeModules into nixpkgs (different naming but the code is the same).
I will archive this repository and consider nixpkgs the new upstream once it's been merged.
For more explanations and rationale see https://discourse.nixos.org/t/buildnodemodules-the-dumbest-node-to-nix-packaging-tool-yet/35733
Example usage:
``` nix
stdenv.mkDerivation {
pname = "my-nodejs-app";
version = "0.1.0";
src = ./.;
nativeBuildInputs = [
importNpmLock.hooks.npmConfigHook
nodejs
nodejs.passthru.python # for node-gyp
npmHooks.npmBuildHook
npmHooks.npmInstallHook
];
npmDeps = buildNodeModules.fetchNodeModules {
npmRoot = ./.;
};
}
```
We need to set -crt-static on musl for rustdoc as well, so let's unify
the wrappers. Ideally, rather than wrapping rustdoc, we'd have
rustdoc use the wrapped rustc, but that's currently only possible with
an unstable option (--test-builder).
The options set by the wrapper, -C target-feature and --sysroot, are
supported by both rustdoc and rustc, but other flags maybe not be
supported by both, so I've introduced different environment
variables (the existing NIX_RUSTFLAGS and a new NIX_RUSTDOCFLAGS) to
allow those to be set independently.
This fixes cargo-auditable in pkgsMusl., which broke because its
doctests stopped working when -crt-static was moved to the wrapper.
Fixes: 79156bf13a ("rustc: move crt-static default override to wrapper (#291829)")
Closes#193336Closes#261694
Related to #108984
The goal here was to get the following flake to build and run on
`aarch64-darwin`:
```nix
{ inputs.nixpkgs.url = <this branch>;
outputs = { nixpkgs, ... }: {
checks.aarch64-darwin.default =
nixpkgs.legacyPackages.aarch64-darwin.nixosTest {
name = "test";
nodes.machine = { };
testScript = "";
};
};
}
```
… and after this change it does. There's no longer a need for the
user to set `nodes.*.nixpkgs.pkgs` or
`nodes.*.virtualisation.host.pkgs` as the correct values are inferred
from the host system.
If -target is explicitly passed to clang, we shouldn't pass our -march
value for the default target, because it probably won't exist for the
target being used. Up until now, clang has been lenient with this,
but it's a hard error with clang 17, so since gcc.arch is always set
on aarch64, fixing this is a hard requirement for upgrading our
default clang to 17.
Before (with clang 17 on aarch64-linux):
$ clang -target bpf -c -o /dev/null test.bpf.c
clang: warning: ignoring '-fstack-protector-strong' option as it is not currently supported for target 'bpf' [-Woption-ignored]
clang: error: unsupported option '-march=' for target 'bpf'
clang: warning: argument unused during compilation: '--gcc-toolchain=/nix/store/cngak08nb1yk44gnjipv5rg1ahh1xkax-gcc-13.2.0' [-Wunused-command-line-argument]
After:
$ clang -target bpf -c -o /dev/null test.bpf.c
clang: warning: ignoring '-fstack-protector-strong' option as it is not currently supported for target 'bpf' [-Woption-ignored]
clang: warning: argument unused during compilation: '--gcc-toolchain=/nix/store/cngak08nb1yk44gnjipv5rg1ahh1xkax-gcc-13.2.0' [-Wunused-command-line-argument]
Add a makeWrapperArgs argument to all script writers under pkgs.writers.
This can be used to set, prefix, or suffix the PATH or other environment variables which improves the ability to generate scripts with reproducible behavior.
Some of the writers (writeBash, writeDash, writeFish, writeNu) previously did not support passing an argument set, for example
```
writeBash "example" "echo hello"
```
In order to add the new capability to these writers as well, their call signature is now overloaded in order to allow the following:
(The old call style from the example above remains intact)
```
writeBash "example"
{ makeWrapperArgs = [ "--prefix" "PATH" ":" "${pkgs.hello}/bin" ]; }
''
hello
''
```
Done as well:
- add tests
- add more docs
- fix some misleading docs
- extend existing docs with more examples
Previously, when cross compiling from non-musl to musl, the crt-static
default override wouldn't be applied, because the compiler wouldn't
have been built with it due to fastCross. Moving it to the wrapper
fixes this without having to introduce extra compiler rebuilds. And
because the wrapper is applied even to the bootstrap rustc, we no
longer need special handling of crt-static in the Cargo expression.
Unlike --sysroot, rustc allows -C target-feature= to be passed
multiple times, with later instances taking precedence over earlier
ones. This means that it's very easy to set the default in the
wrapper, just by our overridden default before any other arguments.
This fixes pkgsCross.aarch64-multiplatform-musl.mesa from x86_64-linux.
The change is insignificant when the owner is root. However, when it
is not root, this change is needed to allow using Nix (as an
unprivileged user) inside the container.
Ada depencencies musst be build with the same gnat version as the
project. Use a namespace as preperation to build with different gnat
versions.
gprbuild and gnatprove are still globaly visable.
this equates to -ftrivial-auto-var-init=pattern
clang has removed support for -ftrivial-auto-var-init=zero and
are unlikely to re-add it, so use -ftrivial-auto-var-init=pattern
on both compilers if only to make behaviour more consistent
between the two.
add to pkgsExtraHardening's defaultHardeningFlags.
