this equates to -ftrivial-auto-var-init=pattern
clang has removed support for -ftrivial-auto-var-init=zero and
are unlikely to re-add it, so use -ftrivial-auto-var-init=pattern
on both compilers if only to make behaviour more consistent
between the two.
add to pkgsExtraHardening's defaultHardeningFlags.
This is a small simplification of the control flow surrounding these cases. It should make it more obvious when each case happens, and also explicitly defines the current behaviour of --replace.
stdenv.cc may throw, e.g. in the case of pkgsCross.ghcjs where we must
not force it for the purpose of attribute accessing (`or` doesn't
implicitly tryEval…).
Regression introduced in 1a5bd697ad.
Aka `checkMeta` goes brrr.
Using the module system type checking works OK & generates good error messages.
The performance of using it however is terrible because of the value merging it does being very allocation heavy.
By implementing a very minimal type checker we can drastically improve the performance when nixpkgs is evaluated with `checkMeta = true`.
this makes it a lot easier to create a modified stdenv with a
different set of defaultHardeningFlags and as a bonus allows us
to inject the correct defaultHardeningFlags into toolchain wrapper
scripts, reducing repetition.
while most hardening flags are arguably more of a compiler thing,
it works better to put them in bintools-wrapper because cc-wrapper
can easily refer to bintools but not vice-versa.
mkDerivation can still easily refer to either when it is constructed.
this also switches fortran-hook.sh to use the same defaults for
NIX_HARDENING_ENABLE as for C. previously NIX_HARDENING_ENABLE
defaults were apparently used to avoid passing problematic flags
to a fortran compiler, but this falls apart as soon as mkDerivation
sets its own NIX_HARDENING_ENABLE - cc.hardeningUnsupportedFlags
is a more appropriate mechanism for this as it actively filters
out flags from being used by the wrapper, so switch to using that
instead.
this is still an imperfect mechanism because it doesn't handle a
compiler which has both langFortran *and* langC very well - applying
the superset of the two's hardeningUnsupportedFlags to either
compiler's invocation. however this is nothing new - cc-wrapper
already poorly handles a langFortran+langC compiler, applying two
setup hooks that have contradictory options.
If a CMake target has a non-default LINKER_LANGUAGE set, CMake will
manually add the libraries it has detected that language's compiler as
linking implicitly. When it does this, it'll pass -Bstatic and
-Bdynamic options based on the vibes it gets from each such detected
library. This in itself isn't a problem, because the compiler
toolchain, or our wrapper, or something, seems to be smart enough to
ignore -Bdynamic for those libraries. But it does create a problem if
the compiler adds extra libraries to the linker command line after
that final -Bdynamic, because those will be linked dynamically. Since
our compiler is static by default, CMake should reset to -Bstatic
after it's done manually specifying libraries, but CMake didn't
actually know that our compiler is static by default. The fix for
that is to tell it, like so.
Until recently, this problem was difficult to notice, because it would
result binaries that worked, but that were dynamically linked. Since
e08ce498f0 ("cc-wrapper: Account for NIX_LDFLAGS and NIX_CFLAGS_LINK
in linkType"), though, -Wl,-dynamic-linker is no longer mistakenly
passed for executables that are supposed to be static, so they end up
created with a /lib interpreter path, and so don't run at all on
NixOS.
This fixes pkgsStatic.graphite2.
Provide a `runPhase` function which wraps the phase running action of
genericBuild. The new function can be used as an interface by `nix
develop`, i.e. `nix develop some#flake --build` may just call `runPhase
build`, which makes its behavior more consistent with `nix build`.
In preparation of fixing https://github.com/NixOS/nix/issues/6202
- These new-cli commands can be used with `-f`, in which case they're
evaluated with pure evaluation disabled.
- Nix 2.4+ is not part of the condition; "flakes" is fully descriptive
and more relatable.
- Don't suggest that it only enables this variable.
- Just don't say too much.
While there is no fetcher or builder (in nixpkgs) that takes an `md5` parameter,
for some inscrutable reason the nix interpreter accepts the following:
```nix
fetchurl {
url = "https://www.perdu.com";
hash = "md5-rrdBU2a35b2PM2ZO+n/zGw==";
}
```
Note that neither MD5 nor SHA1 are allowed by the syntax of SRI hashes.
When specifying the `builder` attribute in `stdenv.mkDerivation`, this
will be effectively transformed into
builtins.derivation {
builder = stdenv.shell;
args = [ "-e" builder ];
}
This also means that `default-builder.sh` is never sourced and as a
result it's not guaranteed that `$NIX_ATTRS_SH_FILE` is set to a correct
location[1].
Also, we need to source `.attrs.sh` to source `$stdenv`. So, the
following is done now:
* If `$NIX_ATTRS_SH_FILE` points to a correct location, then use it.
Directly using `.attrs.sh` is problematic for `nix-shell(1)` usage
(see previous commit for more context), so prefer the environment
variable if possible.
* Otherwise, if `.attrs.sh` exists, then use it. See [1] for when this
can happen.
* If neither applies, it can be assumed that `__structuredAttrs` is
turned off and thus nothing needs to be done.
[1] It's possible that it doesn't exist at all - in case of Nix 2.3 or
it can point to a wrong location on older Nix versions with a bug in
`__structuredAttrs`.
Relying on `.attrs.sh` to exist in `$NIX_BUILD_TOP` is problematic
because that's not compatible with how `nix-shell(1)` behaves. It places
`.attrs.{json,sh}` into a temporary directory and makes them accessible via
`$NIX_ATTRS_{SH,JSON}_FILE` in the environment[1]. The sole reason that
`nix-shell(1)` still works with structured-attrs enabled derivations
is that the contents of `.attrs.sh` are sourced into the
shell before sourcing `$stdenv/setup` (if `$stdenv` exists) by `nix-shell`.
However, the assumption that two files called `.attrs.sh` and
`.attrs.json` exist in `$NIX_BUILD_TOP` is wrong in an interactive shell
session and thus an inconsistency between shell debug session and actual
builds which can lead to unexpected problems.
To be precise, we currently have the following problem: an expression
like
with import ./. {};
runCommand "foo" { __structuredAttrs = true; foo.bar = [ 1 2 3 ]; }
''
echo "''${__structuredAttrs@Q}"
touch $out
''
prints `1` in its build-log. However when building interactively in a
`nix-shell`, it doesn't.
Because of that, I'm considering to propose a full deprecation of
`$NIX_BUILD_TOP/.attrs.{json,sh}`. A first step is to only mention the
environment variables, but not the actual paths anymore in Nix's
manual[2]. The second step - this patch - is to fix nixpkgs' stdenv
accordingly.
Please note that we cannot check for `-e "$NIX_ATTRS_JSON_FILE"` because
certain outdated Nix minors (that are still in the range of supported
Nix versions in `nixpkgs`) have a bug where `NIX_ATTRS_JSON_FILE` points
to the wrong file while building[3].
Also, for compatibility with Nix 2.3 which doesn't provide these
environment variables at all we still need to check for the existence of
.attrs.json/.attrs.sh here. As soon as we bump nixpkgs' minver to 2.4,
this can be dropped.
Finally, dropped the check for ATTRS_SH_FILE because that was never
relevant. In nix#4770 the ATTRS_SH_FILE variable was introduced[4] and
in a review iteration prefixed with NIX_[5]. In other words, these
variables were never part of a release and you'd only have this problem
if you'd use a Nix from a git revision of my branch from back then. In
other words, that's dead code.
[1] https://github.com/nixos/nix/pull/4770#issuecomment-834718851
[2] https://github.com/NixOS/nix/pull/9032
[3] https://github.com/NixOS/nix/issues/6736
[4] 3944a120ec
[5] 27ce722638
Fixes `pkgsCross.musl64.llvmPackages_16.clang.cc` on `x86_64-linux`,
which used to fail with `/bin/sh: clang-tblgen: not found`.
Same hack is used in other projects:
https://github.com/search?q=%2FCMAKE_CROSSCOMPILING_EMULATOR.%2B%5C%2Fusr%5C%2Fbin%5C%2Fenv%2F+NOT+is%3Afork&type=code
Comment from 30435a9d0f/build/cmake/HostLinuxToolchain.cmake (L64)
> Required to run host Linux executables during the build itself.
> An example would be https://gitub.com/KhronosGroup/Vulkan-Loader and
> its "asm_offset" program.
>
> NOTE: Alternatives have been tried unsuccessfully, i.e.:
>
> With $(set CMAKE_CROSSCOMPILING_EMULATOR), the build fails because
> the CMake ninja/Make script tries to find the executable in the current
> path, as in:
>
> [3/16] Generating gen_defines.asm
> FAILED: loader/gen_defines.asm
> cd /tmp/cc/build-Vulkan-Loader/loader && asm_offset GAS
> /bin/sh: asm_offset: command not found
> ninja: build stopped: subcommand failed.
>
> With $(set CMAKE_CROSSCOMPILING_EMULATOR ""), the build fails because
> the shell cannot find the "" program as in:
>
> [3/16] Generating gen_defines.asm
> FAILED: loader/gen_defines.asm
> cd /tmp/cc/build-Vulkan-Loader/loader && "" /tmp/cc/build-Vulkan-Loader/loader/asm_offset GAS
> /bin/sh: : command not found
> ninja: build stopped: subcommand failed.
>
> It seems that the root of the problem comes from how the CMake function
> cmCustomCommandGenerator::GetArgc0Location() computes the target
> executable's location. At this point it's unclear whether this is a CMake
> bug or a feature.
Risicle discovered this hack.
Co-authored-by: Robert Scott <code@humanleg.org.uk>
In the default `fixupPhase` the output of `substituteAllStream` is
streamed to setup-hook.
`stdenv.cc.bintools.overrideAttrs { NIX_DEBUG = 6; }`
With `NIX_DEBUG` contains:
```
@expandResponseParams@ -> /nix/store/yl01rd58vp4m8bbhkihpk132cprfmx6f-expand-response-params/bin/expand-response-params
...
```
```
nix-repl> (pkgs.htop.overrideAttrs { pname = "hello-overriden"; }).pname
error:
… while evaluating a branch condition
at /nix/store/phn5cahwacv9wjgalygw62x8l4xbl6x3-source/lib/customisation.nix:86:7:
85| in
86| if builtins.isAttrs result then
| ^
87| result // {
… while calling the 'isAttrs' builtin
at /nix/store/phn5cahwacv9wjgalygw62x8l4xbl6x3-source/lib/customisation.nix:86:10:
85| in
86| if builtins.isAttrs result then
| ^
87| result // {
(stack trace truncated; use '--show-trace' to show the full trace)
error: attempt to call something which is not a function but a set
at /nix/store/phn5cahwacv9wjgalygw62x8l4xbl6x3-source/pkgs/stdenv/generic/make-derivation.nix:58:21:
57| f = self: super:
58| let x = f0 super;
| ^
59| in
```
Makes overrideAttrs usable in the same way that `override` can be used.
It allows the first argument of `overrideAttrs` to be either a function
or an attrset, instead of only a function:
hello.overrideAttrs (old: { postBuild = "echo hello"; })
hello.overrideAttrs { postBuild = "echo hello"; }
Previously only the first example was possible.
Co-authored-by: adisbladis <adisbladis@gmail.com>
Co-authored-by: matthewcroughan <matt@croughan.sh>
passAsFile passes the values of Nix bindings to the builder as
files, so if those values contained references, they wouldn't end up
in the inputDerivation output. To fix that, append the contents of
every such passed file to the output.
We only have shell builtins in this derivation, so we can't use cat.
The only way I know of appending the contents of one file to another
using only shell builtins is as I've done here, but it requires
putting the contents of the file on echo's argv. This might end up
causing problems with large files. Regardless, I think we should try
this, as a failure is better than silently producing an incorrect
result like the previous behavior.
`nix-2.4+` automatically filters `__contentAddressed` out of the
environment. But not `nix-2.3`. This make `.drv` to differ between
unset and `__contentAddressed = false` derivations.
This change makes them equal by filtering out `__contentAddressed`
unless it's set to `true`.
The NIX_LIB64|32_IN_SELF_RPATH environment variables control whether
to add lib64 and lib32 to rpaths. However, they're set depending
on the build paltform, not the target platform and thus their values
are incorrect for for cross-builds.
On the other hand, setting them according to the build platform introduce
pointless differences in build outputs; see #221350 for details.
This change fixes the issues by boldly removes the NIX_LIB*_IN_SELF_RPATH
facility altogether, in the hope that it is no longer necessary. They
were introduced in 2009, long before nixpkgs had good support for
cross-builds.
Fixes#221350
See https://github.com/NixOS/nixpkgs/pull/222792#pullrequestreview-1356114111
You can't just `lib.filter _ lib.systems.all` -- that throws away
important information, leading to nixpkgs disagreeing with itself
like this:
```
$ NIXPKGS_ALLOW_BROKEN=1 nix-instantiate . -A pkgsStatic.systemd
error: Package ‘systemd-252.5’ in ... is only supported on ... x86_64-linux but not on requested x86_64-linux, refusing to evaluate.
```
After:
```
$ NIXPKGS_ALLOW_BROKEN=1 nix-instantiate . -A pkgsStatic.systemd
error: Package ‘systemd-252.5’ in ... is not available on the requested hostPlatform:
hostPlatform.config = "x86_64-unknown-linux-musl"
package.meta.platforms = [
"aarch64-linux"
"armv5tel-linux"
"armv6l-linux"
"armv7a-linux"
"armv7l-linux"
"i686-linux"
"m68k-linux"
"microblaze-linux"
"microblazeel-linux"
"mipsel-linux"
"mips64el-linux"
"powerpc64-linux"
"powerpc64le-linux"
"riscv32-linux"
"riscv64-linux"
"s390-linux"
"s390x-linux"
"x86_64-linux"
]
package.meta.badPlatforms = [
{
isStatic = true;
parsed = { };
}
]
, refusing to evaluate.
```
The primary motivating example is openssl:
Before the change full package build took 1m54s minutes.
After the change full package build takes 59s.
About a 2x speedup.
The difference is visible because openssl builds hundreds of manpages
spawning a perl process per manual in `install` phase. Such a workload
is very easy to parallelize.
Another example would be `autotools`+`libtool` based build system where
install step requires relinking. The more binaries there are to relink
the more gain it will be to do it in parallel.
The change enables parallel installs by default only for buiilds that
already have parallel builds enabled. There is a high chance those build
systems already handle parallelism well but some packages will fail.
Consistently propagated the enableParallelBuilding to:
- cmake (enabled by default, similar to builds)
- ninja (set parallelism explicitly, don't rely on default)
- bmake (enable when requested)
- scons (enable when requested)
- meson (set parallelism explicitly, don't rely on default)
- waf (set parallelism explicitly, don't rely on default)
- qmake-4/5/6 (enable by default, similar to builds)
- xorg (always enable, similar to builds)
This was disabled here: b86e62d30d (diff-282a02cc3871874f16401347d8fadc90d59d7ab11f6a99eaa5173c3867e1a160)
h/t to @teh: b86e62d30d (commitcomment-77916294)
for pointing out that the failure that @matthewbauer was
seeing was caused by the `separate-debug-info.sh` `build-id` length
requirement that #146275 will relax
`lld` has had `--build-id` support dating back to LLVM4: https://reviews.llvm.org/D18091
This predates every `llvmPackages_` version currently in nixpkgs (and
certainly every version actually still used in `useLLVM` stdenvs) so
with the previous commit (asking `ld` for sufficiently long SHA1 hashes)
I think we can safely enable `separateDebugInfo` when using LLVM
bintools.
See docs.
Follow-up work:
- Existing packages should be converted
- `defaultPkgConfigPackages` should assert on `meta.pkgConfigModules`
and let `tests.pkg-config` alone test the build results.
CC @sternenseemann
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
platform.uname.processor seems to be what we want in many more cases
than what we were using before — it does the right thing for aarch64,
x86_64, riscv32, riscv64, mips, mips64, powerpc, and powerpc64 (the
latter three of which were broken before).
This fixes cross-compilation of systemd for PowerPC/POWER platforms.
Derivations listed as disallowedReferences or disallowedRequisites,
currently end up as build-time dependencies.
This is problematic since the disallowed derivations will be built by nix as
build-time dependencies, while those derivations might take a very long time
to build, or might not even build successfully on the platform used.
However, in order to scan for disallowed references in the final output,
knowing the out path is sufficient, and the out path can be calculated from
the derivation without needing to build it, saving time and resources.
While the problem is less severe for allowedReferences and allowedRequisites,
since we want the derivation to be built eventually, we would still like to
get the error early and without having to wait while nix builds a derivation
that might not be used (e.g. if we listed the wrong one).
Some other packages, for example ruby gems via buildRubyGem, use a
variable called "type" internally, which is overwritten here and
causes failures like:
failure: $gempkg path unspecified
Fix for changes in 11c3127e38.
this is intentional to support both structuredAttrs and non
In pkgs/stdenv/generic/setup.sh line 614:
for pkg in ${depsBuildBuild[@]} ${depsBuildBuildPropagated[@]}; do
^------------------^ SC2068 (error): Double quote array expansions to avoid re-splitting elements.
In pkgs/stdenv/generic/setup.sh line 521:
local varRef="$varVar[$((targetOffset - hostOffset))]"
^-- SC1087 (error): Use braces when expanding arrays, e.g. ${array[idx]} (or ${var}[.. to quiet).
exit -1 == exit 255 but we don't have a reason to use 255
In pkgs/stdenv/generic/setup.sh line 518:
(( hostOffset <= targetOffset )) || exit -1
^-- SC2242 (error): Can only exit with status 0-255. Other data should be wri
tten to stdout/stderr.
we use [*] to support structuredAttrs and non
In pkgs/stdenv/generic/setup.sh line 1542:
for curPhase in ${phases[*]}; do
^----------^ SC2048 (warning): Use "${array[@]}" (with quotes) to prevent whitespace problems.
In pkgs/stdenv/generic/setup.sh line 101:
source "$hookName"
^---------^ SC1090 (warning): ShellCheck can't follow non-constant source. Use a directive to specify location.
In pkgs/stdenv/generic/setup.sh line 166:
mkdir -p "$out/nix-support"
^--^ SC2154 (warning): out is referenced but not assigned.
In pkgs/stdenv/generic/setup.sh line 407:
PATH=
^--^ SC2123 (warning): PATH is the shell search path. Use another name.
In pkgs/stdenv/generic/setup.sh line 452:
declare -a pkgBuildAccumVars=(pkgsBuildBuild pkgsBuildHost pkgsBuildTarget)
^---------------^ SC2034 (warning): pkgBuildAccumVars appears unused. Verify use (or export if used e
xternally).
because pkgBuildAccumVars is used
In pkgs/stdenv/generic/setup.sh line 235:
nameref="$* ${nameref-}"
^-----^ SC2178 (warning): Variable was used as an array but is now assigned a string.
because we theres a useArray conditional
In pkgs/stdenv/generic/setup.sh line 36:
: ${outputs:=out}
^-------------^ SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
`hasUnsupportedPlatform` was not updated with #37395, so it does not
understand attrsets in `meta.[bad]platforms`. In particular,
attrsets in `meta.badPlatforms` will "fail open" and be ignored.
Let's use `lib.meta.availableOn` instead of duplicating its logic.
Thanks to @alyssais for [noticing][1].
[1][https://github.com/NixOS/nixpkgs/pull/194148#discussion_r990817610]
Co-authored-by: sternenseemann <sternenseemann@systemli.org>
otherwise the build just fails with 'make: *** No rule to make target 'install'. Stop.'
and update buildPhase message
i don't know if the 'makefile may have been created in buildPhase' is
true but i guess it might be possible
The documentation for `meta.sourceProvenance` in
`doc/stdenv/meta.chapter.md` says: "the `meta.sourceProvenance`
attribute should be a list containing one or more value..."
Let's update check-meta.nix to require that `meta.sourceProvenance` is
a list, as the documentation says, rather than a single element.
Adding two extra keystrokes `[` and `]` when filling out this field is
an insignificant burden for package authors, and being able to assume
that the `meta.sourceProvenance` field is always a list greatly
simplifies any code that acts on the value of this field.
Since `meta.sourceProvenance` was just merged a few hours ago now is
the easiest time to fix this: nobody is using the feature yet.
Without the change we don't propagate `enableParallelBuilding = true`
and leave most builds sequential.
Noticed on `mythtv` package which did not specify parallelism and
`config.enableParallelBuildingByDefault = true` had no effect.
Derivations not using `__structuredAttrs` should not attempt to set
environment variables from `env`.
Derivations using `__structuredAttrs` should fail if `env` is not
exportable.
