Add a new experimental `impure-env` setting that is a key-value list of
environment variables to inject into FOD derivations that specify the
corresponding `impureEnvVars`.
This allows clients to make use of this feature (without having to change the
environment of the daemon itself) and might eventually deprecate the current
behaviour (pick whatever is in the environment of the daemon) as it's more
principled and might prevent information leakage.
Additionally this skipping of the building is reimplemented to be a bit
more robust and use the same idioms as the functionality for skipping
the tests. In particular, it will now work even if the source files
exist, so we can do this during development too.
I think the our `flake.nix` is currently too large and too scary looking.
I think this matters --- if Nix cannot dog-food itself in a way that is
elegant, why should other people have confidence that their own code can
be elegant and easy to maintain?
We could do this at many points in time, but I think around now, when we
are thinking about stabilizing parts of Flakes, is an especially good
time.
This is a first step to make the `flake.nix` smaller, and make
individual components responsible for their own packaging. I hope we can
do this many more follow-ups like it, until the top-level `flake.nix` is
very small and just coordinates between other things.
I think it is bad for these reasons when `tests/` contains a mix of
functional and integration tests
- Concepts is harder to understand, the documentation makes a good
unit vs functional vs integration distinction, but when the
integration tests are just two subdirs within `tests/` this is not
clear.
- Source filtering in the `flake.nix` is more complex. We need to
filter out some of the dirs from `tests/`, rather than simply pick
the dirs we want and take all of them. This is a good sign the
structure of what we are trying to do is not matching the structure
of the files.
With this change we have a clean:
```shell-session
$ git show 'HEAD:tests'
tree HEAD:tests
functional/
installer/
nixos/
```
A couple of tests require building some libraries that depend on Nix,
and assume it to be built locally.
Don't run these if we only want to run the install tests.
This prevents the CI from rebuilding several times Nix (like in
https://github.com/NixOS/nix/actions/runs/6404422275/job/17384964033#step:6:6412), thus removing a fair amount of build time.
This reverts commit 5e3986f59c. This
un-implements RFC 92 but fixes the critical bug #9052 which many people
are hitting. This is a decent stop-gap until a minimal reproduction of
that bug is found and a proper fix can be made.
Mostly fixed#9052, but I would like to leave that issue open until we
have a regression test, so I can then properly fix the bug (unbreaking
RFC 92) later.
In #4770 I implemented proper `nix-shell(1)` support for derivations
using `__structuredAttrs = true;`. Back then we decided to introduce two
new environment variables, `NIX_ATTRS_SH_FILE` for `.attrs.sh` and
`NIX_ATTRS_JSON_FILE` for `.attrs.json`. This was to avoid having to
copy these files to `$NIX_BUILD_TOP` in a `nix-shell(1)` session which
effectively meant copying these files to the project dir without
cleaning up afterwords[1].
On last NixCon I resumed hacking on `__structuredAttrs = true;` by
default for `nixpkgs` with a few other folks and getting back to it,
I identified a few problems with the how it's used in `nixpkgs`:
* A lot of builders in `nixpkgs` don't care about the env vars and
assume that `.attrs.sh` and `.attrs.json` are in `$NIX_BUILD_TOP`.
The sole reason why this works is that `nix-shell(1)` sources
the contents of `.attrs.sh` and then sources `$stdenv/setup` if it
exists. This may not be pretty, but it mostly works. One notable
difference when using nixpkgs' stdenv as of now is however that
`$__structuredAttrs` is set to `1` on regular builds, but set to
an empty string in a shell session.
Also, `.attrs.json` cannot be used in shell sessions because
it can only be accessed by `$NIX_ATTRS_JSON_FILE` and not by
`$NIX_BUILD_TOP/.attrs.json`.
I considered changing Nix to be compatible with what nixpkgs
effectively does, but then we'd have to either move $NIX_BUILD_TOP for
shell sessions to a temporary location (and thus breaking a lot of
assumptions) or we'd reintroduce all the problems we solved back then
by using these two env vars.
This is partly because I didn't document these variables back
then (mea culpa), so I decided to drop all mentions of
`.attrs.{json,sh}` in the manual and only refer to `$NIX_ATTRS_SH_FILE`
and `$NIX_ATTRS_JSON_FILE`. The same applies to all our integration tests.
Theoretically we could deprecated using `"$NIX_BUILD_TOP"/.attrs.sh` in
the future now.
* `nix develop` and `nix print-dev-env` don't support this environment
variable at all even though they're supposed to be part of the replacement
for `nix-shell` - for the drv debugging part to be precise.
This isn't a big deal for the vast majority of derivations, i.e.
derivations relying on nixpkgs' `stdenv` wiring things together
properly. This is because `nix develop` effectively "clones" the
derivation and replaces the builder with a script that dumps all of
the environment, shell variables, functions etc, so the state of
structured attrs being "sourced" is transmitted into the dev shell and
most of the time you don't need to worry about `.attrs.sh` not
existing because the shell is correctly configured and the
if [ -e .attrs.sh ]; then source .attrs.sh; fi
is simply omitted.
However, this will break when having a derivation that reads e.g. from
`.attrs.json` like
with import <nixpkgs> {};
runCommand "foo" { __structuredAttrs = true; foo.bar = 23; } ''
cat $NIX_ATTRS_JSON_FILE # doesn't work because it points to /build/.attrs.json
''
To work around this I employed a similar approach as it exists for
`nix-shell`: the `NIX_ATTRS_{JSON,SH}_FILE` vars are replaced with
temporary locations.
The contents of `.attrs.sh` and `.attrs.json` are now written into the
JSON by `get-env.sh`, the builder that `nix develop` injects into the
derivation it's debugging. So finally the exact file contents are
present and exported by `nix develop`.
I also made `.attrs.json` a JSON string in the JSON printed by
`get-env.sh` on purpose because then it's not necessary to serialize
the object structure again. `nix develop` only needs the JSON
as string because it's only written into the temporary file.
I'm not entirely sure if it makes sense to also use a temporary
location for `nix print-dev-env` (rather than just skipping the
rewrite in there), but this would probably break certain cases where
it's relied upon `$NIX_ATTRS_SH_FILE` to exist (prime example are the
`nix print-dev-env` test-cases I wrote in this patch using
`tests/shell.nix`, these would fail because the env var exists, but it
cannot read from it).
[1] https://github.com/NixOS/nix/pull/4770#issuecomment-836799719
It was disabled in c6953d1ff6 because
a recent Nixpkgs bump brought in a new systemd which changed how
systemd-nspawn worked.
As far as I can tell, the issue was caused by this upstream systemd
commit:
b71a0192c0
Bind-mounting the host's `/sys` and `/proc` into the container's
`/run/host/{sys,proc}` fixes the issue and allows the test to succeed.
https://hydra.nixos.org/build/235888160
This is needed because Nixpkgs now contains dangling symlinks
(pkgs/test/nixpkgs-check-by-name/tests/symlink-invalid/pkgs/by-name/fo/foo/foo.nix).
This is broken because of a change in systemd in NixOS 23.05. It fails
with
Failed to mount proc (type proc) on /proc (MS_NOSUID|MS_NODEV|MS_NOEXEC ""): Operation not permitted
The Derivation parser and old ATerm unfortunately leaves few ways to get
nice errors when an old version of Nix encounters a new version of the
format. The most likely scenario for this to occur is with a new client
making a derivation that the old daemon it is communicating with cannot
understand.
The extensions we just created for dynamic derivation deps will add a
version field, solving the problem going forward, but there is still the
issue of what to do about old versions of Nix up to now.
The solution here is to carefully catch the bad error from the daemon
that is likely to indicate this problem, and add some extra context to
it.
There is another "Ugly backwards compatibility hack" in
`remote-store.cc` that also works by transforming an error.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
We use the same nested map representation we used for goals, again in
order to save space. We might someday want to combine with `inputDrvs`,
by doing `V = bool` instead of `V = std::set<OutputName>`, but we are
not doing that yet for sake of a smaller diff.
The ATerm format for Derivations also needs to be extended, in addition
to the in-memory format. To accomodate this, we added a new basic
versioning scheme, so old versions of Nix will get nice errors. (And
going forward, if the ATerm format changes again the errors will be even
better.)
`parsedStrings`, an internal function used as part of parsing
derivations in A-Term format, used to consume the final `]` but expect
the initial `[` to already be consumed. This made for what looked like
unbalanced brackets at callsites, which was confusing. Now it consumes
both which is hopefully less confusing.
As part of testing, we also created a unit test for the A-Term format for
regular non-experimental derivations too.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Apply suggestions from code review
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Solves 1/3 of the infinite recursion at unknown location meme.
See #8879 for ensuring we always have a trace (for stack overflows)
We might want to re-add this for finding missing location info
*while hacking on that problem only*.
To avoid dealing with an optional `drvPath` (because we might not know
it yet) everywhere, make an `CreateDerivationAndRealiseGoal`. This goal
just builds/substitutes the derivation file, and then kicks of a build
for that obtained derivation; in other words it does the chaining of
goals when the drv file is missing (as can already be the case) or
computed (new case).
This also means the `getDerivation` state can be removed from
`DerivationGoal`, which makes the `BasicDerivation` / in memory case and
`Derivation` / drv file file case closer together.
The map type is factored out for clarity, and because we will soon hvae
a second use for it (`Derivation` itself).
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
In the Nix language, given a drv path, we should be able to construct
another string referencing to one of its output. We can do this today
with `(import drvPath).output`, but this only works for derivations we
already have.
With dynamic derivations, however, that doesn't work well because the
`drvPath` isn't yet built: importing it like would need to trigger IFD,
when the whole point of this feature is to do "dynamic build graph"
without IFD!
Instead, what we want to do is create a placeholder value with the right
string context to refer to the output of the as-yet unbuilt derivation.
A new primop in the language, analogous to `builtins.placeholder` can be
used to create one. This will achieve all the right properties. The
placeholder machinery also will match out the `outPath` attribute for CA
derivations works.
In 60b7121d2c we added that type of
placeholder, and the derived path and string holder changes necessary to
support it. Then in the previous commit we cleaned up the code
(inspiration finally hit me!) to deduplicate the code and expose exactly
what we need. Now, we can wire up the primop trivally!
Part of RFC 92: dynamic derivations (tracking issue #6316)
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
We want to be able to write down `foo.drv^bar.drv^baz`:
`foo.drv^bar.drv` is the dynamic derivation (since it is itself a
derivation output, `bar.drv` from `foo.drv`).
To that end, we create `Single{Derivation,BuiltPath}` types, that are
very similar except instead of having multiple outputs (in a set or
map), they have a single one. This is for everything to the left of the
rightmost `^`.
`NixStringContextElem` has an analogous change, and now can reuse
`SingleDerivedPath` at the top level. In fact, if we ever get rid of
`DrvDeep`, `NixStringContextElem` could be replaced with
`SingleDerivedPath` entirely!
Important note: some JSON formats have changed.
We already can *produce* dynamic derivations, but we can't refer to them
directly. Today, we can merely express building or example at the top
imperatively over time by building `foo.drv^bar.drv`, and then with a
second nix invocation doing `<result-from-first>^baz`, but this is not
declarative. The ethos of Nix of being able to write down the full plan
everything you want to do, and then execute than plan with a single
command, and for that we need the new inductive form of these types.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
This enables nix to correctly report what will be fetched in the case
that everything is a cache hit.
Note however that if an intermediate build of something which is not
cached could still cause products to end up being substituted if the
intermediate build results in a CA path which is in the cache.
Fixes#8615.
Signed-off-by: Peter Waller <p@pwaller.net>
