This commit creates flakes.nix, which is a library containing functions
which relate to interacting with flakes. It also moves related functions
from trivial.nix into it.
This is essentially a copy of the function of the same name, from
flake-compat. callLocklessFlake is useful when trying to utilise a
flake.nix without a lock file, often for when you want to create a
subflake from within a parent flake.
Co-authored-by: Tom Bereknyei <tomberek@gmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Closes#168327
The issue reported there can be demonstrated with the following
expression:
→ nix-instantiate --eval -E "with import ./. {}; pkgs.lib.options.showDefs [ { file = \"foo\"; value = pkgs.rust.packages.stable.buildRustPackages; } ]"
error: attempt to call something which is not a function but a string
at /home/ma27/Projects/nixpkgs/lib/trivial.nix:442:35:
441| isFunction = f: builtins.isFunction f ||
442| (f ? __functor && isFunction (f.__functor f));
| ^
443|
Basically, if a `__functor` is in an attribute-set at depth-limit,
`__functor` will be set to `"<unevaluated>"`. This however breaks
`lib.isFunction` which checks for a `__functor` by invoking `__functor`
with `f` itself.
The same issue - "magic" attributes being shadowed by `withRecursion` -
also applies to others such as
`__pretty`/`__functionArgs`/`__toString`.
Since these attributes have a low-risk of causing a stack overflow
(because these are flat attr-sets or even functions), ignoring them in
`withRecursion` seems like a valid solution.
This allows other module system consumers to
disable these docs via option merging.
For instance arion uses asciidoc instead of
docbook so that would look awful.
This commit adds an `isPower64` predicate to the two existing
predicates for this architecture (`isPower` and `isPowerPC`).
Note that `isPowerPC` matches only 32-bit machines, whereas `isPower`
matches both 64-bit and 32-bit machines. Prior to this commit there
was no single `isXXX` predicate for `powerpc64le`.
Documents the _module.args option, motivated by many usages in Flakes,
especially with the deprecation of extraArgs
(78ada83361)
The documentation rendering for this option had to be handled a bit
specially, since it's not declared in nixos/modules like all the other
NixOS options.
Co-Authored-By: pennae <github@quasiparticle.net>
Co-Authored-By: Robert Hensing <robert@roberthensing.nl>
This uses the levenshtein distance to look through all possible
arguments to find ones that are close to what was requested:
error: Function in /home/infinisil/src/nixpkgs/pkgs/tools/text/ripgrep/default.nix
called without required argument "fetchFromGithub",
did you mean "fetchFromGitHub" or "fetchFromGitLab"?
With https://github.com/NixOS/nix/pull/3468 (in current nixUnstable) the error
message becomes even better, adding line location info
Adds some functions related to string similarity:
- lib.strings.commonPrefixLength
- lib.strings.commonSuffixLength
- lib.strings.levenshtein
- lib.strings.levenshteinAtMost
types.optionSet has been deprecated for almost 10 years now
(0e333688ce)! A removal
was already attempted in 2019
(27982b408e), but it was promptly
reinstantiated since some third-party uses were discovered
(f531ce75e4).
It's finally time to remove it for good :)
The current logic assumes that everything that isn't a derivation is a
store path, but it can also be something that's *coercible* to a store
path, like a flake input.
Unnecessary uses of `lib.toDerivation` result in errors in pure evaluation
mode when `builtins.storePath` is disabled.
Also document what a `package` is.
MIPS has a large space of {architecture,abi,endianness}; this commit
adds all of them to lib/systems/platforms.nix so we can be done with
it.
Currently lib/systems/inspect.nix has a single "isMips" predicate,
which is a bit ambiguous now that we will have both mips32 and mips64
support, with the latter having two ABIs. Let's add four new
predicates (isMips32, isMips64, isMips64n32, and isMips64n64) and
treat the now-ambiguous isMips as deprecated in favor of the
more-specific predicates. These predicates are used mainly for
enabling/disabling target-specific workarounds, and it is extremely
rare that a platform-specific workaround is needed, and both mips32
and mips64 need exactly the same workaround.
The separate predicates (isMips64n32 and isMips64n64) for ABI
distinctions are, unfortunately, useful. Boost's user-scheduled
threading (used by nix) does does not currently supports mips64n32,
which is a very desirable ABI on routers since they rarely have
more than 2**32 bytes of DRAM.
This reverts commit 6b077c47ff.
Thanks Infinisil for discovering this problem:
> After a lot of trial and error, trying to prove why fixupOptionType should
> be used here or not, I figured it out: It's needed for the sake of file
> locations in error messages.
... where a bare submodule is an option that has a type like
`submoduleWith x`, as opposed to `attrsOf (submoduleWith x)`.
This makes migration unnecessary when introducing a freeform type
in an existing option tree.
Closes#146882
This ensures that the module file locations are propagated to the
freeform type, which makes it so that submodules in freeform types now
have their declaration location shown in the manual, fixing
https://github.com/NixOS/nixpkgs/issues/132085.
In addition, this also newly allows freeformTypes to be declared
multiple times and all declarations being merged together according to
normal option merging.
This also removes some awkwardness regarding the type of `freeformType`
This type correctly merges multiple option types together while also
annotating them with file information. In a future commit this will be
used for `_module.freeformType`
Makes all options rendered in the manual throw an error if they don't
have a type specified.
This is a follow-up to #76184
Co-Authored-By: Silvan Mosberger <contact@infinisil.com>
Change comment type so than nixdoc picks them up into Nixpkgs manual.
Also improve phrasing a bit and move stuff around so that it is formatted better.
This is a squashed commit. These are the original commit messages:
lib/option: Improve comment
better comment
Update documentation
Updated nixos/doc/manual/development/options-declarations.md with info on mkEnableOption and mkPackageOption.
Updated the comment on mkEnableOption in lib/options.nix
remove trailing whitespace
nixos/doc/option-declarations: Update IDs & formatting
nixos/docs/option-declarations: Escape angle brackets
Build DB from MD
(Amended) Fix typo
Co-authored-by: pennae <82953136+pennae@users.noreply.github.com>
(Amended) Build DB from MD (again)
This is a squashed commit. These are the original commit messages:
lib/options: Add mkPackageOption
lib/options: Add missing semicolon
lib/options.nix: Make mkPackageOption more complicated
lib/options: Fix indent. & spacing
lib/options.nix: Remove example and align comment
lib/options: ravenous overuse of arguments
lib/options: Format better
lib/options: Add default examplePath decl
lib/options: Make better mkPackageOption function
lib/options: Remove trailing whitespace
lib/options: Improve mkPackageOptions
lib/options: Remove pkgs prefixing
Co-authored-by: pennae <82953136+pennae@users.noreply.github.com>
lib/options: Slim down mkPackageOption further
lib/options: mkPackageOption: Add "pkgs." to example
lib/options: mkPackageOption: Make name & pkgs single arguments
lib/options: mkPackageOption: Swap name & pkgs
lib/options: Remove unnecessary import
Co-authored-by: pennae <82953136+pennae@users.noreply.github.com>
Allow a \n character at the end of the string and remove it during the
merge function.
An option of this type will resolve to the value "foo" whether it is set
to "foo" or "foo\n".
This is useful when using 'builtins.readFile' or ''-strings, which might
add an unintended newline (for example, bash trim the final newline from
a subshell).
`assert` has the annoying property that it dumps a lot of code at the
user without the built in capability to display a nicer message. We have
worked around this using `assertMsg` which would *additionally* display
a nice message. We can do even better: By using `throw` we can make
evaluation fail before assert draws its conclusions and prevent it from
displaying the code making up the assert condition, so we get the nicer
message of `throw` and the syntactical convenience of `assert`.
Before:
nix-repl> python.override { reproducibleBuild = true; stripBytecode = false; }
trace: Deterministic builds require stripping bytecode.
error: assertion (((lib).assertMsg (reproducibleBuild -> stripBytecode)) "Deterministic builds require stripping bytecode.") failed at /home/lukas/src/nix/nixpkgs/pkgs/development/interpreters/python/cpython/2.7/default.nix:45:1
After:
nix-repl> python.override { reproducibleBuild = true; stripBytecode = false; }
error: Deterministic builds require stripping bytecode.
Add a new type, inheriting 'types.str' but checking whether the value
doesn't contain any newline characters.
The motivation comes from a problem with the
'users.users.${u}.openssh.authorizedKeys' option.
It is easy to unintentionally insert a newline character at the end of a
string, or even in the middle, for example:
restricted_ssh_keys = command: keys:
let
prefix = ''
command="${command}",no-pty,no-agent-forwarding,no-port-forwarding,no-X11-forwarding
'';
in map (key: "${prefix} ${key}") keys;
The 'prefix' string ends with a newline, which ends up in the middle of
a key entry after a few manipulations.
This is problematic because the key file is built by concatenating all
the keys with 'concatStringsSep "\n"', with result in two entries for
the faulty key:
''
command="...",options...
MY_KEY
''
This is hard to debug and might be dangerous. This is now caught at
build time.
most modules can be evaluated for their documentation in a very
restricted environment that doesn't include all of nixpkgs. this
evaluation can then be cached and reused for subsequent builds, merging
only documentation that has changed into the cached set. since nixos
ships with a large number of modules of which only a few are used in any
given config this can save evaluation a huge percentage of nixos
options available in any given config.
in tests of this caching, despite having to copy most of nixos/, saves
about 80% of the time needed to build the system manual, or about two
second on the machine used for testing. build time for a full system
config shrank from 9.4s to 7.4s, while turning documentation off
entirely shortened the build to 7.1s.
- Eta reduce `mapAttrsRecursiveCond`, `foldAttrs`, `getAttrFromPath`.
- Modify `matchAttrs` to use `elemAt` instead of `head (tail xs)` to access
elements.
- Modify `matchAttrs` to use `any id` instead of `foldr and true`.
- Eta reduce formal arguments of `recursiveUpdate'.
- Access elements in `recursiveUpdateUntil` using `elemAt` and `head`
directly instead of `head (tail xs)` which copies a singleton unnecessarily.
(`elemAt` is used instead of `last` to save a primitive call to `length`,
this is possible because the 2-tuple structure is guranteed)
- Use `length` instead of comparison to empty list to save a copy.
the foldl is equivalent to a zip with concat. list concatenation in nix
is an O(n) operation, which makes this operation extremely inefficient
when large numbers of modules are involved.
this change reduces the number of list elements by 7 million on the
system used to write this, total memory spent on lists by 58MB, and
total memory allocated on the GC heap by almost 100MB (with a similar
reduction in GC heap size). it's also slightly faster.
While it is a fact of life that aarch64-darwin is built on Hydra, it has
never formally been elevated from the Tier 7 state it was originally
assigned in RFC 0046. Since platform Tier status is not only
descriptive, but also normative, a consensus to commit to supporting
aarch64-darwin would need to be reached.
`builtins.currentSystem` is not available in pure eval. For this
particular test, we don't really care since it's all about generating
.drv files.
Fixes the following error:
$ nix flake check
warning: unknown flake output 'lib'
error: attribute 'currentSystem' missing
at /nix/store/8wvnlbjxlr90kq2qa6d9zjpj8rqkilr5-source/lib/tests/misc.nix:499:73:
498| let
499| deriv = derivation { name = "test"; builder = "/bin/sh"; system = builtins.currentSystem; };
| ^
500| in {
(use '--show-trace' to show detailed location informat
Move function spdxLicense, internally used in yarn2nix
to lib/meta.nix, and
rename to getLicenseFromSpdxId
A similar function is implemented in poetry2nix,
but the one originally in yarn2nix seems beter.
since it falls back to an license-like attrset
for mismatched case
instead of a plain string
Makes any programming errors more likely to show up early.
Non-obvious changes because of this:
- Ignore the `evalConfig` result in `reportFailure`; we're not checking
it at that point.
- Pre-increment `$fail` and `$pass` to make sure the arithmetic doesn't
result in a zero, which would result in a non-zero exit code for the
expression.
mkDerivedConfig : Option a -> (a -> Definition b) -> Definition b
Create config definitions with the same priority as the definition of another option.
This should be used for option definitions where one option sets the value of another as a convenience.
For instance a config file could be set with a `text` or `source` option, where text translates to a `source`
value using `mkDerivedConfig options.text (pkgs.writeText "filename.conf")`.
It takes care of setting the right priority using `mkOverride`.
recursiveUpdate does not produce an attrset until it has evaluated
both its arguments to weak head normal form.
nix-repl> lib.recursiveUpdate (throw "a") (throw "b")
error: b
nix-repl> lib.recursiveUpdate (throw "a") {}
error: a
The current implementation of `mutuallyExclusive` builds a new list with
length subtracted by one on every recursive call which is expensive. When
b is empty, the function still traverses a in its entirety before returning
a result.
The new implementation uses `any` to check if each element of list b is in
list a using `elem`. This maintains short circuiting when list a or b is empty
and has a worst case time complexity of O(nm).
the fix to extendDerivation in #140051 unwittingly worsened eval performance by
quite a bit. set elements alone needed over 1GB extra after the change, which
seems disproportionate to how small it was. if we flip the logic used to
determine which outputs to install around and keep a "this one exactly" flag in
the specific outputs instead of a "all of them" in the root we can avoid most
of that cost.
if extendDerivation is called on something that already had extendDerivation
called on it (eg a mkDerivation result) the second call will set
outputUnspecified=true on every output by way of propagating attributes of the
full derivation to the individual outputs. this in turn causes buildEnv--and
thus nix-shell and environment.systemPackages--to install every output of such a
derivation even when only a specific output was requested, which renders the
point of multiple outputs moot. this happens in python modules (see #139756),
though it seems that tcl and possibly others should also be affected.
PowerNV was looking for a nonexisting zImage file.
Remove unnecessary .file / .installTarget.
Also add config options needed for default minimal
NixOS config and QEMU VirtIO/VirtFS devices.
I'm working on a project that involves running a virtual machine
monitor program, which creates a control socket in the project
directory (because it doesn't make sense to put it anywhere else).
This obviously isn't part of the source of my program, so I think
cleanSource should filter it out.
Consider a derivation a value to be serialized.
nix-repl> lib.generators.toGitINI { hello = { drv = pkgs.hello; }; }
error: evaluation aborted with the following error message: 'generators.mkValueStringDefault: attrsets not supported: <derivation /nix/store/533q15q67sl6dl0272dyi7m7w5pwkkjh-hello-2.10.drv>'
Fixes#137390
The current implementation of the concatStringsSep fallback references
concatStrings whcih is just a partial application of concatStringsSep,
forming a circular dependency. Although this will almost never be
encountered as (assuming the user does not explicitly trigger it):
1. the or operator will short circuit both in lazy and strict
evaluation
2. this can only occur in Nix versions prior to 1.10
which is not compatible with various nix operations as of 2.3.15
However it is still important if scopedImport is used or the builtins
have been overwritten. lib.foldl' is used instead of builtins.foldl'
as the foldl' primops was introduced in the same release as concatStringsSep.
As suggested in #131205.
Now it's possible to pretty-print a value with `lib.generators` like
this:
with lib.generators;
toPretty { }
(withRecursion { depthLimit = 10; } /* arbitrarily complex value */)
Also, this can be used for any other pretty-printer now if needed.
The message I originally implemented here was to catch a mixup of
`config' and `options' in a `types.submodule'[1]. However it looks
rather weird for a wrongly declared top-level option.
So I decided to throw
error: The option `foo' does not exist. Definition values:
- In `<unknown-file>':
{
bar = {
_type = "option";
type = {
_type = "option-type";
...
It seems as you're trying to declare an option by placing it into `config' rather than `options'!
for an expression like
with import ./lib;
evalModules {
modules = [
{
foo.bar = mkOption {
type = types.str;
};
}
];
}
[1] fa30c9abed
When having a bogus declaration such as
{ lib, ... }:
{
foo.bar = mkOption {
type = types.str;
};
}
the evaluation will terminate with a not-so helpful
error: stack overflow (possible infinite recursion)
To make sure a useful error is still provided, I added a `depthLimit` of
`10` which should be perfectly sufficient to `toPretty` when it's used
in an error-case for `showDefs`.
When having e.g. recursive attr-set, it cannot be printed which is
solved by Nix itself like this:
$ nix-instantiate --eval -E 'let a.b = 1; a.c = a; in builtins.trace a 1'
trace: { b = 1; c = <CYCLE>; }
1
However, `generators.toPretty` tries to evaluate something until it's
done which can result in a spurious `stack-overflow`-error:
$ nix-instantiate --eval -E 'with import <nixpkgs/lib>; generators.toPretty { } (mkOption { type = types.str; })'
error: stack overflow (possible infinite recursion)
Those attr-sets are in fact rather common, one example is shown above, a
`types.<type>`-declaration is such an example. By adding an optional
`depthLimit`-argument, `toPretty` will stop evaluating as soon as the
limit is reached:
$ nix-instantiate --eval -E 'with import ./Projects/nixpkgs-update-int/lib; generators.toPretty { depthLimit = 2; } (mkOption { type = types.str; })' |xargs -0 echo -e
"{
_type = \"option\";
type = {
_type = \"option-type\";
check = <function>;
deprecationMessage = null;
description = \"string\";
emptyValue = { };
functor = {
binOp = <unevaluated>;
name = <unevaluated>;
payload = <unevaluated>;
type = <unevaluated>;
wrapped = <unevaluated>;
};
getSubModules = null;
getSubOptions = <function>;
merge = <function>;
name = \"str\";
nestedTypes = { };
substSubModules = <function>;
typeMerge = <function>;
};
}"
Optionally, it's also possible to let `toPretty` throw an error if the
limit is exceeded.
- Remove inheritance of `lists.fold` as it isn't used anywhere.
- Inherit `foldl'` for consistency as only `cartesianProductOfSets` explicitly
reference lib.
- Inline `foldr` to generate nested attrs instead of using `listToAttrs` and `tail`.
This allows checking e.g. stdenv.hostPlatform.isGnu, just like isMusl
or isUClibc. It was already possible to check for glibc with
stdenv.hostPlatform.libc == "glibc", but when that doesn't line up
with how every other platform check works, this is apparently
sufficiently non-obvious that we've ended up with stuff like adding
glibc.static if !isMusl, which is obviously wrong.
This regressed in 9c213398b3
The recursiveUpdate gave the platform both gcc.cpu and gcc.arch attrs
instead of only gcc.cpu. This is invalid; gcc configuration fails with:
```
Switch "--with-arch" may not be used with switch "--with-cpu"
```
So we revert to using `//` to retain only gcc.cpu
(which is more specific than the processor arch).
Trying to create a simple flake with input:
```
{
inputs.nixpkgs-lib.url = "github:NixOS/nixpkgs?dir=lib";
outputs = {nixpkgs-lib, ...}: { };
}
```
Fails when trying to evaluate the nixpkgs-lib flake with an error like:
```
error: getting status of '/nix/store/xxxx-source/lib/lib': No such file or directory
(use '--show-trace' to show detailed location information)
```
Allows for distinction of licenses that are unfree overall but do grant the
right to redistribute. Defaults to the freeness of the license.
Note: Many unfree but are redistributable licenses aren't marked as such.
I expect that to be fixed in a distributed manner over time.
Closes https://github.com/NixOS/nixpkgs/pull/97789
Previously, if a derivation without a `drvPath` was handled, an error
would be thrown:
nix-repl> lib.generators.toPretty {} { type = "derivation"; }
error: attribute 'drvPath' missing, at /home/infinisil/src/nixpkgs/lib/generators.nix:251:24
With this commit it doesn't anymore:
nix-repl> lib.generators.toPretty {} { type = "derivation"; }
"<derivation ???>"
This matches what `nix repl` outputs:
nix-repl> { type = "derivation"; }
«derivation ???»
For an option definition that uses `lib.options.mergeEqualOption`
underneath, like `types.anything`, an error is thrown when multiple
functions are assigned, indicating that functions can't be compared for
equivalence:
error: The option `test' has conflicting definition values:
- In `xxx': <function>
- In `xxx': <function>
(use '--show-trace' to show detailed location information)
However, the error message didn't use the correct files. While above
error indicates that both definitions are in the xxx file, that's in
fact not true. Above error was generated with
options.test = lib.mkOption {
type = lib.types.anything;
};
imports = [
{
_file = "yyy";
test = y: y;
}
{
_file = "xxx";
test = x: x;
}
];
With this change, the error uses the correct file locations:
error: The option `test' has conflicting definition values:
- In `xxx': <function>
- In `yyy': <function>
(use '--show-trace' to show detailed location information)
so the underlaying use case of the preceding commit is so
generic, that we gain a lot in reasoning to give it an
appropriate name.
As the comment states:
image media needs to override host config short of mkForce
m68k was recently added for Linux and none, but NetBSD also supports
m68k. Nothing will build yet, but I want to make sure we at least
encode the existence of NetBSD support for every applicable
architecture we support for other operating systems.
In Autoconf, some old NetBSD targets like "i686-unknown-netbsd" are
interpreted as a.out, not elf, and virtually nothing supports it. We
need to specify e.g. "i686-unknown-netbsdelf" to get the right
behaviour.
Newer bootloaders for RISC-V (i.e., OpenSBI + U-Boot) support
flat and compressed kernel images but not vmlinux. Therefore,
let's build "Image" like what we do with aarch64.
Also copy DTBs while we are at it.
This will begin the process of breaking up the `useLLVM` monolith. That
is good in general, but I hope will be good for NetBSD and Darwin in
particular.
Co-authored-by: sterni <sternenseemann@systemli.org>
Stating that CPUs and the isCompatible relation forms a category (or
preorder) is correct but overtly technical. We can state it more
clearly for readers unfamiliar with mathematics while retaining some
keywords to be useful to technical readers.
This PR adds a new aarch64 android toolchain, which leverages the
existing crossSystem infrastructure and LLVM builders to generate a
working toolchain with minimal prebuilt components.
The only thing that is prebuilt is the bionic libc. This is because it
is practically impossible to compile bionic outside of an AOSP tree. I
tried and failed, braver souls may prevail. For now I just grab the
relevant binaries from https://android.googlesource.com/.
I also grab the msm kernel sources from there to generate headers. I've
included a minor patch to the existing kernel-headers derivation in
order to expose an internal function.
Everything else, from binutils up, is using stock code. Many thanks to
@Ericson2314 for his help on this, and for building such a powerful
system in the first place!
One motivation for this is to be able to build a toolchain which will
work on an aarch64 linux machine. To my knowledge, there is no existing
toolchain for an aarch64-linux builder and an aarch64-android target.
When a list is passed to isStorePath this is most likely a mistake and
it is therefore better to just return false. There is one case where
this theoretically makes sense (if a list contains a single element for
which isStorePath elem), but since that case is also probably seldomly
intentional, it may save someone from debbuging unclear evaluation
errors.
I am taking the non-invasive parts of #110914 to hopefully help out with #111988.
In particular:
- Use `lib.makeScopeWithSplicing` to make the `darwin` package set have
a proper `callPackage`.
- Adjust Darwin `stdenv`'s overlays keeping things from the previous
stage to not stick around too much.
- Expose `binutilsNoLibc` / `darwin.binutilsNoLibc` to hopefully get us
closer to a unified LLVM and GCC bootstrap.
Previously, an option of type
attrsOf string
wouldn't throw a deprecation warning, even though the string type is
deprecated. This was because the deprecation warning trigger only looked
at the type of the option itself, not any of its subtypes.
This commit fixes this, causing each of the types deprecationMessages to
trigger for the option. This relies on the subtypes mkOptionType
attribute introduced in 26607a5a2e06653fec453c83d063cdfc4b59185f
In 2d45a62899, the submodule type
description was amended with the freeformType description. This causes
all the modules passed to the submodule to be evaluated once on their
own, without any extra definitions from the config section. This means
that the specified modules need to be valid on their own, without any
undeclared options.
This commit adds a test that evaluates a submodules option description,
which would trigger the above problem for one of the tests, if it were
not fixed by this commit as well.
This is done because the next commit makes option evaluation a bit more
strict, which would also trigger this test failure, even though it's not
related to the change at all.
This will be used to issue deprecation warnings recursively in the next
commit
In addition, this allows easily getting nested types of other options, which
is useful when you want to create an option that aliases a part of
another one.
It's a common pattern in Nixpkgs to want to emit a warning in certain
cases, but not actually change behaviours.
This is often expressed as either
if cond then lib.warn "Don't do that thing" x else x
Or
(if cond then lib.warn "Don't do that thing" else lib.id) x
Neither of which really expresses the intent here, because it looks
like 'x' is being chosen conditionally.
To make this clearer, I introduce a "warnIf" function, which makes it
clear that the only thing being affected by the condition is whether
the warning is generated, not the value being returned.
