with everything being rendered from markdown now we no longer need to
postprocess any options.xml that may be requested from elsewhere. we'll
don't need to keep the module path check either since that's done by
optionsJSON now.
docbook is now gone and we can flip the defaults. we won't keep the
command line args around (unlike the make-options-docs argument) because
nixos-render-docs should not be considered an exposed API.
with docbook no longer supported we can default to markdown option docs.
we'll keep the parameter around for a bit to not break external users
who set it to true. we don't know of any users that do, so the
deprecation period may be rather short for this one.
it's been long in the making, and with 23.05 out we can finally disable
docbook option docs and default to markdown instead. this brings a
massive speed boost in manual and manpage builds, so much so that we may
consider enabling user module documentation by default.
we don't remove the docbook support code entirely yet because it's a lot
all over, and probably better removed in multiple separate changes.
- `wait_until_fails` was not passing through its `timeout` argument to
the internal `retry` function, hence was always using 900 seconds (the
default timeout for `retry`) rather than the user-specified value.
Previously, `wait_for_console_text` would block indefinitely until there were lines
shown in the buffer.
This is highly annoying when testing for things that can just hang for some reasons.
This introduces a classical timeout mechanism via non-blocking get on the Queue.
This is useful whenever you want to diagnose the current state of UEFI
variables, to assert that bootloaders or boot programs (systemd-stub)
did their job correctly and set their variables accordingly.
In the future, it can enable inspecting SecureBoot keys also.
This warning was added a year and a half ago, but still no test in
NixOS directly instantiates the machine class, presumably because it's
not actually possible for a test to do so without losing
functionality. For example, there's no way for a NixOS test to access
the output directory that create_machine passes to the Machine
constructor.
This warning is therefore just contributing to alert fatigue for
users, who are unable to follow its advice. Once it's actually
possible to do what it suggests, the warning can be reintroduced.
Adds a new option to the virtualisation modules that enables specifying explicitly named network interfaces in QEMU VMs.
The existing `virtualisation.vlans` option is still supported for cases where the name of the network interface is irrelevant.
By adding this option indirection, a test can declare all by itself
that it needs a custom nixpkgs. This is a more convenient way of
going about this when the caller of the test framework receives a
`node.pkgs` unconditionally.
mkIf is unnecessary when the condition is statically known - that is
knowable before entering the module evaluation.
By changing this to a precomputed module, we support changing the
defined options to readOnly options.
This change is made for two reasons:
1. If `toString config.restartTriggers` containes `\n`, systemd unit
file will be ill-formed.
2. This change can limit length of the trigger, although it doesn't
matter in most cases.
This allows modules that declare their class to be checked.
While that's not most user modules, frameworks can take advantage
of this by setting declaring the module class for their users.
That way, the mistake of importing a module into the wrong hierarchy
can be reported more clearly in some cases.
`make-disk-image` is a tool for creating VM images. It takes an argument
`contents` that allows one to specify files and directories that should
be copied into the VM image. However, directories end up not at the
specified target, but instead at a subdirectory of the target, with a
nix-store-like path, e.g.
`/target/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-source`. See issue
https://github.com/NixOS/nixpkgs/issues/226203 .
This change adds a test for make-disk-image's contents directory
handling and adds a fix (appending `/` to rsync input directory names).
This closes issue https://github.com/NixOS/nixpkgs/issues/226203 .
What the code was trying to do was helpfully add a directory and
extension if none were specified, but it did this by checking whether
the filename was composed of a very limited character set that didn't
even include dashes.
With this change, the intention of the code is clearer, and I can put
dashes in my screenshot names.
the old method of pasting parts of options.json into a markdown document
and hoping for the best no longer works now that options.json contains
more than just docbook. given the infrastructure we have now we can
actually render options.md properly, so we may as well do that.
options processing is pretty slow right now, mostly because the
markdown-it-py parser is pure python (and with performance
pessimizations at that). options parsing *is* embarassingly parallel
though, so we can just fork out all the work to worker processes and
collect the results.
multiprocessing probably has a greater benefit on linux than on darwin
since the worker spawning method darwin uses is less efficient than
fork() on linux. this hasn't been tested on darwin, only on linux, but
if anything darwin will be faster with its preferred method.
The output of a command is not guaranteed to be valid UTF-8, so the
decoding can fail raising UnicodeDecodeError. If this happens during a
`succeeds` the check will be erroneously marked failed.
This changes the error handling to the "replace" mode, where invalid
codepoints are replaced with � (REPLACEMENT CHARACTER U+FFFD) and the
decoding can go on.
...for explicitly named network interfaces
This reverts commit 6ae3e7695e.
(and evaluation fixups 08d26bbb727aed90a969)
Some of the tests fail or time out after the merge.
using environment variables isn't great once multiple input or output
formats get involved (which will happen soon). now is a good time to set
a pattern for future converters.
this new package shall eventually contain the rendering code necessary
to produce the entirety of the nixos (not nixpkgs) manual, in all of its
various output formats.
`shell_interact()` is currently not nice to use. If you try to cancel
the socat process, it will also break the nixos test. Furthermore
ptpython creates it's own terminal that subprocesses are running in,
which breaks some of the terminal features of socat.
Hence this commit extends `shell_interact` to allow also to connect to
arbitrary servers i.e. tcp servers started by socat.
the rest of the nixos manual has them enabled, so we should enable them
here too for consistency.
this changes rendered output pervasively. changes also include quotes in
types (eg in `strings concatenated with "\n"`), but since those are not
code this is probably fine. if not we can probably add a myst role to
inhibit replacements.
the rules are fixed, and we want to support all of them (or throw a
useful error message). this will also become the base for a generic
renderer system, so let's just list all the rules statically.
this restores mergeJSON to its former glory if…merging json, and
extracts the MD rendering into a new script that will run instead of the
py+nix+xslt pipeline we previously ran to convert options.json to docbook.
this change alone gives a noticable performance boost when building
docs (18s instead of 27s to build optionsDocBook).
no changes to rendered output, except for a single example in the
rsnapshot module that uses hard tabs for indentation instead of spaces.
this probably isn't important.
docbook warnings remain with mergeJSON since the other processing steps
output single files instead of directories. since we'll only keep the
check until 23.11 this is probably also not important to fix.
also contains a few improvements to error reporting in the MD renderers.
Adds a new option to the virtualisation modules that enables specifying
explicitly named network interfaces in QEMU VMs. The existing
`virtualisation.vlans` is still supported for cases where the name of
the network interface is irrelevant.
only whitespace changes (mostly empty descriptions rendered as literal
line breaks and trailing space toPretty generates, but that were dropped
by mistune).
don't generate docbook for related packages, generate markdown instead.
this could be extended further to not even generate markdown but have
mergeJSON handle all of the rendering. markdown will work fine for now
though.
only whitespace changes to rendered outputs, all in the vicinity or body
of admonitions. previously admonitions would not receive paragraph
breaks even when they should have because the description postprocessing
did not match on their contents.
markdown-it-py creates different whitespace leaders/trailers than are
currently emitted, and when we convert examples and defaults to render
via markdown the spacing will change too. this has no effect on rendered
output.
mistune already does escaping. it does escaping for html, but the
difference is small enough that can just ignore that we're actually
targeting docbook here.
this was done only to make the conversion to MD easier to verify. we no
longer need it, and not keeping whitespace does not affect rendered outputs.
stripping will have to stay for now because description postprocessing
would add empty paragraphs otherwise.
following the plan in https://github.com/NixOS/nixpkgs/pull/189318#discussion_r961764451
also adds an activation script to print the warning during activation
instead of during build, otherwise folks using the new CLI that hides
build logs by default might never see the warning.
checkInputs used to be added to nativeBuildInputs. Now we have
nativeCheckInputs to do that instead. Doing this treewide change allows
to keep hashes identical to before the introduction of
nativeCheckInputs.
mkAliasOptionModule should not default to mdDoc descriptions because
that can break out-of-tree users of documentation infrastructure. add an
explicitly-MD variant for now, to be removed some time after the MD
transition is complete.
A few places used Unicode U+2018/U+2019 left/right single quotes (but
not always correctly balanced). Let's just use plain ASCII single quotes
everywhere.
- Extensive documentation in NixOS manual
- Deterministic mode that fixes various identifiers relative to disk
partitions and filesystems in ext4 case
- UEFI variable recording
Render un`_type`d defaults and examples as `literalExpression`s using
`lib.generators.toPretty` so that consumers don't have to reinvent Nix
pretty-printing. `renderOptionValue` is kept internal for now intentionally.
Make `toPretty` print floats as valid Nix values (without a tilde).
Get rid of the now-obsolete `substSpecial` function.
Move towards disallowing evaluation of packages in the manual by
raising a warning on `pkgs.foo.{outPath,drvPath}`; later, this should
throw an error. Instead, module authors should use `literalExpression`
and `mkPackageOption`.
Unlike the XML doc renderer, the AsciiDoc and CommonMark renderers don't
pretty-print certain complex types, like literal expressions, DocBook
literals, and derivations. These types are dumped into the documentation
as JSON.
This commit parses and unwraps these types when loading the
JSON-formatted NixOS options. The AsciiDoc and CommonMark renders have
also been combined into a single script to allow code reuse.
Before this patch, if we:
1. add a new user in the config,
2. mount /home seperately,
3. not set neededForBoot for /home,
4. and run `nixos-rebuild boot`,
the newly added user's home will not be created after a reboot. This
is because when nixos/modules/config/update-users-groups.pl is running
in stage 2 to setup users, /home is not mounted.
This patch fixes this issue.
\<foo\> will often be displayed like \<foo>, for example by mkdocs.
I've tested a number of markdown renderers and they render html escape
sequences fine.
Until this commit, the documentation suggested `multi-user.target`
as `wantedBy` for all services.
Since `multi-user.target` is not available for user services,
propose a different default for those in the documentation.
Co-authored-by: Naïm Favier <n@monade.li>
Co-authored-by: Florian Warzecha <liketechnik@disroot.org>
Provide a window during which both solutions are valid without
warnings, in order to fight warning fatigue, and not to push 3rd
party repo maintainers to add unnecessary compat code.
Fixes the problem introduced by 12b3066aae
which caused nixos/release.nix to return the wrong attributes, while
intending to only affect nixos/lib's runTest.
This also removes callTest from the test options, because callTest is
only ever invoked by all-tests.nix.
It has been replaced by the modular test framework in nixos/lib/testing.
If you are looking for a way to produce a VM-test-like configuration
outside of the test framework, use the nixos/lib/testing/nixos-test-base.nix
NixOS module, possibly in combination with { _module.args.nodes = .....; }.
Before this, it relied on being able to `imports` from the `name`
module argument, which wasn't really necessary and required a
potentially quite breaking change. We can revert that now.
This is a decomposition of the testing-python.nix and build-vms.nix
files into modules.
By refactoring the glue, we accomplish the following:
- NixOS tests can now use `imports` and other module system features.
- Network-wide test setup can now be reusable; example:
- A setup with all VMs configured to use a DNS server
- Split long, slow tests into multiple tests that import a
common module that has most of the setup.
- Type checking for the test arguments
- (TBD) "generated" options reference docs
- Aspects that had to be wired through all the glue are now in their
own files.
- Chief example: interactive.nix.
- Also: network.nix
In rewriting this, I've generally stuck as close as possible to the
existing code; copying pieces of logic and rewiring them, without
changing the logic itself.
I've made two exceptions to this rule
- Introduction of `extraDriverArgs` instead of hardcoded
interactivity logic.
- Incorporation of https://github.com/NixOS/nixpkgs/pull/144110
in testScript.nix.
I might revert the latter and split it into a new commit.
deprecate literalDocBook by adding a warning (that will not fire yet) to
its uses and other docbook literal strings by adding optional warning
message to mergeJSON.
For example, the wait_for_unit() call in the Moodle test times out for
myself and others[1], so it would be good to be able to increase it to
something less likely to be hit by a test that would otherwise pass.
[1]: https://github.com/NixOS/nixpkgs/pull/177052#issue-1266336706
conversions were done using https://github.com/pennae/nix-doc-munge
using (probably) rev f34e145 running
nix-doc-munge nixos/**/*.nix
nix-doc-munge --import nixos/**/*.nix
the tool ensures that only changes that could affect the generated
manual *but don't* are committed, other changes require manual review
and are discarded.
leaving some newlines around after an admonition was closed causes the
newline rule to match, which in turn inserts literallayout newlines into
te xml output. that's not what we want.
Pass `-t` to pixz to prevent it from appending an index to the end of
the uncompressed stream, confusing tools such as `machinectl import-tar`.
Fixes: #187816
with ever more options being markdown rather than docbook the conversion
time is starting to become a significant factor of doc build time.
luckily we can pre-convert all nixos option docs to MD and cache the
result of this conversion, then merge the already-converted json file
with user option docs. we leave options.json unconverted to keep it as
close to the actual nix code as possible.
during docs conversion it can be very useful to know exactly *where* the
error the script complained about is. the name of the option should be
sufficient since option merging is rather rare, and won't merge doc
attributes anyway.
Within a dual VM test-setup a strange behaviour was observed.
The two VMs are connected via one vde_switch instance
(instancevirtualisation.vlans = [ 1 ]; IMO a bad attribute name for
switch instances, has nothing to do with VLANs in sense of 802.1Q).
A ping on the base interface (eth1) works, but not on VLAN
subinterfaces (vlan1@eth1). A tcpdump of eth1 includes the ARP requests
tagged with the subinterfaces VLAN ID, but responses seems not to pass
the vde_switch. This works fine if performed on the base interface.
Putting the vde_switch in hub mode results in flooding
traffic to all vde_switch ports. This results in a expected behaviour
and a ping on a VLAN subinterface works as expected.
Signed-off-by: Philippe Schaaf <philippe.schaaf@secunet.com>
Previously, the location logic was hardcoded, supporting only
Nixpkgs and NixOps properly, leaving other uses of the module
system without good location support.
- initialSystem was keeping track of the evaluating system
- it had been used by `nesting.children`
- since, 20.09, `nesting.children` has been replaced with named
specializations
It appears that this option was left over and not cleand up properly.
Without this fix, setting the shellopts in `machine.execute` is
inconsitent. When no timeout is used, shellopts `set -euo pipefail` are
applied to the command as expected. When a timeout is specified, the
shellopts are not applied to the command itself (which is called inside
a `sh -c` that doesn't inherit the shellopts) but rather to the
`timeout` command, leading to the following full command:
```bash
(set -euo pipefail; timeout 900 sh -c 'cmd') | (base64 --wrap 0; echo)\n
```
With this fix, this is the command we get:
```bash
timeout 900 sh -c 'set -euo pipefail; false | true') | (base64 --wrap 0; echo)\n
```
Documents the _module.args option, motivated by many usages in Flakes,
especially with the deprecation of extraArgs
(78ada83361)
The documentation rendering for this option had to be handled a bit
specially, since it's not declared in nixos/modules like all the other
NixOS options.
Co-Authored-By: pennae <github@quasiparticle.net>
Co-Authored-By: Robert Hensing <robert@roberthensing.nl>
