also updates nixdoc to 2.3.0. the nixdoc update is not a separate commit
because that would leave the manual build broken for one commit,
potentially breaking bisects and rebases.
this is only used in the stdenv chapter, but footnotes could be useful
in other places as well. since markdown-it has a plugin to parse
footnote syntax we may as well just support them even if they're rare.
examples and figures behave identically regarding numbering and titling,
they just don't share a common number space. make the numbering/titling
function generic over block types now so figures can just use it.
currently only supported for html. docbook could also support images,
but it's on the way out for manual generation anyway so we won't add
image support there. options docs can't use images because they also
target manpages, which leaves no viable users.
the nixpkgs manual uses section tocs for eg the nixpkgs library function
reference. we will only not emit tocs for subsections at this point, but
maybe we should consider doing so in the future.
docbook always emits intra-file links if only a single file is emitted.
it doesn't seem to do this for intra-file links when multiple files are
emitted, so we don't do that yet either.
This automated de-linting has applied a few different refactors:
- Remove unused imports and variables
- Change f-strings with no variables to regular strings
- Remove trailing semicolon
docbook is now gone and we can flip the defaults. we won't keep the
command line args around (unlike the make-options-docs argument) because
nixos-render-docs should not be considered an exposed API.
the nixos manual contains enough examples to support them as a proper
toc entity with specialized rendering, and if in the future the nixpkgs
wants to use nixos-render-docs we will definitely have to support them.
this also allows us to restore some examples that were lost in previous
translation steps because there were too few to add renderer support
back then.
this converter is currently supposed to be able to reproduce the
docbook-generated html DOMs exactly, though not necessarily the
html *files*. it mirrors many docbook behaviours that seem rather odd,
such as top-level sections in chapters using the same heading depth as
understood by html as their parent chapters do. over time we can
hopefully remove all special casing needed to reproduce docbook
rendering, but for now at least it doesn't hurt *too* much.
this will be necessary for html since there we have to do chunking into
multiple files ourselves. writing one file from the caller of the
converter and all others from within the converter is unnecessarily
spread out, and returning a dict of file names and their contents is not
quite as meaningful for docbook (which has only one file to begin with).
it's not hooked up to anything yet, but that will come soon. there's a
bit of docbook compat here that must be interoperable with the actual
docbook exporter, but luckily it's not all that much.
the basic html renderer. it doesn't have all the docbook compatibility
codes embedded into it, but there is a good amount. this renderer is
unaware of manual structure and does not traverse structural include
tokens (if it finds any it'll just fail), that task falls to derived
classes. once we have more uses for structural includes than just the
manual we may revisit this decision.
the docbook toolchain uses docbook-xsl to generate its TOC, our html
renderer will have to do this on its own. this generator uses a very
straight-forward algorithm of only inspecting headings, but anything
else could be inspected as well. (examples come to mind, but those do
not have titles and would thus make for bad toc entries)
we also use path information (that will be taken from include block args
in the html renderer) to produce navigation information. the algorithm
we use mirrors what docbook does, linking to the next/previous files in
depth-first toc order.
toc entries are linked to the tokens they refer to for easy use later.
while docbook relies on external chunk-toc info to do chunking of the
rendered manual we have nothing of the sort for html. there it seems
easiest to add annotations to blocks to create new chunks. such
annotations could be extended to docbook to create the chunk-toc instead
of passing it in externally, but with docbook on the way out that seems
like a waste of effort.
text content in the toplevel file of a book will not render properly.
the first proper element will be a preface, part, or chapter anyway, and
those require includes to produce.
parts do not currently allow headings in the part file itself, but
that's mainly a renderer limitation. we can add support for headings in
part intros when we need them
in all other cases includes must be followed by either another include,
a heading, or end of file. text content could not be properly linked to
from a TOC without a preceding heading.
without this we cannot build a TOC to arbitrary depth without generating
ids for headings, but generated ids are fragile and liable to either
break or point to different things if the manual changes shape. we
already have the convention that all headings should have an id, this
formalizes it.
while not technically necessary for correct rendering of *contents* we
do need to disallow heading levels being skipped to build a correct
TOC. treating headings that have skipped a number of levels to actually
be headings that many levels up only gets confusing, and inserting
artifical intermediate headings suffers from problems, such as which ids
to use and what to call them.
check that all required headings are present during parsing, not during
rendering. building a correct TOC will need this since every TOC entry
needs a heading to set its title, and every included substructure needs
a title.
also improve the error message on repeated title headings slightly,
giving the end line turns out to not be very useful.
for most of our data classes we can use dataclasses.dataclass with
frozen=True or even plain named tuples. the TOC structure we'll need to
generate proper navigation links is most easily represented and used as
a cyclic structure though, and for that we can use neither. if we want
to make the TOC structures immutable (which seems like a good idea)
we'll need a hack of *some* kind, and this hack seems like the least intrusive.
we should really be rendering options at *rendering* time, not at parse
time. currently this is just an academic exercise, but the html renderer
will have to inspect the options.json data after the entire document has
been parsed, but before anything gets rendered.
these weren't used for anything. options never was (and does not contain
any information for the renderer that we *want* to honor), and env is
not used because typed renderer state is much more useful for all our cases.
our renderers carry significantly more state than markdown-it wants to
easily cater for, and the html renderer will need even more state still.
relying on the markdown-it-provided rendering functions has already
proven to be a nuisance, and since parsing and rendering are split well
enough we can just replace the rendering part with our own stuff outright.
this also frees us from the tyranny of having to set instance variables
before calling super().__init__ just to make sure that the renderer
creation callback has access to everything it needs.
the old method of pasting parts of options.json into a markdown document
and hoping for the best no longer works now that options.json contains
more than just docbook. given the infrastructure we have now we can
actually render options.md properly, so we may as well do that.
inline anchors are not allowed in option docs per the manual, and the
sole class we current have (.keycap) is never used anyway. disallow them
for now to avoid future surprises.
the same goes for examples, which aren't even documented in the manual yet.
move the restrictions we care about into a mixin class. a few more
restrictions will appear soon and a few new converters as well, the
renderers of which need not have these restrictions already baked in by
accident (like the manpage renderer does right now).
add the ability to set the info string for a newly created fenced code
block, and a flag to always emit a fenced block. the commonmark
converter will need this to faithfully recreate fenced and indented code
blocks.
with mypy type checking and Mapping types this is a lot less useful than
anticipated. let's drop it for simplicity and having fewer dependencies.
frozendict 2.3.5 also broke the mypy checks.
options processing is pretty slow right now, mostly because the
markdown-it-py parser is pure python (and with performance
pessimizations at that). options parsing *is* embarassingly parallel
though, so we can just fork out all the work to worker processes and
collect the results.
multiprocessing probably has a greater benefit on linux than on darwin
since the worker spawning method darwin uses is less efficient than
fork() on linux. this hasn't been tested on darwin, only on linux, but
if anything darwin will be faster with its preferred method.
this adds support for structural includes to nixos-render-docs.
structural includes provide a way to denote the (sub)structure of the
nixos manual in the markdown source files, very similar to how we used
literal docbook blocks before, and are processed by nixos-render-docs
without involvement of xml tooling. this will ultimately allow us to
emit the nixos manual in other formats as well, e.g. html, without going
through docbook at all.
alternatives to this source layout were also considered:
a parallel structure using e.g. toml files that describe the document
tree and links to each part is possible, but much more complicated to
implement than the solution chosen here and makes it harder to follow
which files have what substructure. it also makes it much harder to
include a substructure in the middle of a file.
much the same goes for command-line arguments to the converter, only
that command-lined arguments are even harder to specify correctly and
cannot be reasonably pulled together from many places without involving
another layer of tooling. cli arguments would also mean that the manual
structure would be fixed in default.nix, which is also not ideal.
<part> is different from all other blocks we care about in that it
requires textual content to be wrapped in <partintro>. add support for
this to the generic docbook renderer, which will just assume that a part
is the whole document start to finish. we do make provision for the
manual renderer to close a partintro tag early though.
__context__ is always set to the prior exception, even when not using
the raise from form. __cause__ is only set during raise from. use
__cause__ so we can override a leaf exception (eg KeyError to something
more meaningful).
render all manual chapters to docbook from scratch every time the manual
is built. nixos-render-docs is quick enough at this to not worry about
the cost (needing only about a second), and it means we can remove
md-to-db.sh in the next commit.
no changes to the rendered html manual except for replacements and smartquotes.
we'll soon add another docbook converter that does not emit a section as
a collection of chapters, but sections or chapters on their own. this
should clarify naming a bit before there can be any confusion.
this is currently only supported by the docbook exporter, and even the
docbook exporter doesn't do much with them. we mirror the conversion
pandoc did for consistency with the previous manual chapter conversion,
which is to add just an anchor with the given id. future exporters that
go directly to html might want to do more.
this is a subset of pandoc's fenced divs. currently we only use this for
admonitions (which get a new name to differentiate them from other kinds
of blocks), but more users will appear soon.
this is used by release notes (and we don't want to break links to
those), and is also technically allowed anyway. we will *not* extend the
regex to allow more characters just yet due to a mozilla recommendation
against it (cf https://developer.mozilla.org/en-US/docs/Web/HTML/Global_attributes/id)
this should've been a core rule from the beginning. not being a core
rule made it always run after smartquotes and replacements, which
could've wrecked the id.
this lets us parse the `[F12]{.keycap}` syntax we recently introduced to
the nixos manual markdown sources. the docbook renderer emits the keycap
element for this class, the manpage renderer will reject it because it's
not entirely clear what to do with it: while html has <kbd> mandoc has
nothing of the sort, and with no current occurences in options doc we
don't have to settle on a (potentially bad) way to render these.
this is pretty much what pandoc calls bracketed spans. since we only
want to support ids and classes it doesn't seem fair to copy the name,
so we'll call them "attributed span" for now. renderers are expected to
know about *all* classes they could encounter and act appropriately, and
since there are currently no classes with any defined behavior the most
appropriate thing to do for now is to reject all classes.
now that the renderer produces the output we want to keep for the future
we can add a test that checks all of its features. this test notably
does not include markdown headings since we don't want to have those in
manpages (at least right now), but tests for other converters may add
headings for themselves.
for the longest time we completely dropped link targets in
configuration.nix.5. let's stop doing this now and instead provide a
footnote for each link in a given option, numbered locally per option.
we will currently duplicate the link for <labelless-links> because it
makes it easier to get the collection of all links in a given option.
this may not be useful enough, so over time we might decide to drop the
footnotes for such links.
indent the entire list by 4, just like each definition is already
indented by 4. this matches rendering in html, which indents terms once
and indents definitions twice.
other output types already have markings for inline code, manpages do
not. this can be somewhat confusing, so we'll do the least intrusive
thing: surrounding inline code blocks in ‘’. doing so separates inline
code from the rest of the text and is unlikely to collide with the
quoted contents. it's also what mdoc does with its Ql macro.
no reason to differentiate between links by source of their label. this
feature seems to be mostly used to change labels of links to other
options, but this should ultimately be done by auto-linking from
{option}`...`. at some point we may want to introduce a warning when
this pattern is encountered, but there's a lot to work out still before
we can do that.
most of the lists in option docs are actually compact, but docbook to
manpage processing always rendered them as non-compact. compactifying
these lists improves readability somewhat since most lists and their
contents are pretty short.
mdoc is just too slow to render on groff, and semantic markup doesn't
help us any for generated pages.
this produces a lot of changes to configuration.nix.5, but only few
rendering changes. most of those seem to be place losing a space where
docbook emitted roff code that did not faithfully represent the input
text, though a few places also gained space where docbook dropped them.
notably we also don't need the compatibility code docbook-xsl emitted
because that problem was fixed over a decade ago.
this will handle block quotes, which the docbook stylesheets turned into
a mess of roff requests that ended up showing up in the output instead
of being processed.
Token.attr is a dict[str, str | int | float], meta has no restriction on
the value type. attrs is ostensibly meant for html attributes, meta for
any information whatsoever.
these work together with render and renderInline to produce an output
from either of the two. rendering manpages will need both: to join
blocks with newlines, and to run some postprocessing and the rendered inlines.
pulling mypy into the build closure is unfortunately not reasonable, the
closure for mypy is rather large and takes a long time to build. if we
have the type checks hooked into CI we'll get most of the benefit though.
this is not yet able to produce manual-combined.xml, but the intention
is to add that support before too long. for now we'll concentrate on
getting the basics working: concatenating a list of chapters into a
manual-combined fragment, which will be rendered via docbook.
previously we did not detect whether lists were supposed to be compact
or not. this will make a difference for manual chapters, so let's stop
not doing that.
headings are not supported in options docs (since it's unclear what that
would be in the final manual, and the docbook stylesheets already have
trouble rendering all docbook constructs correctly). ordered lists
should be supported, but obviously nothing uses them yet.
options do not use comments, but a number of manual chapters do. since
we don't want to enable html just so we can then inspect the html and
figure out whether it's a comment we'll instead add a plugin that
detects comments natively.
we will soon add plugins to this tool to support nixos markdown features
that aren't readily supported with markdown-it plugins. since we will
have to test these plugin we'll need access to the parser, and since
we'll also want to add functions that require postprocessing of a parsed
token stream we also add the necessary hooks now.
with some fiddling and custom validation logic we can avoid needing
multiple instances of this plugin. originally this wasn't done because
it does need a type stack to emit correct docbook, but since we can
easily do that now we may as well.
using environment variables isn't great once multiple input or output
formats get involved (which will happen soon). now is a good time to set
a pattern for future converters.
while we won't have other converters (with other output formats) for a
while yet it still seems like a good idea to generalize *now* so we have
a pattern to follow.
these modules will be extended with more functionality. md will
ultimately contain MD-related code such as parsers and converter base
classes. docbook will contain renderers.
this new package shall eventually contain the rendering code necessary
to produce the entirety of the nixos (not nixpkgs) manual, in all of its
various output formats.
checkInputs used to be added to nativeBuildInputs. Now we have
nativeCheckInputs to do that instead. Doing this treewide change allows
to keep hashes identical to before the introduction of
nativeCheckInputs.
Fixes `error: unable to execute '/nix/store/arrb5j23znf00p1i0kvd9bmb7ddamlxx-nix-output-monitor-2.0.0.2/bin/nix-output-monitor': No such file or directory` with nix run
