they're no longer necessary for us and will almost definitely start to
rot now (like commonmark and asciidoc outputs did previously). most
existing users seem to take the docbook output and run it through pandoc
to generate html, those can easily migrate to use commonmark instead.
other users will hopefully pipe up when they notice that things they rely
on are going away.
optionsUsedDocbook has only been around for one release and only exposed
to allow other places to generate warnings, so that does not deserve
such precautions.
it's been long in the making, and with 23.05 out we can finally disable
docbook option docs and default to markdown instead. this brings a
massive speed boost in manual and manpage builds, so much so that we may
consider enabling user module documentation by default.
we don't remove the docbook support code entirely yet because it's a lot
all over, and probably better removed in multiple separate changes.
epub manuals are holding back the transition away from docbook, and
cursory research does not suggest that they are used very much. it's
still very early in the 23.11 release cycle, so if we're going to find
out just how many people do use the epub manuals it should be now.
this need not be the end of epub manuals. nixos-render-docs could be
extended to also export epubs, but that has not been done yet since it's
going to be some effort with unknown real-world usefulness.
Remove absolute reference to xsltproc. This reference breaks cross-compilation: instead of host version builder tries to execute binary for target and fails.
the conversion to the markdown-based workflow missed that generating the
manual as docbook also generates the option docs with nixos-specific
wrapper elements that require postprocessing before the document is
real, valid docbook. restore this processing to the full manual.
it's not the prettiest thing, done like this, but we only need it for
one release so it doesn't have to be.
this reproduces the docbook-generated html manual exactly enough to
appease the compare workflows while we still support both toolchains.
it's also a lot faster than the docbook toolchain, rendering the entire
html manual in about two seconds on this machine (while docbook needs
about 20).
options processing is pretty slow right now, mostly because the
markdown-it-py parser is pure python (and with performance
pessimizations at that). options parsing *is* embarassingly parallel
though, so we can just fork out all the work to worker processes and
collect the results.
multiprocessing probably has a greater benefit on linux than on darwin
since the worker spawning method darwin uses is less efficient than
fork() on linux. this hasn't been tested on darwin, only on linux, but
if anything darwin will be faster with its preferred method.
this adds support for structural includes to nixos-render-docs.
structural includes provide a way to denote the (sub)structure of the
nixos manual in the markdown source files, very similar to how we used
literal docbook blocks before, and are processed by nixos-render-docs
without involvement of xml tooling. this will ultimately allow us to
emit the nixos manual in other formats as well, e.g. html, without going
through docbook at all.
alternatives to this source layout were also considered:
a parallel structure using e.g. toml files that describe the document
tree and links to each part is possible, but much more complicated to
implement than the solution chosen here and makes it harder to follow
which files have what substructure. it also makes it much harder to
include a substructure in the middle of a file.
much the same goes for command-line arguments to the converter, only
that command-lined arguments are even harder to specify correctly and
cannot be reasonably pulled together from many places without involving
another layer of tooling. cli arguments would also mean that the manual
structure would be fixed in default.nix, which is also not ideal.
- inline copySources into single user
- remove `inherit sources` where it's not necessary
- inline generatedSources. this will go away completely soon so we may
as well.
- inline modulesDoc into manual-combined. this too will go away soon.
- inline sources into manual-combined. this too will go away soon.
once we generate the entire manual-combined.xml with a single
nixos-render-docs invocation we will no longer need any options xml
files. likewise we do not need the test options xml in the manpage
build. splitting manpages-combined from manual-combined also allows
these two to run in parallel, slightly improving build times.
render all manual chapters to docbook from scratch every time the manual
is built. nixos-render-docs is quick enough at this to not worry about
the cost (needing only about a second), and it means we can remove
md-to-db.sh in the next commit.
no changes to the rendered html manual except for replacements and smartquotes.
we'll soon add another docbook converter that does not emit a section as
a collection of chapters, but sections or chapters on their own. this
should clarify naming a bit before there can be any confusion.
mdoc is just too slow to render on groff, and semantic markup doesn't
help us any for generated pages.
this produces a lot of changes to configuration.nix.5, but only few
rendering changes. most of those seem to be place losing a space where
docbook emitted roff code that did not faithfully represent the input
text, though a few places also gained space where docbook dropped them.
notably we also don't need the compatibility code docbook-xsl emitted
because that problem was fixed over a decade ago.
this will handle block quotes, which the docbook stylesheets turned into
a mess of roff requests that ended up showing up in the output instead
of being processed.
since we want to move away from docbook and having docbook manpages
around is going to block further progress we have to translate the
current nixos manpages from docbook it *something* else. mdoc seems the
most appropriate choice since markdown can't represent all the things
manpages can differentiate with even more extensions. if we ever need
html manpages we can render them using troff, like many of the online
manpage viewers, or rewrite them again. since we haven't had html
manpages for any of these in many years that seems unlikely to happen.
unlike most of the recent markdown conversions this comes with a lot of
minor rendering changes, mostly in spacing. docbook-xslt creates manual
pages in a very different dialect than mdoc (which is used here).
this converts meta.doc into an md pointer, not an xml pointer. since we
no longer need xml for manual chapters we can also remove support for
manual chapters from md-to-db.sh
since pandoc converts smart quotes to docbook quote elements and our
nixos-render-docs does not we lose this distinction in the rendered
output. that's probably not that bad, our stylesheet didn't make use of
this anyway (and pre-23.05 versions of the chapters didn't use quote
elements either).
also updates the nixpkgs manual to clarify that option docs support all
extensions (although it doesn't support headings at all, so heading
anchors don't work by extension).
as far as we can tell nixos has only ever had a total of one olink, and
currently has no olinks at all. we can't currently represent olinks in
markdown docs, and if we re-add support for cross-document links they
will take a different form (and not use docbook, which will have to be
phased out before we re-add anything).
the olinkdb is thus unused and takes 10 seconds on our machine to build,
holding up the rest of the manual for no benefit.
following the plan in https://github.com/NixOS/nixpkgs/pull/189318#discussion_r961764451
also adds an activation script to print the warning during activation
instead of during build, otherwise folks using the new CLI that hides
build logs by default might never see the warning.
deprecate literalDocBook by adding a warning (that will not fire yet) to
its uses and other docbook literal strings by adding optional warning
message to mergeJSON.
By writing the unchecked outputs before checking them, they will
be written to a store path, which appears in the log, and which
sticks around even if the build fails. Eventually it is GCed, but
until then, you can open the file.
If you run it in a terminal+editor combination like VSCode, the
failure location is just one Ctrl+click away.
a few things should've used buildPackages/nativeBuildInputs to not not require
the host architecture for building docs. tested by building aarch64-linux docs
on x86_64-linux, and the result looks good.
this partially solves the problem of "missing description" warnings of the
options doc build being lost by nix build, at the cost of failing builds that
previously ran. an option to disable this behaviour is provided.
most modules can be evaluated for their documentation in a very
restricted environment that doesn't include all of nixpkgs. this
evaluation can then be cached and reused for subsequent builds, merging
only documentation that has changed into the cached set. since nixos
ships with a large number of modules of which only a few are used in any
given config this can save evaluation a huge percentage of nixos
options available in any given config.
in tests of this caching, despite having to copy most of nixos/, saves
about 80% of the time needed to build the system manual, or about two
second on the machine used for testing. build time for a full system
config shrank from 9.4s to 7.4s, while turning documentation off
entirely shortened the build to 7.1s.
Motivation is to support other repositories containing nixos
modules that would like to generate options documentation:
- nix-darwin
- private repos
- arion
- ??
Because when I see "config.system.build.manual.manual" after I forgot
what it means I ask "Why do I need that second `.manual` there again?".
Doesn't happen with `config.system.build.manual.manualHTML`.
What annoyed me for a long time was the fact, that in order to break
into a new paragraph, you need to insert </para><para> in the
description attribute of an option.
Now we will automatically create <para/> elements for every block that
is separated by two consecutive newlines.
I first tried to do this within options-to-docbook.xsl, but it turns
out[1] that this isn't directly possible with XSLT 1.0, so I added
another XSLT file that postprocesses the option descriptions that are
now enclosed in <nixos:option-description/> by options-to-docbook.xsl.
The splitting itself is a bit more involved, because we can't simply
split on every \n\n because we'd also split text nodes of elements, for
example:
<screen><![CDATA[
one line
another one
]]></screen>
This would create one <para/> element for "one line" and another for
"another line", which we obviously don't want because <screen/> is used
to display verbatim contents of what a user is seeing on the screen.
So what we do instead is splitting *only* the top-level text nodes
within the outermost <para/> and leave all elements as-is. If there are
more than one <para/> elements at the top-level, we simply don't process
it at all, because the description then already contains </para><para>.
https://www.mhonarc.org/archive/html/xsl-list/2012-09/msg00319.html
Signed-off-by: aszlig <aszlig@nix.build>
Cc: @edolstra, @domenkozar
This makes the command ‘nix-env -qa -f. --arg config '{skipAliases =
true;}'’ work in Nixpkgs.
Misc...
- qtikz: use libsForQt5.callPackage
This ensures we get the right poppler.
- rewrites:
docbook5_xsl -> docbook_xsl_ns
docbook_xml_xslt -> docbook_xsl
diffpdf: fixup
