Commit Graph

92 Commits

Author SHA1 Message Date
pennae
9c0803f84d nixos-render-docs: fix empty inlines
`children` only has to be not-None, not also not-[]. empty table cells
would can produce an empty inline token and cause these checks to fail,
even though the token actually is totally valid.

showed up in
https://github.com/NixOS/nixpkgs/pull/246956#discussion_r1283146120.
2023-08-03 18:36:34 +02:00
pennae
538b3d1b3c nixos-render-docs: add footnote support
this is only used in the stdenv chapter, but footnotes could be useful
in other places as well. since markdown-it has a plugin to parse
footnote syntax we may as well just support them even if they're rare.
2023-07-01 20:27:29 +02:00
pennae
ac7be1f106 nixos-render-docs: add support for tables 2023-07-01 20:27:29 +02:00
pennae
8fb4cf8b7c nixos-render-docs: add support for figures 2023-07-01 20:27:29 +02:00
pennae
e5e738b72a nixos-render-docs: genericize block numbering
examples and figures behave identically regarding numbering and titling,
they just don't share a common number space. make the numbering/titling
function generic over block types now so figures can just use it.
2023-07-01 20:27:29 +02:00
pennae
8c2d14a6b8 nixos-render-docs: add image support
currently only supported for html. docbook could also support images,
but it's on the way out for manual generation anyway so we won't add
image support there. options docs can't use images because they also
target manpages, which leaves no viable users.
2023-07-01 20:27:28 +02:00
pennae
b962ff92ff nixos-render-docs: add support for section tocs
the nixpkgs manual uses section tocs for eg the nixpkgs library function
reference. we will only not emit tocs for subsections at this point, but
maybe we should consider doing so in the future.
2023-07-01 18:21:16 +02:00
pennae
2bf3bb4f6b nixos-render-docs: hide nav headers/footers if not needed
there's not point in rendering these if we're rendering to a single file
for all content.
2023-07-01 17:31:29 +02:00
pennae
effbaf0ab4 nixos-render-docs: maybe omit link target files
docbook always emits intra-file links if only a single file is emitted.
it doesn't seem to do this for intra-file links when multiple files are
emitted, so we don't do that yet either.
2023-07-01 17:31:29 +02:00
pennae
8c33134465 nixos-render-docs: don't double-escape link titles 2023-07-01 17:31:29 +02:00
Pol Dellaiera
246000e996 nixos/manual: fix html <p> tag closure 2023-06-28 13:50:48 +02:00
pennae
03ca5a47c1 nixos-render-docs: add missing head tag to html output 2023-06-26 21:13:11 +02:00
pennae
c8b4e5d557
Merge pull request #237557 from pennae/dedocbookify-nixos
nixos/doc: dedocbookify
2023-06-19 14:05:03 +02:00
Victor Engmark
7b396875bb nixos-render-docs: De-lint using ruff --fix
This automated de-linting has applied a few different refactors:

- Remove unused imports and variables
- Change f-strings with no variables to regular strings
- Remove trailing semicolon
2023-06-14 14:39:00 +12:00
pennae
34eeac5544 nixos-render-docs: default to markdown for options
docbook is now gone and we can flip the defaults. we won't keep the
command line args around (unlike the make-options-docs argument) because
nixos-render-docs should not be considered an exposed API.
2023-06-13 16:56:31 +02:00
pennae
5f35ae16ec nixos-render-docs: remove literalDocBook support
with literalDocBook itself gone we can also remove the support code in
nixos-render-docs.
2023-06-13 16:56:30 +02:00
pennae
407f6196a2 nixos-render-docs: add examples support
the nixos manual contains enough examples to support them as a proper
toc entity with specialized rendering, and if in the future the nixpkgs
wants to use nixos-render-docs we will definitely have to support them.
this also allows us to restore some examples that were lost in previous
translation steps because there were too few to add renderer support
back then.
2023-05-03 19:58:21 +02:00
Sandro Jäckel
d1a388ae8e
nixos-render-docs: use packageOverrides to construct python packages 2023-03-14 00:12:34 +01:00
pennae
36f04733dd nixos-render-docs: add manual html converter
this converter is currently supposed to be able to reproduce the
docbook-generated html DOMs exactly, though not necessarily the
html *files*. it mirrors many docbook behaviours that seem rather odd,
such as top-level sections in chapters using the same heading depth as
understood by html as their parent chapters do. over time we can
hopefully remove all special casing needed to reproduce docbook
rendering, but for now at least it doesn't hurt *too* much.
2023-02-21 18:26:41 +01:00
pennae
feaa97e5dc nixos-render-docs: render directly from file to file
this will be necessary for html since there we have to do chunking into
multiple files ourselves. writing one file from the caller of the
converter and all others from within the converter is unnecessarily
spread out, and returning a dict of file names and their contents is not
quite as meaningful for docbook (which has only one file to begin with).
2023-02-21 18:26:41 +01:00
pennae
d520d55dee nixos-render-docs: add options html renderer
it's not hooked up to anything yet, but that will come soon. there's a
bit of docbook compat here that must be interoperable with the actual
docbook exporter, but luckily it's not all that much.
2023-02-21 18:26:41 +01:00
pennae
82e62614e9 nixos-render-docs: add html renderer
the basic html renderer. it doesn't have all the docbook compatibility
codes embedded into it, but there is a good amount. this renderer is
unaware of manual structure and does not traverse structural include
tokens (if it finds any it'll just fail), that task falls to derived
classes. once we have more uses for structural includes than just the
manual we may revisit this decision.
2023-02-21 18:26:41 +01:00
pennae
7a74ce51a1 nixos-render-docs: add toc generator
the docbook toolchain uses docbook-xsl to generate its TOC, our html
renderer will have to do this on its own. this generator uses a very
straight-forward algorithm of only inspecting headings, but anything
else could be inspected as well. (examples come to mind, but those do
not have titles and would thus make for bad toc entries)

we also use path information (that will be taken from include block args
in the html renderer) to produce navigation information. the algorithm
we use mirrors what docbook does, linking to the next/previous files in
depth-first toc order.

toc entries are linked to the tokens they refer to for easy use later.
2023-02-21 18:26:41 +01:00
pennae
23dc31a975 nixos-render-docs: allow for options in include blocks
while docbook relies on external chunk-toc info to do chunking of the
rendered manual we have nothing of the sort for html. there it seems
easiest to add annotations to blocks to create new chunks. such
annotations could be extended to docbook to create the chunk-toc instead
of passing it in externally, but with docbook on the way out that seems
like a waste of effort.
2023-02-21 18:26:41 +01:00
pennae
768794d6c1 nixos-render-docs: check book structure
text content in the toplevel file of a book will not render properly.
the first proper element will be a preface, part, or chapter anyway, and
those require includes to produce.

parts do not currently allow headings in the part file itself, but
that's mainly a renderer limitation. we can add support for headings in
part intros when we need them

in all other cases includes must be followed by either another include,
a heading, or end of file. text content could not be properly linked to
from a TOC without a preceding heading.
2023-02-21 18:26:40 +01:00
pennae
163b667352 nixos-render-docs: require headings to have ids
without this we cannot build a TOC to arbitrary depth without generating
ids for headings, but generated ids are fragile and liable to either
break or point to different things if the manual changes shape. we
already have the convention that all headings should have an id, this
formalizes it.
2023-02-21 18:26:40 +01:00
pennae
ba20114460 nixos-render-docs: check heading continuity
while not technically necessary for correct rendering of *contents* we
do need to disallow heading levels being skipped to build a correct
TOC. treating headings that have skipped a number of levels to actually
be headings that many levels up only gets confusing, and inserting
artifical intermediate headings suffers from problems, such as which ids
to use and what to call them.
2023-02-21 18:26:40 +01:00
pennae
7b0824c003 nixos-render-docs: check heading presence during parsing
check that all required headings are present during parsing, not during
rendering. building a correct TOC will need this since every TOC entry
needs a heading to set its title, and every included substructure needs
a title.

also improve the error message on repeated title headings slightly,
giving the end line turns out to not be very useful.
2023-02-21 18:26:40 +01:00
pennae
a7c25bb01f nixos-render-docs: add Freezable class
for most of our data classes we can use dataclasses.dataclass with
frozen=True or even plain named tuples. the TOC structure we'll need to
generate proper navigation links is most easily represented and used as
a cyclic structure though, and for that we can use neither. if we want
to make the TOC structures immutable (which seems like a good idea)
we'll need a hack of *some* kind, and this hack seems like the least intrusive.
2023-02-21 18:26:40 +01:00
pennae
2ab8e742a5 nixos-render-docs: move recursive manual parsing to base class
the html renderer will need all of these functions as well. some
extensions will be needed, but we'll add those as they become necessary.
2023-02-21 18:26:40 +01:00
pennae
5b8be28e66 nixos-render-docs: don't render options during manual parsing
we should really be rendering options at *rendering* time, not at parse
time. currently this is just an academic exercise, but the html renderer
will have to inspect the options.json data after the entire document has
been parsed, but before anything gets rendered.
2023-02-21 18:26:40 +01:00
pennae
068916ae8f nixos-render-docs: keep revision in renderer, not converter
ultimately it's the renderer that needs it, for the options rendering
that will be simplified in a bit.
2023-02-21 18:26:39 +01:00
pennae
6f253fc70b nixos-render-docs: drop options, env parameters
these weren't used for anything. options never was (and does not contain
any information for the renderer that we *want* to honor), and env is
not used because typed renderer state is much more useful for all our cases.
2023-02-21 18:26:39 +01:00
pennae
0236dcb59f nixos-render-docs: don't use markdown-it RendererProtocol
our renderers carry significantly more state than markdown-it wants to
easily cater for, and the html renderer will need even more state still.
relying on the markdown-it-provided rendering functions has already
proven to be a nuisance, and since parsing and rendering are split well
enough we can just replace the rendering part with our own stuff outright.

this also frees us from the tyranny of having to set instance variables
before calling super().__init__ just to make sure that the renderer
creation callback has access to everything it needs.
2023-02-21 18:26:39 +01:00
pennae
417dd2ad16 nixos-render-docs: add options asciidoc converter
same reasoning as for the earlier commonmark converter.
2023-02-21 18:19:00 +01:00
pennae
4d3aef762f nixos-render-docs: add options commonmark converter
the old method of pasting parts of options.json into a markdown document
and hoping for the best no longer works now that options.json contains
more than just docbook. given the infrastructure we have now we can
actually render options.md properly, so we may as well do that.
2023-02-21 18:19:00 +01:00
pennae
6c182075bb nixos-render-docs: forbid attrspans and examples in options
inline anchors are not allowed in option docs per the manual, and the
sole class we current have (.keycap) is never used anyway. disallow them
for now to avoid future surprises.

the same goes for examples, which aren't even documented in the manual yet.
2023-02-21 18:19:00 +01:00
pennae
82d066ffe3 nixos-render-docs: refactor option docs restrictions
move the restrictions we care about into a mixin class. a few more
restrictions will appear soon and a few new converters as well, the
renderers of which need not have these restrictions already baked in by
accident (like the manpage renderer does right now).
2023-02-21 18:19:00 +01:00
pennae
45619b3c4a nixos-render-docs: extend md_make_code
add the ability to set the info string for a newly created fenced code
block, and a flag to always emit a fenced block. the commonmark
converter will need this to faithfully recreate fenced and indented code
blocks.
2023-02-21 18:19:00 +01:00
pennae
895d9e69dd nixos-render-docs: extract md code block factory
the commonmark exporter can make good use once it appears.
2023-02-21 18:19:00 +01:00
pennae
00bffb84da nixos-render-docs: drop frozendict
with mypy type checking and Mapping types this is a lot less useful than
anticipated. let's drop it for simplicity and having fewer dependencies.

frozendict 2.3.5 also broke the mypy checks.
2023-02-21 18:19:00 +01:00
pennae
81cf173256 nixos-render-docs: use multiprocessing for options
options processing is pretty slow right now, mostly because the
markdown-it-py parser is pure python (and with performance
pessimizations at that). options parsing *is* embarassingly parallel
though, so we can just fork out all the work to worker processes and
collect the results.

multiprocessing probably has a greater benefit on linux than on darwin
since the worker spawning method darwin uses is less efficient than
fork() on linux. this hasn't been tested on darwin, only on linux, but
if anything darwin will be faster with its preferred method.
2023-02-18 13:17:57 +01:00
pennae
1229e735ac nixos-render-docs: add structural includes, use for manual
this adds support for structural includes to nixos-render-docs.
structural includes provide a way to denote the (sub)structure of the
nixos manual in the markdown source files, very similar to how we used
literal docbook blocks before, and are processed by nixos-render-docs
without involvement of xml tooling. this will ultimately allow us to
emit the nixos manual in other formats as well, e.g. html, without going
through docbook at all.

alternatives to this source layout were also considered:

a parallel structure using e.g. toml files that describe the document
tree and links to each part is possible, but much more complicated to
implement than the solution chosen here and makes it harder to follow
which files have what substructure. it also makes it much harder to
include a substructure in the middle of a file.

much the same goes for command-line arguments to the converter, only
that command-lined arguments are even harder to specify correctly and
cannot be reasonably pulled together from many places without involving
another layer of tooling. cli arguments would also mean that the manual
structure would be fixed in default.nix, which is also not ideal.
2023-02-12 13:02:42 +01:00
pennae
d30da4d9cd nixos-render-docs: add support for <part>
<part> is different from all other blocks we care about in that it
requires textual content to be wrapped in <partintro>. add support for
this to the generic docbook renderer, which will just assume that a part
is the whole document start to finish. we do make provision for the
manual renderer to close a partintro tag early though.
2023-02-12 11:55:09 +01:00
pennae
ad2b150af7 nixos-render-docs: use Mapping for options converter
this way we can pass in frozendicts from other converters.
2023-02-12 11:55:09 +01:00
pennae
d004105003 nixos-render-docs: print exception trees by __cause__
__context__ is always set to the prior exception, even when not using
the raise from form. __cause__ is only set during raise from. use
__cause__ so we can override a leaf exception (eg KeyError to something
more meaningful).
2023-02-12 11:55:09 +01:00
pennae
652a283e51 nixos-render-docs: render manual chapters during manual build
render all manual chapters to docbook from scratch every time the manual
is built. nixos-render-docs is quick enough at this to not worry about
the cost (needing only about a second), and it means we can remove
md-to-db.sh in the next commit.

no changes to the rendered html manual except for replacements and smartquotes.
2023-02-10 06:40:02 +01:00
pennae
67917ac102 nixos-render-docs: rename manual docbook converter to docbook-section
we'll soon add another docbook converter that does not emit a section as
a collection of chapters, but sections or chapters on their own. this
should clarify naming a bit before there can be any confusion.
2023-02-10 06:40:02 +01:00
pennae
b59b0230ae nixos-render-docs: add example blocks
this is currently only supported by the docbook exporter, and even the
docbook exporter doesn't do much with them. we mirror the conversion
pandoc did for consistency with the previous manual chapter conversion,
which is to add just an anchor with the given id. future exporters that
go directly to html might want to do more.
2023-02-10 06:40:02 +01:00
pennae
bb6526e0de nixos-render-docs: add generic attributed-block parsing
this is a subset of pandoc's fenced divs. currently we only use this for
admonitions (which get a new name to differentiate them from other kinds
of blocks), but more users will appear soon.
2023-02-10 06:40:02 +01:00