nordic-dev.net/rust
nordic-dev.net/rust
2
0
Fork 0
mirror of https://github.com/rust-lang/rust.git synced 2024-11-22 14:55:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

Auto merge of #110800 - GuillaumeGomez:custom_code_classes_in_docs, r=t-rustdoc

Browse Source 
Accept additional user-defined syntax classes in fenced code blocks

Part of #79483.

This is a re-opening of https://github.com/rust-lang/rust/pull/79454 after a big update/cleanup. I also converted the syntax to pandoc as suggested by `@notriddle:` the idea is to be as compatible as possible with the existing instead of having our own syntax.

## Motivation

From the original issue: https://github.com/rust-lang/rust/issues/78917

> The technique used by `inline-c-rs` can be ported to other languages. It's just super fun to see C code inside Rust documentation that is also tested by `cargo doc`. I'm sure this technique can be used by other languages in the future.

Having custom CSS classes for syntax highlighting will allow tools like `highlight.js` to be used in order to provide highlighting for languages other than Rust while not increasing technical burden on rustdoc.

## What is the feature about?

In short, this PR changes two things, both related to codeblocks in doc comments in Rust documentation:

 * Allow to disable generation of `language-*` CSS classes with the `custom` attribute.
 * Add your own CSS classes to a code block so that you can use other tools to highlight them.

#### The `custom` attribute

Let's start with the new `custom` attribute: it will disable the generation of the `language-*` CSS class on the generated HTML code block. For example:

```rust
/// ```custom,c
/// int main(void) {
///     return 0;
/// }
/// ```
```

The generated HTML code block will not have `class="language-c"` because the `custom` attribute has been set. The `custom` attribute becomes especially useful with the other thing added by this feature: adding your own CSS classes.

#### Adding your own CSS classes

The second part of this feature is to allow users to add CSS classes themselves so that they can then add a JS library which will do it (like `highlight.js` or `prism.js`), allowing to support highlighting for other languages than Rust without increasing burden on rustdoc. To disable the automatic `language-*` CSS class generation, you need to use the `custom` attribute as well.

This allow users to write the following:

```rust
/// Some code block with `{class=language-c}` as the language string.
///
/// ```custom,{class=language-c}
/// int main(void) {
///     return 0;
/// }
/// ```
fn main() {}
```

This will notably produce the following HTML:

```html
<pre class="language-c">
int main(void) {
    return 0;
}</pre>
```

Instead of:

```html
<pre class="rust rust-example-rendered">
<span class="ident">int</span> <span class="ident">main</span>(<span class="ident">void</span>) {
    <span class="kw">return</span> <span class="number">0</span>;
}
</pre>
```

To be noted, we could have written `{.language-c}` to achieve the same result. `.` and `class=` have the same effect.

One last syntax point: content between parens (`(like this)`) is now considered as comment and is not taken into account at all.

In addition to this, I added an `unknown` field into `LangString` (the parsed code block "attribute") because of cases like this:

```rust
/// ```custom,class:language-c
/// main;
/// ```
pub fn foo() {}
```

Without this `unknown` field, it would generate in the DOM: `<pre class="language-class:language-c language-c">`, which is quite bad. So instead, it now stores all unknown tags into the `unknown` field and use the first one as "language". So in this case, since there is no unknown tag, it'll simply generate `<pre class="language-c">`. I added tests to cover this.

Finally, I added a parser for the codeblock attributes to make it much easier to maintain. It'll be pretty easy to extend.

As to why this syntax for adding attributes was picked: it's [Pandoc's syntax](https://pandoc.org/MANUAL.html#extension-fenced_code_attributes). Even if it seems clunkier in some cases, it's extensible, and most third-party Markdown renderers are smart enough to ignore Pandoc's brace-delimited attributes (from [this comment](https://github.com/rust-lang/rust/pull/110800#issuecomment-1522044456)).

## Raised concerns

#### It's not obvious when the `language-*` attribute generation will be added or not.

It is added by default. If you want to disable it, you will need to use the `custom` attribute.

#### Why not using HTML in markdown directly then?

Code examples in most languages are likely to contain `<`, `>`, `&` and `"` characters. These characters [require escaping](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/pre) when written inside the `<pre>` element. Using the \`\`\` code blocks allows rustdoc to take care of escaping, which means doc authors can paste code samples directly without manually converting them to HTML.

cc `@poliorcetics`
r? `@notriddle`
This commit is contained in:
bors 2023-09-16 13:10:11 +00:00
commit 41bafc4ff3
18 changed files with 971 additions and 80 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
compiler/rustc_builtin_macros/src/deriving/generic/mod.rs
View File

@ -88,7 +88,7 @@
//!
//! When generating the `expr` for the `A` impl, the `SubstructureFields` is
//!
//! ```{.text}
//! ```text
//! Struct(vec![FieldInfo {
//! span: <span of x>
//! name: Some(<ident of x>),
@ -99,7 +99,7 @@
//!
//! For the `B` impl, called with `B(a)` and `B(b)`,
//!
//! ```{.text}
//! ```text
//! Struct(vec![FieldInfo {
//! span: <span of `i32`>,
//! name: None,
@ -113,7 +113,7 @@
//! When generating the `expr` for a call with `self == C0(a)` and `other
//! == C0(b)`, the SubstructureFields is
//!
//! ```{.text}
//! ```text
//! EnumMatching(0, <ast::Variant for C0>,
//! vec![FieldInfo {
//! span: <span of i32>
@ -125,7 +125,7 @@
//!
//! For `C1 {x}` and `C1 {x}`,
//!
//! ```{.text}
//! ```text
//! EnumMatching(1, <ast::Variant for C1>,
//! vec![FieldInfo {
//! span: <span of x>
@ -137,7 +137,7 @@
//!
//! For the tags,
//!
//! ```{.text}
//! ```text
//! EnumTag(
//! &[<ident of self tag>, <ident of other tag>], <expr to combine with>)
//! ```
@ -149,7 +149,7 @@
//!
//! A static method on the types above would result in,
//!
//! ```{.text}
//! ```text
//! StaticStruct(<ast::VariantData of A>, Named(vec![(<ident of x>, <span of x>)]))
//!
//! StaticStruct(<ast::VariantData of B>, Unnamed(vec![<span of x>]))

2
compiler/rustc_feature/src/active.rs
View File

@ -401,6 +401,8 @@ declare_features! (
/// Allows function attribute `#[coverage(on/off)]`, to control coverage
/// instrumentation of that function.
(active, coverage_attribute, "CURRENT_RUSTC_VERSION", Some(84605), None),
/// Allows users to provide classes for fenced code block using `class:classname`.
(active, custom_code_classes_in_docs, "CURRENT_RUSTC_VERSION", Some(79483), None),
/// Allows non-builtin attributes in inner attribute position.
(active, custom_inner_attributes, "1.30.0", Some(54726), None),
/// Allows custom test frameworks with `#![test_runner]` and `#[test_case]`.

4
compiler/rustc_middle/src/ty/typeck_results.rs
View File

@ -165,7 +165,7 @@ pub struct TypeckResults<'tcx> {
/// reading places that are mentioned in a closure (because of _ patterns). However,
/// to ensure the places are initialized, we introduce fake reads.
/// Consider these two examples:
/// ``` (discriminant matching with only wildcard arm)
/// ```ignore (discriminant matching with only wildcard arm)
/// let x: u8;
/// let c = || match x { _ => () };
/// ```
@ -173,7 +173,7 @@ pub struct TypeckResults<'tcx> {
/// want to capture it. However, we do still want an error here, because `x` should have
/// to be initialized at the point where c is created. Therefore, we add a "fake read"
/// instead.
/// ``` (destructured assignments)
/// ```ignore (destructured assignments)
/// let c = || {
/// let (t1, t2) = t;
/// }

1
compiler/rustc_span/src/symbol.rs
View File

@ -592,6 +592,7 @@ symbols! {
cttz,
cttz_nonzero,
custom_attribute,
custom_code_classes_in_docs,
custom_derive,
custom_inner_attributes,
custom_mir,

44
src/doc/rustdoc/src/unstable-features.md
View File

@ -625,3 +625,47 @@ and check the values of `feature`: `foo` and `bar`.
This flag enables the generation of links in the source code pages which allow the reader
to jump to a type definition.
### Custom CSS classes for code blocks
```rust
#![feature(custom_code_classes_in_docs)]
/// ```custom,{class=language-c}
/// int main(void) { return 0; }
/// ```
pub struct Bar;
```
The text `int main(void) { return 0; }` is rendered without highlighting in a code block
with the class `language-c`. This can be used to highlight other languages through JavaScript
libraries for example.
Without the `custom` attribute, it would be generated as a Rust code example with an additional
`language-C` CSS class. Therefore, if you specifically don't want it to be a Rust code example,
don't forget to add the `custom` attribute.
To be noted that you can replace `class=` with `.` to achieve the same result:
```rust
#![feature(custom_code_classes_in_docs)]
/// ```custom,{.language-c}
/// int main(void) { return 0; }
/// ```
pub struct Bar;
```
To be noted, `rust` and `.rust`/`class=rust` have different effects: `rust` indicates that this is
a Rust code block whereas the two others add a "rust" CSS class on the code block.
You can also use double quotes:
```rust
#![feature(custom_code_classes_in_docs)]
/// ```"not rust" {."hello everyone"}
/// int main(void) { return 0; }
/// ```
pub struct Bar;
```

25
src/librustdoc/html/highlight.rs
View File

@ -52,8 +52,9 @@ pub(crate) fn render_example_with_highlighting(
out: &mut Buffer,
tooltip: Tooltip,
playground_button: Option<&str>,
extra_classes: &[String],
) {
write_header(out, "rust-example-rendered", None, tooltip);
write_header(out, "rust-example-rendered", None, tooltip, extra_classes);
write_code(out, src, None, None);
write_footer(out, playground_button);
}
@ -65,7 +66,13 @@ pub(crate) fn render_item_decl_with_highlighting(src: &str, out: &mut Buffer) {
write!(out, "</pre>");
}
fn write_header(out: &mut Buffer, class: &str, extra_content: Option<Buffer>, tooltip: Tooltip) {
fn write_header(
out: &mut Buffer,
class: &str,
extra_content: Option<Buffer>,
tooltip: Tooltip,
extra_classes: &[String],
) {
write!(
out,
"<div class=\"example-wrap{}\">",
@ -100,9 +107,19 @@ fn write_header(out: &mut Buffer, class: &str, extra_content: Option<Buffer>, to
out.push_buffer(extra);
}
if class.is_empty() {
write!(out, "<pre class=\"rust\">");
write!(
out,
"<pre class=\"rust{}{}\">",
if extra_classes.is_empty() { "" } else { " " },
extra_classes.join(" "),
);
} else {
write!(out, "<pre class=\"rust {class}\">");
write!(
out,
"<pre class=\"rust {class}{}{}\">",
if extra_classes.is_empty() { "" } else { " " },
extra_classes.join(" "),
);
}
write!(out, "<code>");
}

401
src/librustdoc/html/markdown.rs
View File

@ -26,6 +26,7 @@
//! ```
use rustc_data_structures::fx::FxHashMap;
use rustc_errors::{DiagnosticMessage, SubdiagnosticMessage};
use rustc_hir::def_id::DefId;
use rustc_middle::ty::TyCtxt;
pub(crate) use rustc_resolve::rustdoc::main_body_opts;
@ -37,8 +38,9 @@ use once_cell::sync::Lazy;
use std::borrow::Cow;
use std::collections::VecDeque;
use std::fmt::Write;
use std::iter::Peekable;
use std::ops::{ControlFlow, Range};
use std::str;
use std::str::{self, CharIndices};
use crate::clean::RenderedLink;
use crate::doctest;
@ -243,11 +245,21 @@ impl<'a, I: Iterator<Item = Event<'a>>> Iterator for CodeBlocks<'_, 'a, I> {
let parse_result =
LangString::parse_without_check(lang, self.check_error_codes, false);
if !parse_result.rust {
let added_classes = parse_result.added_classes;
let lang_string = if let Some(lang) = parse_result.unknown.first() {
format!("language-{}", lang)
} else {
String::new()
};
let whitespace = if added_classes.is_empty() { "" } else { " " };
return Some(Event::Html(
format!(
"<div class=\"example-wrap\">\
<pre class=\"language-{lang}\"><code>{text}</code></pre>\
<pre class=\"{lang_string}{whitespace}{added_classes}\">\
<code>{text}</code>\
</pre>\
</div>",
added_classes = added_classes.join(" "),
text = Escape(&original_text),
)
.into(),
@ -258,6 +270,7 @@ impl<'a, I: Iterator<Item = Event<'a>>> Iterator for CodeBlocks<'_, 'a, I> {
CodeBlockKind::Indented => Default::default(),
};
let added_classes = parse_result.added_classes;
let lines = original_text.lines().filter_map(|l| map_line(l).for_html());
let text = lines.intersperse("\n".into()).collect::<String>();
@ -315,6 +328,7 @@ impl<'a, I: Iterator<Item = Event<'a>>> Iterator for CodeBlocks<'_, 'a, I> {
&mut s,
tooltip,
playground_button.as_deref(),
&added_classes,
);
Some(Event::Html(s.into_inner().into()))
}
@ -711,6 +725,17 @@ pub(crate) fn find_testable_code<T: doctest::Tester>(
error_codes: ErrorCodes,
enable_per_target_ignores: bool,
extra_info: Option<&ExtraInfo<'_>>,
) {
find_codes(doc, tests, error_codes, enable_per_target_ignores, extra_info, false)
}
pub(crate) fn find_codes<T: doctest::Tester>(
doc: &str,
tests: &mut T,
error_codes: ErrorCodes,
enable_per_target_ignores: bool,
extra_info: Option<&ExtraInfo<'_>>,
include_non_rust: bool,
) {
let mut parser = Parser::new(doc).into_offset_iter();
let mut prev_offset = 0;
@ -734,7 +759,7 @@ pub(crate) fn find_testable_code<T: doctest::Tester>(
}
CodeBlockKind::Indented => Default::default(),
};
if !block_info.rust {
if !include_non_rust && !block_info.rust {
continue;
}
@ -784,7 +809,23 @@ impl<'tcx> ExtraInfo<'tcx> {
ExtraInfo { def_id, sp, tcx }
}
fn error_invalid_codeblock_attr(&self, msg: String, help: &'static str) {
fn error_invalid_codeblock_attr(&self, msg: impl Into<DiagnosticMessage>) {
if let Some(def_id) = self.def_id.as_local() {
self.tcx.struct_span_lint_hir(
crate::lint::INVALID_CODEBLOCK_ATTRIBUTES,
self.tcx.hir().local_def_id_to_hir_id(def_id),
self.sp,
msg,
|l| l,
);
}
}
fn error_invalid_codeblock_attr_with_help(
&self,
msg: impl Into<DiagnosticMessage>,
help: impl Into<SubdiagnosticMessage>,
) {
if let Some(def_id) = self.def_id.as_local() {
self.tcx.struct_span_lint_hir(
crate::lint::INVALID_CODEBLOCK_ATTRIBUTES,
@ -808,6 +849,8 @@ pub(crate) struct LangString {
pub(crate) compile_fail: bool,
pub(crate) error_codes: Vec<String>,
pub(crate) edition: Option<Edition>,
pub(crate) added_classes: Vec<String>,
pub(crate) unknown: Vec<String>,
}
#[derive(Eq, PartialEq, Clone, Debug)]
@ -817,6 +860,276 @@ pub(crate) enum Ignore {
Some(Vec<String>),
}
/// This is the parser for fenced codeblocks attributes. It implements the following eBNF:
///
/// ```eBNF
/// lang-string = *(token-list / delimited-attribute-list / comment)
///
/// bareword = CHAR *(CHAR)
/// quoted-string = QUOTE *(NONQUOTE) QUOTE
/// token = bareword / quoted-string
/// sep = COMMA/WS *(COMMA/WS)
/// attribute = (DOT token)/(token EQUAL token)
/// attribute-list = [sep] attribute *(sep attribute) [sep]
/// delimited-attribute-list = OPEN-CURLY-BRACKET attribute-list CLOSE-CURLY-BRACKET
/// token-list = [sep] token *(sep token) [sep]
/// comment = OPEN_PAREN *(all characters) CLOSE_PAREN
///
/// OPEN_PAREN = "("
/// CLOSE_PARENT = ")"
/// OPEN-CURLY-BRACKET = "{"
/// CLOSE-CURLY-BRACKET = "}"
/// CHAR = ALPHA / DIGIT / "_" / "-" / ":"
/// QUOTE = %x22
/// NONQUOTE = %x09 / %x20 / %x21 / %x23-7E ; TAB / SPACE / all printable characters except `"`
/// COMMA = ","
/// DOT = "."
/// EQUAL = "="
///
/// ALPHA = %x41-5A / %x61-7A ; A-Z / a-z
/// DIGIT = %x30-39
/// WS = %x09 / " "
/// ```
pub(crate) struct TagIterator<'a, 'tcx> {
inner: Peekable<CharIndices<'a>>,
data: &'a str,
is_in_attribute_block: bool,
extra: Option<&'a ExtraInfo<'tcx>>,
}
#[derive(Clone, Debug, Eq, PartialEq)]
pub(crate) enum LangStringToken<'a> {
LangToken(&'a str),
ClassAttribute(&'a str),
KeyValueAttribute(&'a str, &'a str),
}
fn is_bareword_char(c: char) -> bool {
c == '_' || c == '-' || c == ':' || c.is_ascii_alphabetic() || c.is_ascii_digit()
}
fn is_separator(c: char) -> bool {
c == ' ' || c == ',' || c == '\t'
}
struct Indices {
start: usize,
end: usize,
}
impl<'a, 'tcx> TagIterator<'a, 'tcx> {
pub(crate) fn new(data: &'a str, extra: Option<&'a ExtraInfo<'tcx>>) -> Self {
Self { inner: data.char_indices().peekable(), data, is_in_attribute_block: false, extra }
}
fn emit_error(&self, err: impl Into<DiagnosticMessage>) {
if let Some(extra) = self.extra {
extra.error_invalid_codeblock_attr(err);
}
}
fn skip_separators(&mut self) -> Option<usize> {
while let Some((pos, c)) = self.inner.peek() {
if !is_separator(*c) {
return Some(*pos);
}
self.inner.next();
}
None
}
fn parse_string(&mut self, start: usize) -> Option<Indices> {
while let Some((pos, c)) = self.inner.next() {
if c == '"' {
return Some(Indices { start: start + 1, end: pos });
}
}
self.emit_error("unclosed quote string `\"`");
None
}
fn parse_class(&mut self, start: usize) -> Option<LangStringToken<'a>> {
while let Some((pos, c)) = self.inner.peek().copied() {
if is_bareword_char(c) {
self.inner.next();
} else {
let class = &self.data[start + 1..pos];
if class.is_empty() {
self.emit_error(format!("unexpected `{c}` character after `.`"));
return None;
} else if self.check_after_token() {
return Some(LangStringToken::ClassAttribute(class));
} else {
return None;
}
}
}
let class = &self.data[start + 1..];
if class.is_empty() {
self.emit_error("missing character after `.`");
None
} else if self.check_after_token() {
Some(LangStringToken::ClassAttribute(class))
} else {
None
}
}
fn parse_token(&mut self, start: usize) -> Option<Indices> {
while let Some((pos, c)) = self.inner.peek() {
if !is_bareword_char(*c) {
return Some(Indices { start, end: *pos });
}
self.inner.next();
}
self.emit_error("unexpected end");
None
}
fn parse_key_value(&mut self, c: char, start: usize) -> Option<LangStringToken<'a>> {
let key_indices =
if c == '"' { self.parse_string(start)? } else { self.parse_token(start)? };
if key_indices.start == key_indices.end {
self.emit_error("unexpected empty string as key");
return None;
}
if let Some((_, c)) = self.inner.next() {
if c != '=' {
self.emit_error(format!("expected `=`, found `{}`", c));
return None;
}
} else {
self.emit_error("unexpected end");
return None;
}
let value_indices = match self.inner.next() {
Some((pos, '"')) => self.parse_string(pos)?,
Some((pos, c)) if is_bareword_char(c) => self.parse_token(pos)?,
Some((_, c)) => {
self.emit_error(format!("unexpected `{c}` character after `=`"));
return None;
}
None => {
self.emit_error("expected value after `=`");
return None;
}
};
if value_indices.start == value_indices.end {
self.emit_error("unexpected empty string as value");
None
} else if self.check_after_token() {
Some(LangStringToken::KeyValueAttribute(
&self.data[key_indices.start..key_indices.end],
&self.data[value_indices.start..value_indices.end],
))
} else {
None
}
}
/// Returns `false` if an error was emitted.
fn check_after_token(&mut self) -> bool {
if let Some((_, c)) = self.inner.peek().copied() {
if c == '}' || is_separator(c) || c == '(' {
true
} else {
self.emit_error(format!("unexpected `{c}` character"));
false
}
} else {
// The error will be caught on the next iteration.
true
}
}
fn parse_in_attribute_block(&mut self) -> Option<LangStringToken<'a>> {
while let Some((pos, c)) = self.inner.next() {
if c == '}' {
self.is_in_attribute_block = false;
return self.next();
} else if c == '.' {
return self.parse_class(pos);
} else if c == '"' || is_bareword_char(c) {
return self.parse_key_value(c, pos);
} else {
self.emit_error(format!("unexpected character `{c}`"));
return None;
}
}
self.emit_error("unclosed attribute block (`{}`): missing `}` at the end");
None
}
/// Returns `false` if an error was emitted.
fn skip_paren_block(&mut self) -> bool {
while let Some((_, c)) = self.inner.next() {
if c == ')' {
return true;
}
}
self.emit_error("unclosed comment: missing `)` at the end");
false
}
fn parse_outside_attribute_block(&mut self, start: usize) -> Option<LangStringToken<'a>> {
while let Some((pos, c)) = self.inner.next() {
if c == '"' {
if pos != start {
self.emit_error("expected ` `, `{` or `,` found `\"`");
return None;
}
let indices = self.parse_string(pos)?;
if let Some((_, c)) = self.inner.peek().copied() && c != '{' && !is_separator(c) && c != '(' {
self.emit_error(format!("expected ` `, `{{` or `,` after `\"`, found `{c}`"));
return None;
}
return Some(LangStringToken::LangToken(&self.data[indices.start..indices.end]));
} else if c == '{' {
self.is_in_attribute_block = true;
return self.next();
} else if is_bareword_char(c) {
continue;
} else if is_separator(c) {
if pos != start {
return Some(LangStringToken::LangToken(&self.data[start..pos]));
}
return self.next();
} else if c == '(' {
if !self.skip_paren_block() {
return None;
}
if pos != start {
return Some(LangStringToken::LangToken(&self.data[start..pos]));
}
return self.next();
} else {
self.emit_error(format!("unexpected character `{c}`"));
return None;
}
}
let token = &self.data[start..];
if token.is_empty() { None } else { Some(LangStringToken::LangToken(&self.data[start..])) }
}
}
impl<'a, 'tcx> Iterator for TagIterator<'a, 'tcx> {
type Item = LangStringToken<'a>;
fn next(&mut self) -> Option<Self::Item> {
let Some(start) = self.skip_separators() else {
if self.is_in_attribute_block {
self.emit_error("unclosed attribute block (`{}`): missing `}` at the end");
}
return None;
};
if self.is_in_attribute_block {
self.parse_in_attribute_block()
} else {
self.parse_outside_attribute_block(start)
}
}
}
impl Default for LangString {
fn default() -> Self {
Self {
@ -829,6 +1142,8 @@ impl Default for LangString {
compile_fail: false,
error_codes: Vec::new(),
edition: None,
added_classes: Vec::new(),
unknown: Vec::new(),
}
}
}
@ -838,86 +1153,67 @@ impl LangString {
string: &str,
allow_error_code_check: ErrorCodes,
enable_per_target_ignores: bool,
) -> LangString {
) -> Self {
Self::parse(string, allow_error_code_check, enable_per_target_ignores, None)
}
fn tokens(string: &str) -> impl Iterator<Item = &str> {
// Pandoc, which Rust once used for generating documentation,
// expects lang strings to be surrounded by `{}` and for each token
// to be proceeded by a `.`. Since some of these lang strings are still
// loose in the wild, we strip a pair of surrounding `{}` from the lang
// string and a leading `.` from each token.
let string = string.trim();
let first = string.chars().next();
let last = string.chars().last();
let string = if first == Some('{') && last == Some('}') {
&string[1..string.len() - 1]
} else {
string
};
string
.split(|c| c == ',' || c == ' ' || c == '\t')
.map(str::trim)
.map(|token| token.strip_prefix('.').unwrap_or(token))
.filter(|token| !token.is_empty())
}
fn parse(
string: &str,
allow_error_code_check: ErrorCodes,
enable_per_target_ignores: bool,
extra: Option<&ExtraInfo<'_>>,
) -> LangString {
) -> Self {
let allow_error_code_check = allow_error_code_check.as_bool();
let mut seen_rust_tags = false;
let mut seen_other_tags = false;
let mut seen_custom_tag = false;
let mut data = LangString::default();
let mut ignores = vec![];
data.original = string.to_owned();
for token in Self::tokens(string) {
for token in TagIterator::new(string, extra) {
match token {
"should_panic" => {
LangStringToken::LangToken("should_panic") => {
data.should_panic = true;
seen_rust_tags = !seen_other_tags;
}
"no_run" => {
LangStringToken::LangToken("no_run") => {
data.no_run = true;
seen_rust_tags = !seen_other_tags;
}
"ignore" => {
LangStringToken::LangToken("ignore") => {
data.ignore = Ignore::All;
seen_rust_tags = !seen_other_tags;
}
x if x.starts_with("ignore-") => {
LangStringToken::LangToken(x) if x.starts_with("ignore-") => {
if enable_per_target_ignores {
ignores.push(x.trim_start_matches("ignore-").to_owned());
seen_rust_tags = !seen_other_tags;
}
}
"rust" => {
LangStringToken::LangToken("rust") => {
data.rust = true;
seen_rust_tags = true;
}
"test_harness" => {
LangStringToken::LangToken("custom") => {
seen_custom_tag = true;
}
LangStringToken::LangToken("test_harness") => {
data.test_harness = true;
seen_rust_tags = !seen_other_tags || seen_rust_tags;
}
"compile_fail" => {
LangStringToken::LangToken("compile_fail") => {
data.compile_fail = true;
seen_rust_tags = !seen_other_tags || seen_rust_tags;
data.no_run = true;
}
x if x.starts_with("edition") => {
LangStringToken::LangToken(x) if x.starts_with("edition") => {
data.edition = x[7..].parse::<Edition>().ok();
}
x if allow_error_code_check && x.starts_with('E') && x.len() == 5 => {
LangStringToken::LangToken(x)
if allow_error_code_check && x.starts_with('E') && x.len() == 5 =>
{
if x[1..].parse::<u32>().is_ok() {
data.error_codes.push(x.to_owned());
seen_rust_tags = !seen_other_tags || seen_rust_tags;
@ -925,7 +1221,7 @@ impl LangString {
seen_other_tags = true;
}
}
x if extra.is_some() => {
LangStringToken::LangToken(x) if extra.is_some() => {
let s = x.to_lowercase();
if let Some((flag, help)) = if s == "compile-fail"
|| s == "compile_fail"
@ -958,15 +1254,30 @@ impl LangString {
None
} {
if let Some(extra) = extra {
extra.error_invalid_codeblock_attr(
extra.error_invalid_codeblock_attr_with_help(
format!("unknown attribute `{x}`. Did you mean `{flag}`?"),
help,
);
}
}
seen_other_tags = true;
data.unknown.push(x.to_owned());
}
LangStringToken::LangToken(x) => {
seen_other_tags = true;
data.unknown.push(x.to_owned());
}
LangStringToken::KeyValueAttribute(key, value) => {
if key == "class" {
data.added_classes.push(value.to_owned());
} else if let Some(extra) = extra {
extra
.error_invalid_codeblock_attr(format!("unsupported attribute `{key}`"));
}
}
LangStringToken::ClassAttribute(class) => {
data.added_classes.push(class.to_owned());
}
_ => seen_other_tags = true,
}
}
@ -975,7 +1286,7 @@ impl LangString {
data.ignore = Ignore::Some(ignores);
}
data.rust &= !seen_other_tags || seen_rust_tags;
data.rust &= !seen_custom_tag && (!seen_other_tags || seen_rust_tags);
data
}

182
src/librustdoc/html/markdown/tests.rs
View File

@ -1,5 +1,8 @@
use super::{find_testable_code, plain_text_summary, short_markdown_summary};
use super::{ErrorCodes, HeadingOffset, IdMap, Ignore, LangString, Markdown, MarkdownItemInfo};
use super::{
ErrorCodes, HeadingOffset, IdMap, Ignore, LangString, LangStringToken, Markdown,
MarkdownItemInfo, TagIterator,
};
use rustc_span::edition::{Edition, DEFAULT_EDITION};
#[test]
@ -51,10 +54,32 @@ fn test_lang_string_parse() {
t(Default::default());
t(LangString { original: "rust".into(), ..Default::default() });
t(LangString { original: ".rust".into(), ..Default::default() });
t(LangString { original: "{rust}".into(), ..Default::default() });
t(LangString { original: "{.rust}".into(), ..Default::default() });
t(LangString { original: "sh".into(), rust: false, ..Default::default() });
t(LangString {
original: "rusta".into(),
rust: false,
unknown: vec!["rusta".into()],
..Default::default()
});
// error
t(LangString { original: "{rust}".into(), rust: true, ..Default::default() });
t(LangString {
original: "{.rust}".into(),
rust: true,
added_classes: vec!["rust".into()],
..Default::default()
});
t(LangString {
original: "custom,{.rust}".into(),
rust: false,
added_classes: vec!["rust".into()],
..Default::default()
});
t(LangString {
original: "sh".into(),
rust: false,
unknown: vec!["sh".into()],
..Default::default()
});
t(LangString { original: "ignore".into(), ignore: Ignore::All, ..Default::default() });
t(LangString {
original: "ignore-foo".into(),
@ -70,41 +95,56 @@ fn test_lang_string_parse() {
compile_fail: true,
..Default::default()
});
t(LangString { original: "no_run,example".into(), no_run: true, ..Default::default() });
t(LangString {
original: "no_run,example".into(),
no_run: true,
unknown: vec!["example".into()],
..Default::default()
});
t(LangString {
original: "sh,should_panic".into(),
should_panic: true,
rust: false,
unknown: vec!["sh".into()],
..Default::default()
});
t(LangString { original: "example,rust".into(), ..Default::default() });
t(LangString {
original: "test_harness,.rust".into(),
original: "example,rust".into(),
unknown: vec!["example".into()],
..Default::default()
});
t(LangString {
original: "test_harness,rusta".into(),
test_harness: true,
unknown: vec!["rusta".into()],
..Default::default()
});
t(LangString {
original: "text, no_run".into(),
no_run: true,
rust: false,
unknown: vec!["text".into()],
..Default::default()
});
t(LangString {
original: "text,no_run".into(),
no_run: true,
rust: false,
unknown: vec!["text".into()],
..Default::default()
});
t(LangString {
original: "text,no_run, ".into(),
no_run: true,
rust: false,
unknown: vec!["text".into()],
..Default::default()
});
t(LangString {
original: "text,no_run,".into(),
no_run: true,
rust: false,
unknown: vec!["text".into()],
..Default::default()
});
t(LangString {
@ -117,29 +157,125 @@ fn test_lang_string_parse() {
edition: Some(Edition::Edition2018),
..Default::default()
});
t(LangString {
original: "{class=test}".into(),
added_classes: vec!["test".into()],
rust: true,
..Default::default()
});
t(LangString {
original: "custom,{class=test}".into(),
added_classes: vec!["test".into()],
rust: false,
..Default::default()
});
t(LangString {
original: "{.test}".into(),
added_classes: vec!["test".into()],
rust: true,
..Default::default()
});
t(LangString {
original: "custom,{.test}".into(),
added_classes: vec!["test".into()],
rust: false,
..Default::default()
});
t(LangString {
original: "rust,{class=test,.test2}".into(),
added_classes: vec!["test".into(), "test2".into()],
rust: true,
..Default::default()
});
t(LangString {
original: "{class=test:with:colon .test1}".into(),
added_classes: vec!["test:with:colon".into(), "test1".into()],
rust: true,
..Default::default()
});
t(LangString {
original: "custom,{class=test:with:colon .test1}".into(),
added_classes: vec!["test:with:colon".into(), "test1".into()],
rust: false,
..Default::default()
});
t(LangString {
original: "{class=first,class=second}".into(),
added_classes: vec!["first".into(), "second".into()],
rust: true,
..Default::default()
});
t(LangString {
original: "custom,{class=first,class=second}".into(),
added_classes: vec!["first".into(), "second".into()],
rust: false,
..Default::default()
});
t(LangString {
original: "{class=first,.second},unknown".into(),
added_classes: vec!["first".into(), "second".into()],
rust: false,
unknown: vec!["unknown".into()],
..Default::default()
});
t(LangString {
original: "{class=first .second} unknown".into(),
added_classes: vec!["first".into(), "second".into()],
rust: false,
unknown: vec!["unknown".into()],
..Default::default()
});
// error
t(LangString { original: "{.first.second}".into(), rust: true, ..Default::default() });
// error
t(LangString { original: "{class=first=second}".into(), rust: true, ..Default::default() });
// error
t(LangString { original: "{class=first.second}".into(), rust: true, ..Default::default() });
// error
t(LangString { original: "{class=.first}".into(), rust: true, ..Default::default() });
t(LangString {
original: r#"{class="first"}"#.into(),
added_classes: vec!["first".into()],
rust: true,
..Default::default()
});
t(LangString {
original: r#"custom,{class="first"}"#.into(),
added_classes: vec!["first".into()],
rust: false,
..Default::default()
});
// error
t(LangString { original: r#"{class=f"irst"}"#.into(), rust: true, ..Default::default() });
}
#[test]
fn test_lang_string_tokenizer() {
fn case(lang_string: &str, want: &[&str]) {
let have = LangString::tokens(lang_string).collect::<Vec<&str>>();
fn case(lang_string: &str, want: &[LangStringToken<'_>]) {
let have = TagIterator::new(lang_string, None).collect::<Vec<_>>();
assert_eq!(have, want, "Unexpected lang string split for `{}`", lang_string);
}
case("", &[]);
case("foo", &["foo"]);
case("foo,bar", &["foo", "bar"]);
case(".foo,.bar", &["foo", "bar"]);
case("{.foo,.bar}", &["foo", "bar"]);
case(" {.foo,.bar} ", &["foo", "bar"]);
case("foo bar", &["foo", "bar"]);
case("foo\tbar", &["foo", "bar"]);
case("foo\t, bar", &["foo", "bar"]);
case(" foo , bar ", &["foo", "bar"]);
case(",,foo,,bar,,", &["foo", "bar"]);
case("foo=bar", &["foo=bar"]);
case("a-b-c", &["a-b-c"]);
case("a_b_c", &["a_b_c"]);
case("foo", &[LangStringToken::LangToken("foo")]);
case("foo,bar", &[LangStringToken::LangToken("foo"), LangStringToken::LangToken("bar")]);
case(".foo,.bar", &[]);
case(
"{.foo,.bar}",
&[LangStringToken::ClassAttribute("foo"), LangStringToken::ClassAttribute("bar")],
);
case(
" {.foo,.bar} ",
&[LangStringToken::ClassAttribute("foo"), LangStringToken::ClassAttribute("bar")],
);
case("foo bar", &[LangStringToken::LangToken("foo"), LangStringToken::LangToken("bar")]);
case("foo\tbar", &[LangStringToken::LangToken("foo"), LangStringToken::LangToken("bar")]);
case("foo\t, bar", &[LangStringToken::LangToken("foo"), LangStringToken::LangToken("bar")]);
case(" foo , bar ", &[LangStringToken::LangToken("foo"), LangStringToken::LangToken("bar")]);
case(",,foo,,bar,,", &[LangStringToken::LangToken("foo"), LangStringToken::LangToken("bar")]);
case("foo=bar", &[]);
case("a-b-c", &[LangStringToken::LangToken("a-b-c")]);
case("a_b_c", &[LangStringToken::LangToken("a_b_c")]);
}
#[test]

77
src/librustdoc/passes/check_custom_code_classes.rs Normal file
View File

@ -0,0 +1,77 @@
//! NIGHTLY & UNSTABLE CHECK: custom_code_classes_in_docs
//!
//! This pass will produce errors when finding custom classes outside of
//! nightly + relevant feature active.
use super::Pass;
use crate::clean::{Crate, Item};
use crate::core::DocContext;
use crate::fold::DocFolder;
use crate::html::markdown::{find_codes, ErrorCodes, LangString};
use rustc_session::parse::feature_err;
use rustc_span::symbol::sym;
pub(crate) const CHECK_CUSTOM_CODE_CLASSES: Pass = Pass {
name: "check-custom-code-classes",
run: check_custom_code_classes,
description: "check for custom code classes without the feature-gate enabled",
};
pub(crate) fn check_custom_code_classes(krate: Crate, cx: &mut DocContext<'_>) -> Crate {
let mut coll = CustomCodeClassLinter { cx };
coll.fold_crate(krate)
}
struct CustomCodeClassLinter<'a, 'tcx> {
cx: &'a DocContext<'tcx>,
}
impl<'a, 'tcx> DocFolder for CustomCodeClassLinter<'a, 'tcx> {
fn fold_item(&mut self, item: Item) -> Option<Item> {
look_for_custom_classes(&self.cx, &item);
Some(self.fold_item_recur(item))
}
}
#[derive(Debug)]
struct TestsWithCustomClasses {
custom_classes_found: Vec<String>,
}
impl crate::doctest::Tester for TestsWithCustomClasses {
fn add_test(&mut self, _: String, config: LangString, _: usize) {
self.custom_classes_found.extend(config.added_classes.into_iter());
}
}
pub(crate) fn look_for_custom_classes<'tcx>(cx: &DocContext<'tcx>, item: &Item) {
if !item.item_id.is_local() {
// If non-local, no need to check anything.
return;
}
let mut tests = TestsWithCustomClasses { custom_classes_found: vec![] };
let dox = item.attrs.doc_value();
find_codes(&dox, &mut tests, ErrorCodes::No, false, None, true);
if !tests.custom_classes_found.is_empty() && !cx.tcx.features().custom_code_classes_in_docs {
feature_err(
&cx.tcx.sess.parse_sess,
sym::custom_code_classes_in_docs,
item.attr_span(cx.tcx),
"custom classes in code blocks are unstable",
)
.note(
// This will list the wrong items to make them more easily searchable.
// To ensure the most correct hits, it adds back the 'class:' that was stripped.
format!(
"found these custom classes: class={}",
tests.custom_classes_found.join(",class=")
),
)
.emit();
}
}

5
src/librustdoc/passes/mod.rs
View File

@ -35,6 +35,9 @@ pub(crate) use self::calculate_doc_coverage::CALCULATE_DOC_COVERAGE;
mod lint;
pub(crate) use self::lint::RUN_LINTS;
mod check_custom_code_classes;
pub(crate) use self::check_custom_code_classes::CHECK_CUSTOM_CODE_CLASSES;
/// A single pass over the cleaned documentation.
///
/// Runs in the compiler context, so it has access to types and traits and the like.
@ -66,6 +69,7 @@ pub(crate) enum Condition {
/// The full list of passes.
pub(crate) const PASSES: &[Pass] = &[
CHECK_CUSTOM_CODE_CLASSES,
CHECK_DOC_TEST_VISIBILITY,
STRIP_HIDDEN,
STRIP_PRIVATE,
@ -79,6 +83,7 @@ pub(crate) const PASSES: &[Pass] = &[
/// The list of passes run by default.
pub(crate) const DEFAULT_PASSES: &[ConditionalPass] = &[
ConditionalPass::always(CHECK_CUSTOM_CODE_CLASSES),
ConditionalPass::always(COLLECT_TRAIT_IMPLS),
ConditionalPass::always(CHECK_DOC_TEST_VISIBILITY),
ConditionalPass::new(STRIP_HIDDEN, WhenNotDocumentHidden),

85
tests/rustdoc-ui/custom_code_classes_in_docs-warning.rs Normal file
View File

@ -0,0 +1,85 @@
// This test ensures that warnings are working as expected for "custom_code_classes_in_docs"
// feature.
#![feature(custom_code_classes_in_docs)]
#![deny(warnings)]
#![feature(no_core)]
#![no_core]
/// ```{. }
/// main;
/// ```
//~^^^ ERROR unexpected ` ` character after `.`
pub fn foo() {}
/// ```{class= a}
/// main;
/// ```
//~^^^ ERROR unexpected ` ` character after `=`
pub fn foo2() {}
/// ```{#id}
/// main;
/// ```
//~^^^ ERROR unexpected character `#`
pub fn foo3() {}
/// ```{{
/// main;
/// ```
//~^^^ ERROR unexpected character `{`
pub fn foo4() {}
/// ```}
/// main;
/// ```
//~^^^ ERROR unexpected character `}`
pub fn foo5() {}
/// ```)
/// main;
/// ```
//~^^^ ERROR unexpected character `)`
pub fn foo6() {}
/// ```{class=}
/// main;
/// ```
//~^^^ ERROR unexpected `}` character after `=`
pub fn foo7() {}
/// ```(
/// main;
/// ```
//~^^^ ERROR unclosed comment: missing `)` at the end
pub fn foo8() {}
/// ```{class=one=two}
/// main;
/// ```
//~^^^ ERROR unexpected `=`
pub fn foo9() {}
/// ```{.one.two}
/// main;
/// ```
//~^^^ ERROR unexpected `.` character
pub fn foo10() {}
/// ```{class=.one}
/// main;
/// ```
//~^^^ ERROR unexpected `.` character after `=`
pub fn foo11() {}
/// ```{class=one.two}
/// main;
/// ```
//~^^^ ERROR unexpected `.` character
pub fn foo12() {}
/// ```{(comment)}
/// main;
/// ```
//~^^^ ERROR unexpected character `(`
pub fn foo13() {}

113
tests/rustdoc-ui/custom_code_classes_in_docs-warning.stderr Normal file
View File

@ -0,0 +1,113 @@
error: unexpected ` ` character after `.`
--> $DIR/custom_code_classes_in_docs-warning.rs:9:1
|
LL | / /// ```{. }
LL | | /// main;
LL | | /// ```
| |_______^
|
note: the lint level is defined here
--> $DIR/custom_code_classes_in_docs-warning.rs:5:9
|
LL | #![deny(warnings)]
| ^^^^^^^^
= note: `#[deny(rustdoc::invalid_codeblock_attributes)]` implied by `#[deny(warnings)]`
error: unexpected ` ` character after `=`
--> $DIR/custom_code_classes_in_docs-warning.rs:15:1
|
LL | / /// ```{class= a}
LL | | /// main;
LL | | /// ```
| |_______^
error: unexpected character `#`
--> $DIR/custom_code_classes_in_docs-warning.rs:21:1
|
LL | / /// ```{#id}
LL | | /// main;
LL | | /// ```
| |_______^
error: unexpected character `{`
--> $DIR/custom_code_classes_in_docs-warning.rs:27:1
|
LL | / /// ```{{
LL | | /// main;
LL | | /// ```
| |_______^
error: unexpected character `}`
--> $DIR/custom_code_classes_in_docs-warning.rs:33:1
|
LL | / /// ```}
LL | | /// main;
LL | | /// ```
| |_______^
error: unexpected character `)`
--> $DIR/custom_code_classes_in_docs-warning.rs:39:1
|
LL | / /// ```)
LL | | /// main;
LL | | /// ```
| |_______^
error: unexpected `}` character after `=`
--> $DIR/custom_code_classes_in_docs-warning.rs:45:1
|
LL | / /// ```{class=}
LL | | /// main;
LL | | /// ```
| |_______^
error: unclosed comment: missing `)` at the end
--> $DIR/custom_code_classes_in_docs-warning.rs:51:1
|
LL | / /// ```(
LL | | /// main;
LL | | /// ```
| |_______^
error: unexpected `=` character
--> $DIR/custom_code_classes_in_docs-warning.rs:57:1
|
LL | / /// ```{class=one=two}
LL | | /// main;
LL | | /// ```
| |_______^
error: unexpected `.` character
--> $DIR/custom_code_classes_in_docs-warning.rs:63:1
|
LL | / /// ```{.one.two}
LL | | /// main;
LL | | /// ```
| |_______^
error: unexpected `.` character after `=`
--> $DIR/custom_code_classes_in_docs-warning.rs:69:1
|
LL | / /// ```{class=.one}
LL | | /// main;
LL | | /// ```
| |_______^
error: unexpected `.` character
--> $DIR/custom_code_classes_in_docs-warning.rs:75:1
|
LL | / /// ```{class=one.two}
LL | | /// main;
LL | | /// ```
| |_______^
error: unexpected character `(`
--> $DIR/custom_code_classes_in_docs-warning.rs:81:1
|
LL | / /// ```{(comment)}
LL | | /// main;
LL | | /// ```
| |_______^
error: aborting due to 13 previous errors

17
tests/rustdoc-ui/custom_code_classes_in_docs-warning3.rs Normal file
View File

@ -0,0 +1,17 @@
// This test ensures that warnings are working as expected for "custom_code_classes_in_docs"
// feature.
#![feature(custom_code_classes_in_docs)]
#![deny(warnings)]
#![feature(no_core)]
#![no_core]
/// ```{class="}
/// main;
/// ```
//~^^^ ERROR unclosed quote string
//~| ERROR unclosed quote string
/// ```"
/// main;
/// ```
pub fn foo() {}

33
tests/rustdoc-ui/custom_code_classes_in_docs-warning3.stderr Normal file
View File

@ -0,0 +1,33 @@
error: unclosed quote string `"`
--> $DIR/custom_code_classes_in_docs-warning3.rs:9:1
|
LL | / /// ```{class="}
LL | | /// main;
LL | | /// ```
LL | |
... |
LL | | /// main;
LL | | /// ```
| |_______^
|
note: the lint level is defined here
--> $DIR/custom_code_classes_in_docs-warning3.rs:5:9
|
LL | #![deny(warnings)]
| ^^^^^^^^
= note: `#[deny(rustdoc::invalid_codeblock_attributes)]` implied by `#[deny(warnings)]`
error: unclosed quote string `"`
--> $DIR/custom_code_classes_in_docs-warning3.rs:9:1
|
LL | / /// ```{class="}
LL | | /// main;
LL | | /// ```
LL | |
... |
LL | | /// main;
LL | | /// ```
| |_______^
error: aborting due to 2 previous errors

5
tests/rustdoc-ui/feature-gate-custom_code_classes_in_docs.rs Normal file
View File

@ -0,0 +1,5 @@
/// ```{class=language-c}
/// int main(void) { return 0; }
/// ```
//~^^^ ERROR 1:1: 3:8: custom classes in code blocks are unstable [E0658]
pub struct Bar;

15
tests/rustdoc-ui/feature-gate-custom_code_classes_in_docs.stderr Normal file
View File

@ -0,0 +1,15 @@
error[E0658]: custom classes in code blocks are unstable
--> $DIR/feature-gate-custom_code_classes_in_docs.rs:1:1
|
LL | / /// ```{class=language-c}
LL | | /// int main(void) { return 0; }
LL | | /// ```
| |_______^
|
= note: see issue #79483 <https://github.com/rust-lang/rust/issues/79483> for more information
= help: add `#![feature(custom_code_classes_in_docs)]` to the crate attributes to enable
= note: found these custom classes: class=language-c
error: aborting due to previous error
For more information about this error, try `rustc --explain E0658`.

2
tests/rustdoc-ui/issues/issue-91713.stdout
View File

@ -1,4 +1,5 @@
Available passes for running rustdoc:
check-custom-code-classes - check for custom code classes without the feature-gate enabled
check_doc_test_visibility - run various visibility-related lints on doctests
strip-hidden - strips all `#[doc(hidden)]` items from the output
strip-private - strips all private items from a crate which cannot be seen externally, implies strip-priv-imports
@ -10,6 +11,7 @@ calculate-doc-coverage - counts the number of items with and without documentati
run-lints - runs some of rustdoc's lints
Default passes for rustdoc:
check-custom-code-classes
collect-trait-impls
check_doc_test_visibility
strip-hidden (when not --document-hidden-items)

28
tests/rustdoc/custom_code_classes.rs Normal file
View File

@ -0,0 +1,28 @@
// Test for `custom_code_classes_in_docs` feature.
#![feature(custom_code_classes_in_docs)]
#![crate_name = "foo"]
#![feature(no_core)]
#![no_core]
// @has 'foo/struct.Bar.html'
// @has - '//*[@id="main-content"]//pre[@class="language-whatever hoho-c"]' 'main;'
// @has - '//*[@id="main-content"]//pre[@class="language-whatever2 haha-c"]' 'main;'
// @has - '//*[@id="main-content"]//pre[@class="language-whatever4 huhu-c"]' 'main;'
/// ```{class=hoho-c},whatever
/// main;
/// ```
///
/// Testing multiple kinds of orders.
///
/// ```whatever2 {class=haha-c}
/// main;
/// ```
///
/// Testing with multiple "unknown". Only the first should be used.
///
/// ```whatever4,{.huhu-c} whatever5
/// main;
/// ```
pub struct Bar;