macro_rules: Preserve all metavariable spans in a global side table
This PR preserves spans of `tt` metavariables used to pass tokens to declarative macros.
Such metavariable spans can then be used in span combination operations like `Span::to` to improve all kinds of diagnostics.
Spans of non-`tt` metavariables are currently kept in nonterminal tokens, but the long term plan is remove all nonterminal tokens from rustc parser and rely on the proc macro model with invisible delimiters (#114647, #67062).
In particular, `NtIdent` nonterminal (corresponding to `ident` metavariables) becomes easy to remove when this PR lands (#119412 does it).
The metavariable spans are kept in a global side table keyed by `Span`s of original tokens.
The alternative to the side table is keeping them in `SpanData` instead, but the performance regressions would be large because any spans from tokens passed to declarative macros would stop being inline and would work through span interner instead, and the penalty would be paid even if we never use the metavar span for the given original span.
(But also see the comment on `fn maybe_use_metavar_location` describing the map collision issues with the side table approach.)
There are also other alternatives - keeping the metavar span in `Token` or `TokenTree`, but associating it with `Span` itsel is the most natural choice because metavar spans are used in span combining operations, and those operations are not necessarily tied to tokens.
There are a number of cases where we erroneously omit the space between
two tokens, all involving an exception to a more general case. The
affected tokens are `$`, `!`, `.`, `,`, and `let` followed by a
parenthesis.
This fixes a lot of FIXME comments.
fix: Overlapping spans in delimited meta-vars
Closes#118786
Delimited meta-vars inside of MBE's spans were set to have the same opening and closing position resulting in an ICE when debug assertions were enabled and an error was present in the templated code.
This ensures that the spans do not overlap, whilst still having the spans point at the usage of the meta-var inside the macro definition.
It includes a regression test.
🖤
[`RFC 3086`] Attempt to try to resolve blocking concerns
Implements what is described at https://github.com/rust-lang/rust/issues/83527#issuecomment-1744822345 to hopefully make some progress.
It is unknown if such approach is or isn't desired due to the lack of further feedback, as such, it is probably best to nominate this PR to the official entities.
`@rustbot` labels +I-compiler-nominated
Do not parenthesize exterior struct lit inside match guards
Before this PR, the AST pretty-printer injects parentheses around expressions any time parens _could_ be needed depending on what else is in the code that surrounds that expression. But the pretty-printer did not pass around enough context to understand whether parentheses really _are_ needed on any particular expression. As a consequence, there are false positives where unneeded parentheses are being inserted.
Example:
```rust
#![feature(if_let_guard)]
macro_rules! pp {
($e:expr) => {
stringify!($e)
};
}
fn main() {
println!("{}", pp!(match () { () if let _ = Struct {} => {} }));
}
```
**Before:**
```console
match () { () if let _ = (Struct {}) => {} }
```
**After:**
```console
match () { () if let _ = Struct {} => {} }
```
This PR introduces a bit of state that is passed across various expression printing methods to help understand accurately whether particular situations require parentheses injected by the pretty printer, and it fixes one such false positive involving match guards as shown above.
There are other parenthesization false positive cases not fixed by this PR. I intend to address these in follow-up PRs. For example here is one: the expression `{ let _ = match x {} + 1; }` is pretty-printed as `{ let _ = (match x {}) + 1; }` despite there being no reason for parentheses to appear there.
This is an extension of the previous commit. It means the output of
something like this:
```
stringify!(let a: Vec<u32> = vec![];)
```
goes from this:
```
let a: Vec<u32> = vec![] ;
```
With this PR, it now produces this string:
```
let a: Vec<u32> = vec![];
```
`tokenstream::Spacing` appears on all `TokenTree::Token` instances,
both punct and non-punct. Its current usage:
- `Joint` means "can join with the next token *and* that token is a
punct".
- `Alone` means "cannot join with the next token *or* can join with the
next token but that token is not a punct".
The fact that `Alone` is used for two different cases is awkward.
This commit augments `tokenstream::Spacing` with a new variant
`JointHidden`, resulting in:
- `Joint` means "can join with the next token *and* that token is a
punct".
- `JointHidden` means "can join with the next token *and* that token is a
not a punct".
- `Alone` means "cannot join with the next token".
This *drastically* improves the output of `print_tts`. For example,
this:
```
stringify!(let a: Vec<u32> = vec![];)
```
currently produces this string:
```
let a : Vec < u32 > = vec! [] ;
```
With this PR, it now produces this string:
```
let a: Vec<u32> = vec![] ;
```
(The space after the `]` is because `TokenTree::Delimited` currently
doesn't have spacing information. The subsequent commit fixes this.)
The new `print_tts` doesn't replicate original code perfectly. E.g.
multiple space characters will be condensed into a single space
character. But it's much improved.
`print_tts` still produces the old, uglier output for code produced by
proc macros. Because we have to translate the generated code from
`proc_macro::Spacing` to the more expressive `token::Spacing`, which
results in too much `proc_macro::Along` usage and no
`proc_macro::JointHidden` usage. So `space_between` still exists and
is used by `print_tts` in conjunction with the `Spacing` field.
This change will also help with the removal of `Token::Interpolated`.
Currently interpolated tokens are pretty-printed nicely via AST pretty
printing. `Token::Interpolated` removal will mean they get printed with
`print_tts`. Without this change, that would result in much uglier
output for code produced by decl macro expansions. With this change, AST
pretty printing and `print_tts` produce similar results.
The commit also tweaks the comments on `proc_macro::Spacing`. In
particular, it refers to "compound tokens" rather than "multi-char
operators" because lifetimes aren't operators.
To avoid `!matches!(...)`, which is hard to think about. Instead every
case now uses direct pattern matching and returns true or false.
Also add a couple of cases to the `stringify.rs` test that currently
print badly.
Currently it only tests AST pretty-printing. This commit changes it to
run every example through both AST pretty-printing and TokenStream
pretty-printing. This makes it clear where there two pretty-printing
approaches produce different results.
The assertion in `assert-long-condition.rs` used to be fail like this, all on
one line:
```
thread 'main' panicked at 'assertion failed: 1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + 10 + 11 + 12 + 13 + 14 + 15 + 16 + 17 + 18\n + 19 + 20 + 21 + 22 + 23 + 24 + 25 == 0', tests/ui/macros/assert-long-condition.rs:7:5
```
The `\n` and subsequent indent is because the condition is pretty-printed, and
the pretty-printer inserts a newline. Printing the newline in this way is
arguably reasonable given that the message appears within single quotes, which
is very similar to a string literal.
However, after the assertion printing improvements that were released in 1.73,
the assertion now fails like this:
```
thread 'main' panicked at tests/ui/macros/assert-long-condition.rs:7:5:
assertion failed: 1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + 10 + 11 + 12 + 13 + 14 + 15 + 16 + 17 + 18\n + 19 + 20 + 21 + 22 + 23 + 24 + 25 == 0
```
Now that there are no single quotes around the pretty-printed condition, the
`\n` is quite strange.
This commit gets rid of the `\n`, by removing the `escape_debug` done on the
pretty-printed message. This results in the following:
```
thread 'main' panicked at tests/ui/macros/assert-long-condition.rs:7:5:
assertion failed: 1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + 10 + 11 + 12 + 13 + 14 + 15 + 16 + 17 + 18
+ 19 + 20 + 21 + 22 + 23 + 24 + 25 == 0
```
The overly-large indent is still strange, but that's a separate pretty-printing issue.
This change helps with #108341.
`Nonterminal`-related cleanups
In #114647 I am trying to remove `Nonterminal`. It has a number of preliminary cleanups that are worth merging even if #114647 doesn't merge, so let's do them in this PR.
r? `@petrochenkov`
Modify panic message for `assert_eq!`, `assert_ne!`, the currently unstable `assert_matches!`, as well as the corresponding `debug_assert_*` macros.
```rust
assert_eq!(1 + 1, 3);
assert_eq!(1 + 1, 3, "my custom message value={}!", 42);
```
```plain
thread 'main' panicked at $DIR/main.rs:6:5:
assertion failed: `(left == right)`
left: `2`,
right: `3`
```
```plain
thread 'main' panicked at $DIR/main.rs:6:5:
assertion failed: `(left == right)`
left: `2`,
right: `3`: my custom message value=42!
```
```plain
thread 'main' panicked at $DIR/main.rs:6:5:
assertion `left == right` failed
left: 2
right: 3
```
```plain
thread 'main' panicked at $DIR/main.rs:6:5:
assertion `left == right` failed: my custom message value=42!
left: 2
right: 3
```
This PR is a simpler subset of the #111030, but it does NOT stringify the original left and right source code assert expressions, thus should be faster to compile.
If a raw string was used in the `env!` invocation, then it should also
be shown in the diagnostic messages as a raw string.
Signed-off-by: David Wood <david@davidtw.co>
