Fix parenthesization of subexprs containing statement boundary
This PR fixes a multitude of false negatives and false positives in the AST pretty printer's parenthesis insertion related to statement boundaries — statements which terminate unexpectedly early if there aren't parentheses.
Without this fix, the AST pretty printer (including both `stringify!` and `rustc -Zunpretty=expanded`) is prone to producing output which is not syntactically valid Rust. Invalid output is problematic because it means Rustfmt is unable to parse the output of `cargo expand`, for example, causing friction by forcing someone trying to debug a macro into reading poorly formatted code.
I believe the set of bugs fixed in this PR account for the most prevalent reason that `cargo expand` produces invalid output in real-world usage.
Fixes#98790.
## False negatives
The following is a correct program — `cargo check` succeeds.
```rust
macro_rules! m {
($e:expr) => {
match () { _ => $e }
};
}
fn main() {
m!({ 1 } - 1);
}
```
But `rustc -Zunpretty=expanded main.rs` produces output that is invalid Rust syntax, because parenthesization is needed and not being done by the pretty printer.
```rust
fn main() { match () { _ => { 1 } - 1, }; }
```
Piping this expanded code to rustfmt, it fails to parse.
```console
error: unexpected `,` in pattern
--> <stdin>:1:38
|
1 | fn main() { match () { _ => { 1 } - 1, }; }
| ^
|
help: try adding parentheses to match on a tuple...
|
1 | fn main() { match () { _ => { 1 } (- 1,) }; }
| + +
help: ...or a vertical bar to match on multiple alternatives
|
1 | fn main() { match () { _ => { 1 } - 1 | }; }
| ~~~~~
```
Fixed output after this PR:
```rust
fn main() { match () { _ => ({ 1 }) - 1, }; }
```
## False positives
Less problematic, but worth fixing (just like #118726).
```rust
fn main() {
let _ = match () { _ => 1 } - 1;
}
```
Output of `rustc -Zunpretty=expanded lib.rs` before this PR. There is no reason parentheses would need to be inserted there.
```rust
fn main() { let _ = (match () { _ => 1, }) - 1; }
```
After this PR:
```rust
fn main() { let _ = match () { _ => 1, } - 1; }
```
## Alternatives considered
In this PR I opted to parenthesize only the leading subexpression causing the statement boundary, rather than the entire statement. Example:
```rust
macro_rules! m {
($e:expr) => {
$e
};
}
fn main() {
m!(loop { break [1]; }[0] - 1);
}
```
This PR produces the following pretty-printed contents for fn main:
```rust
(loop { break [1]; })[0] - 1;
```
A different equally correct output would be:
```rust
(loop { break [1]; }[0] - 1);
```
I chose the one I did because it is the *only* approach used by handwritten code in the standard library and compiler. There are 4 places where parenthesization is being used to prevent a statement boundary, and in all 4, the developer has chosen to parenthesize the smallest subexpression rather than the whole statement:
b37d43efd9/compiler/rustc_codegen_cranelift/example/alloc_system.rs (L102)b37d43efd9/compiler/rustc_parse/src/errors.rs (L1021-L1029)b37d43efd9/library/core/src/future/poll_fn.rs (L151)b37d43efd9/library/core/src/ops/range.rs (L824-L828)
Clairify `ast::PatKind::Struct` presese of `..` by using an enum instead of a bool
The bool is mainly used for when a `..` is present, but it is also set on recovery to avoid errors. The doc comment not describes both of these cases.
See cee794ee98/compiler/rustc_parse/src/parser/pat.rs (L890-L897) for the only place this is constructed.
r? ``@compiler-errors``
Rid the AST & HIR pretty printer of cruft
Found while working on #119163.
For `trait Trait: ?Sized {}` (semantically malformed), we currently output `trait Trait for ? Sized {}` (sic!) / `trait Trait for ? Sized { }` (sic!) if `-Zunpretty=expanded` / `-Zunpretty=hir` is passed.
`trait Tr for Sized? {}` (#15521) and later also `trait Tr for ?Sized {}` (I guess, #20194) is former Rust syntax. Hence I'm removing these outdated branches.
~~This will conflict with #119163, therefore marking this PR as blocked.~~ Rebased
Add support for `for await` loops
This adds support for `for await` loops. This includes parsing, desugaring in AST->HIR lowering, and adding some support functions to the library.
Given a loop like:
```rust
for await i in iter {
...
}
```
this is desugared to something like:
```rust
let mut iter = iter.into_async_iter();
while let Some(i) = loop {
match core::pin::Pin::new(&mut iter).poll_next(cx) {
Poll::Ready(i) => break i,
Poll::Pending => yield,
}
} {
...
}
```
This PR also adds a basic `IntoAsyncIterator` trait. This is partly for symmetry with the way `Iterator` and `IntoIterator` work. The other reason is that for async iterators it's helpful to have a place apart from the data structure being iterated over to store state. `IntoAsyncIterator` gives us a good place to do this.
I've gated this feature behind `async_for_loop` and opened #118898 as the feature tracking issue.
r? `@compiler-errors`
Refactor AST trait bound modifiers
Instead of having two types to represent trait bound modifiers in the parser / the AST (`parser::ty::BoundModifiers` & `ast::TraitBoundModifier`), only to map one to the other later, just use `parser::ty::BoundModifiers` (moved & renamed to `ast::TraitBoundModifiers`).
The struct type is more extensible and easier to deal with (see [here](https://github.com/rust-lang/rust/pull/119099/files#r1430749981) and [here](https://github.com/rust-lang/rust/pull/119099/files#r1430752116) for context) since it more closely models what it represents: A compound of two kinds of modifiers, constness and polarity. Modeling this as an enum (the now removed `ast::TraitBoundModifier`) meant one had to add a new variant per *combination* of modifier kind, which simply isn't scalable and which lead to a lot of explicit non-DRY matches.
NB: `hir::TraitBoundModifier` being an enum is fine since HIR doesn't need to worry representing invalid modifier kind combinations as those get rejected during AST validation thereby immensely cutting down the number of possibilities.
Do not parenthesize exterior struct lit inside match guards
Before this PR, the AST pretty-printer injects parentheses around expressions any time parens _could_ be needed depending on what else is in the code that surrounds that expression. But the pretty-printer did not pass around enough context to understand whether parentheses really _are_ needed on any particular expression. As a consequence, there are false positives where unneeded parentheses are being inserted.
Example:
```rust
#![feature(if_let_guard)]
macro_rules! pp {
($e:expr) => {
stringify!($e)
};
}
fn main() {
println!("{}", pp!(match () { () if let _ = Struct {} => {} }));
}
```
**Before:**
```console
match () { () if let _ = (Struct {}) => {} }
```
**After:**
```console
match () { () if let _ = Struct {} => {} }
```
This PR introduces a bit of state that is passed across various expression printing methods to help understand accurately whether particular situations require parentheses injected by the pretty printer, and it fixes one such false positive involving match guards as shown above.
There are other parenthesization false positive cases not fixed by this PR. I intend to address these in follow-up PRs. For example here is one: the expression `{ let _ = match x {} + 1; }` is pretty-printed as `{ let _ = (match x {}) + 1; }` despite there being no reason for parentheses to appear there.
This is an extension of the previous commit. It means the output of
something like this:
```
stringify!(let a: Vec<u32> = vec![];)
```
goes from this:
```
let a: Vec<u32> = vec![] ;
```
With this PR, it now produces this string:
```
let a: Vec<u32> = vec![];
```
`tokenstream::Spacing` appears on all `TokenTree::Token` instances,
both punct and non-punct. Its current usage:
- `Joint` means "can join with the next token *and* that token is a
punct".
- `Alone` means "cannot join with the next token *or* can join with the
next token but that token is not a punct".
The fact that `Alone` is used for two different cases is awkward.
This commit augments `tokenstream::Spacing` with a new variant
`JointHidden`, resulting in:
- `Joint` means "can join with the next token *and* that token is a
punct".
- `JointHidden` means "can join with the next token *and* that token is a
not a punct".
- `Alone` means "cannot join with the next token".
This *drastically* improves the output of `print_tts`. For example,
this:
```
stringify!(let a: Vec<u32> = vec![];)
```
currently produces this string:
```
let a : Vec < u32 > = vec! [] ;
```
With this PR, it now produces this string:
```
let a: Vec<u32> = vec![] ;
```
(The space after the `]` is because `TokenTree::Delimited` currently
doesn't have spacing information. The subsequent commit fixes this.)
The new `print_tts` doesn't replicate original code perfectly. E.g.
multiple space characters will be condensed into a single space
character. But it's much improved.
`print_tts` still produces the old, uglier output for code produced by
proc macros. Because we have to translate the generated code from
`proc_macro::Spacing` to the more expressive `token::Spacing`, which
results in too much `proc_macro::Along` usage and no
`proc_macro::JointHidden` usage. So `space_between` still exists and
is used by `print_tts` in conjunction with the `Spacing` field.
This change will also help with the removal of `Token::Interpolated`.
Currently interpolated tokens are pretty-printed nicely via AST pretty
printing. `Token::Interpolated` removal will mean they get printed with
`print_tts`. Without this change, that would result in much uglier
output for code produced by decl macro expansions. With this change, AST
pretty printing and `print_tts` produce similar results.
The commit also tweaks the comments on `proc_macro::Spacing`. In
particular, it refers to "compound tokens" rather than "multi-char
operators" because lifetimes aren't operators.
tidy error: /git/rust/compiler/rustc_ast_pretty/src/pprust/state.rs:1165: unexplained "```ignore" doctest; try one:
* make the test actually pass, by adding necessary imports and declarations, or
* use "```text", if the code is not Rust code, or
* use "```compile_fail,Ennnn", if the code is expected to fail at compile time, or
* use "```should_panic", if the code is expected to fail at run time, or
* use "```no_run", if the code should type-check but not necessary linkable/runnable, or
* explain it like "```ignore (cannot-test-this-because-xxxx)", if the annotation cannot be avoided.
tidy error: /git/rust/compiler/rustc_ast_pretty/src/pprust/state.rs:1176: unexplained "```ignore" doctest; try one:
* make the test actually pass, by adding necessary imports and declarations, or
* use "```text", if the code is not Rust code, or
* use "```compile_fail,Ennnn", if the code is expected to fail at compile time, or
* use "```should_panic", if the code is expected to fail at run time, or
* use "```no_run", if the code should type-check but not necessary linkable/runnable, or
* explain it like "```ignore (cannot-test-this-because-xxxx)", if the annotation cannot be avoided.
In all four of Break, Closure, Ret, Yeet, the needs_par_as_let_scrutinee
is guaranteed to return true because the .precedence().order() of those
expr kinds is <= AssocOp::LAnd.precedence().
The relevant functions in rustc_ast::util::parser are:
fn needs_par_as_let_scrutinee(order: i8) -> bool {
order <= prec_let_scrutinee_needs_par() as i8
}
fn prec_let_scrutinee_needs_par() -> usize {
AssocOp::LAnd.precedence()
}
The .precedence().order() of Closure is PREC_CLOSURE (-40) and of Break,
Ret, Yeet is PREC_JUMP (-30).
The value of AssocOp::LAnd.precedence() is 6.
So this commit causes no change in behavior, only potentially
performance by doing a redundant call to contains_exterior_struct_lit in
those four cases. This is fine because Break, Closure, Ret, Yeet should
be exceedingly rare in the position of a let scrutinee.
Introduce support for `async gen` blocks
I'm delighted to demonstrate that `async gen` block are not very difficult to support. They're simply coroutines that yield `Poll<Option<T>>` and return `()`.
**This PR is WIP and in draft mode for now** -- I'm mostly putting it up to show folks that it's possible. This PR needs a lang-team experiment associated with it or possible an RFC, since I don't think it falls under the jurisdiction of the `gen` RFC that was recently authored by oli (https://github.com/rust-lang/rfcs/pull/3513, https://github.com/rust-lang/rust/issues/117078).
### Technical note on the pre-generator-transform yield type:
The reason that the underlying coroutines yield `Poll<Option<T>>` and not `Poll<T>` (which would make more sense, IMO, for the pre-transformed coroutine), is because the `TransformVisitor` that is used to turn coroutines into built-in state machine functions would have to destructure and reconstruct the latter into the former, which requires at least inserting a new basic block (for a `switchInt` terminator, to match on the `Poll` discriminant).
This does mean that the desugaring (at the `rustc_ast_lowering` level) of `async gen` blocks is a bit more involved. However, since we already need to intercept both `.await` and `yield` operators, I don't consider it much of a technical burden.
r? `@ghost`
never_patterns: Parse match arms with no body
Never patterns are meant to signal unreachable cases, and thus don't take bodies:
```rust
let ptr: *const Option<!> = ...;
match *ptr {
None => { foo(); }
Some(!),
}
```
This PR makes rustc accept the above, and enforces that an arm has a body xor is a never pattern. This affects parsing of match arms even with the feature off, so this is delicate. (Plus this is my first non-trivial change to the parser).
~~The last commit is optional; it introduces a bit of churn to allow the new suggestions to be machine-applicable. There may be a better solution? I'm not sure.~~ EDIT: I removed that commit
r? `@compiler-errors`
Add `never_patterns` feature gate
This PR adds the feature gate and most basic parsing for the experimental `never_patterns` feature. See the tracking issue (https://github.com/rust-lang/rust/issues/118155) for details on the experiment.
`@scottmcm` has agreed to be my lang-team liaison for this experiment.
Remove HIR opkinds
`hir::BinOp`, `hir::BinOpKind`, and `hir::UnOp` are identical to `ast::BinOp`, `ast::BinOpKind`, and `ast::UnOp`, respectively. This seems silly, so this PR removes the HIR ones. (A re-export lets the AST ones be referred to using a `hir::` qualifier, which avoids renaming churn.)
r? `@cjgillot`
- Rename them both `as_str`, which is the typical name for a function
that returns a `&str`. (`to_string` is appropriate for functions
returning `String` or maybe `Cow<'a, str>`.)
- Change `UnOp::as_str` from an associated function (weird!) to a
method.
- Avoid needless `self` dereferences.
The AST and HIR versions of `State::print_ident` are textually
identical, but the types differ slightly. This commit factors out the
common code they both have by replacing `print_ident` with `ann_post`,
which is a smaller function that still captures the type difference.
`PrintState` is a trait containing code that can be used by both AST and
HIR pretty-printing. But several of its methods are only used by AST
printing.
This commit moves those methods out of the trait and into the AST
`State` impl, so they are not exposed unnecessarily. This commit also
removes four unused methods: `param_to_string`,
`foreign_item_to_string`, `assoc_item_to_string`, and
`print_inner_attributes_inline`.
To avoid `!matches!(...)`, which is hard to think about. Instead every
case now uses direct pattern matching and returns true or false.
Also add a couple of cases to the `stringify.rs` test that currently
print badly.
- Sort dependencies and features sections.
- Add `tidy` markers to the sorted sections so they stay sorted.
- Remove empty `[lib`] sections.
- Remove "See more keys..." comments.
Excluded files:
- rustc_codegen_{cranelift,gcc}, because they're external.
- rustc_lexer, because it has external use.
- stable_mir, because it has external use.
