These are all functions with a single callsite, where having a separate
function does nothing to help with readability. These changes make the
code a little shorter and easier to read.
There are four related dataflow structs: `MaybeInitializedPlaces`,
`MaybeUninitializedPlaces`, and `EverInitializedPlaces`,
`DefinitelyInitializedPlaces`. They all have a `&Body` and a
`&MoveData<'tcx>` field. The first three use different lifetimes for the
two fields, but the last one uses the same lifetime for both.
This commit changes the first three to use the same lifetime, removing
the need for one of the lifetimes. Other structs that also lose a
lifetime as a result of this are `LivenessContext`, `LivenessResults`,
`InitializationData`.
It then does similar things in various other structs.
Currently it constructs two vectors `calls_to_terminated` and
`cleanups_to_remove` in the main loop, and then processes them after the
main loop. But the processing can be done in the main loop, avoiding the
need for the vectors.
Do not call query to compute coroutine layout for synthetic body of async closure
There is code in the MIR validator that attempts to prevent query cycles when inlining a coroutine into itself, and will use the coroutine layout directly from the body when it detects that's the same coroutine as the one that's being validated. After #128506, this logic didn't take into account the fact that the coroutine def id will differ if it's the "by-move body" of an async closure. This PR implements that.
Fixes#129811
coverage: Count await when the Future is immediately ready
Currently `await` is only counted towards coverage if the containing
function is suspended and resumed at least once. This is because it
expands to code which contains a branch on the discriminant of `Poll`.
By treating it like a branching macro (e.g. `assert!`), these
implementation details will be hidden from the coverage results.
I added a test to ensure the fix works in simple cases, but the heuristic of picking only the first await-related covspan might be unreliable. I plan on testing more thoroughly with a real codebase over the next couple of weeks.
closes#98712
Make `Ty::boxed_ty` return an `Option`
Looks like a good place to use Rust's type system.
---
Most of 4ac7bcbaad/compiler/rustc_middle/src/ty/sty.rs (L971-L1963) looks like it could be moved to `TyKind` (then I guess `Ty` should be made to deref to `TyKind`).
Currently `await` is only counted towards coverage if the containing
function is suspended and resumed at least once. This is because it
expands to code which contains a branch on the discriminant of `Poll`.
By treating it like a branching macro (e.g. `assert!`), these
implementation details will be hidden from the coverage results.
Rename dump of coroutine by-move-body to be more consistent, fix ICE in dump_mir
First, we add a missing match for `DefKind::SyntheticCoroutineBody` in `dump_mir`. Fixes#129703. The second commit (directly below) serves as a test.
Second, we reorder the `dump_mir` in `coroutine_by_move_body_def_id` to be *after* we adjust the body source, and change the disambiguator so it reads more like any other MIR body. This also serves as a test for the ICE, since we're dumping the MIR of a body with `DefKind::SyntheticCoroutineBody`.
Third, we change the parenting of the synthetic MIR body to have the *coroutine-closure* (i.e. async closure) as its parent, so we don't have long strings of `{closure#0}-{closure#0}-{closure#0}`.
try-job: test-various
Move `SanityCheck` and `MirPass`
They are currently in `rustc_middle`. This PR moves them to `rustc_mir_transform`, which makes more sense.
r? ``@cjgillot``
Because that's now the only crate that uses it.
Moving stuff out of `rustc_middle` is always welcome.
I chose to use `impl crate::MirPass`/`impl crate::MirLint` (with
explicit `crate::`) everywhere because that's the only mention of
`MirPass`/`MirLint` used in all of these files. (Prior to this change,
`MirPass` was mostly imported via `use rustc_middle::mir::*` items.)
The actual implementation remains in `rustc_mir_dataflow`, but this
commit moves the `MirPass` impl to `rustc_mir_transform` and changes it
to a `MirLint` (fixing a `FIXME` comment).
(I originally tried moving the full implementation from
`rustc_mir_dataflow` but I had some trait problems with `HasMoveData`
and `RustcPeekAt` and `MaybeLiveLocals`. This commit was much smaller
and simpler, but still will allow some follow-up cleanups.)
Remove `#[macro_use] extern crate tracing`, round 4
Because explicit importing of macros via use items is nicer (more standard and readable) than implicit importing via #[macro_use]. Continuing the work from #124511, #124914, and #125434. After this PR no `rustc_*` crates use `#[macro_use] extern crate tracing` except for `rustc_codegen_gcc` which is a special case and I will do separately.
r? ```@jieyouxu```
Remove `Option<!>` return types.
Several compiler functions have `Option<!>` for their return type. That's odd. The only valid return value is `None`, so why is this type used?
Because it lets you write certain patterns slightly more concisely. E.g. if you have these common patterns:
```
let Some(a) = f() else { return };
let Ok(b) = g() else { return };
```
you can shorten them to these:
```
let a = f()?;
let b = g().ok()?;
```
Huh.
An `Option` return type typically designates success/failure. How should I interpret the type signature of a function that always returns (i.e. doesn't panic), does useful work (modifying `&mut` arguments), and yet only ever fails? This idiom subverts the type system for a cute syntactic trick.
Furthermore, returning `Option<!>` from a function F makes things syntactically more convenient within F, but makes things worse at F's callsites. The callsites can themselves use `?` with F but should not, because they will get an unconditional early return, which is almost certainly not desirable. Instead the return value should be ignored. (Note that some of callsites of `process_operand`, `process_immedate`, `process_assign` actually do use `?`, though the early return doesn't matter in these cases because nothing of significance comes after those calls. Ugh.)
When I first saw this pattern I had no idea how to interpret it, and it took me several minutes of close reading to understand everything I've written above. I even started a Zulip thread about it to make sure I understood it properly. "Save a few characters by introducing types so weird that compiler devs have to discuss it on Zulip" feels like a bad trade-off to me. This commit replaces all the `Option<!>` return values and uses `else`/`return` (or something similar) to replace the relevant `?` uses. The result is slightly more verbose but much easier to understand.
r? ``````@cjgillot``````
Several compiler functions have `Option<!>` for their return type.
That's odd. The only valid return value is `None`, so why is this type
used?
Because it lets you write certain patterns slightly more concisely. E.g.
if you have these common patterns:
```
let Some(a) = f() else { return };
let Ok(b) = g() else { return };
```
you can shorten them to these:
```
let a = f()?;
let b = g().ok()?;
```
Huh.
An `Option` return type typically designates success/failure. How should
I interpret the type signature of a function that always returns (i.e.
doesn't panic), does useful work (modifying `&mut` arguments), and yet
only ever fails? This idiom subverts the type system for a cute
syntactic trick.
Furthermore, returning `Option<!>` from a function F makes things
syntactically more convenient within F, but makes things worse at F's
callsites. The callsites can themselves use `?` with F but should not,
because they will get an unconditional early return, which is almost
certainly not desirable. Instead the return value should be ignored.
(Note that some of callsites of `process_operand`, `process_immedate`,
`process_assign` actually do use `?`, though the early return doesn't
matter in these cases because nothing of significance comes after those
calls. Ugh.)
When I first saw this pattern I had no idea how to interpret it, and it
took me several minutes of close reading to understand everything I've
written above. I even started a Zulip thread about it to make sure I
understood it properly. "Save a few characters by introducing types so
weird that compiler devs have to discuss it on Zulip" feels like a bad
trade-off to me. This commit replaces all the `Option<!>` return values
and uses `else`/`return` (or something similar) to replace the relevant
`?` uses. The result is slightly more verbose but much easier to
understand.
By making it own the index maps, instead of holding references to them.
This requires moving the free function `find_candidate` into
`Candidate::reset_and_find`. It lets the `'alloc` lifetime be removed
everywhere that still has it.
LLVM uses the word "code" to refer to a particular kind of coverage mapping.
This unrelated usage of the word is confusing, and makes it harder to introduce
types whose names correspond to the LLVM classification of coverage kinds.
When deduplicating unreachable blocks, erase the source information.
After deduplication the block conceptually belongs to multiple locations in the source. Although these blocks are unreachable, in #123341 we did come across a real side effect, an unreachable block that survives into the compiled code can cause a debugger to set a breakpoint on the wrong instruction. Erasing the source information ensures that a debugger will never be misled into thinking that the unreachable block is worth setting a breakpoint on, especially after #128627.
Technically we don't need to erase the source information if all the deduplicated blocks have identical source information, but tracking that seems like more effort than it's worth.
I'll let njn redirect this one too. r? `@nnethercote`
Fix projections when parent capture is by-ref but child capture is by-value in the `ByMoveBody` pass
This fixes a somewhat strange bug where we build the incorrect MIR in #129074. This one is weird, but I don't expect it to actually matter in practice since it almost certainly results in a move error in borrowck. However, let's not ICE.
Given the code:
```
#![feature(async_closure)]
// NOT copy.
struct Ty;
fn hello(x: &Ty) {
let c = async || {
*x;
//~^ ERROR cannot move out of `*x` which is behind a shared reference
};
}
fn main() {}
```
The parent coroutine-closure captures `x: &Ty` by-ref, resulting in an upvar of `&&Ty`. The child coroutine captures `x` by-value, resulting in an upvar of `&Ty`. When constructing the by-move body for the coroutine-closure, we weren't applying an additional deref projection to convert the parent capture into the child capture, resulting in an type error in assignment, which is a validation ICE.
As I said above, this only occurs (AFAICT) in code that eventually results in an error, because it is only triggered by HIR that attempts to move a non-copy value out of a ref. This doesn't occur if `Ty` is `Copy`, since we'd instead capture `x` by-ref in the child coroutine.
Fixes#129074
Use `append` instead of `extend(drain(..))`
The first commit adds `IndexVec::append` that forwards to `Vec::append`, and uses it in a couple places.
The second commit updates `indexmap` for its new `IndexMap::append`, and also uses that in a couple places.
These changes are similar to what [`clippy::extend_with_drain`](https://rust-lang.github.io/rust-clippy/master/index.html#/extend_with_drain) would suggest, just for other collection types.
Shrink `TyKind::FnPtr`.
By splitting the `FnSig` within `TyKind::FnPtr` into `FnSigTys` and `FnHeader`, which can be packed more efficiently. This reduces the size of the hot `TyKind` type from 32 bytes to 24 bytes on 64-bit platforms. This reduces peak memory usage by a few percent on some benchmarks. It also reduces cache misses and page faults similarly, though this doesn't translate to clear cycles or wall-time improvements on CI.
r? `@compiler-errors`
Use more slice patterns inside the compiler
Nothing super noteworthy. Just replacing the common 'fragile' pattern of "length check followed by indexing or unwrap" with slice patterns for legibility and 'robustness'.
r? ghost
Fix `ElaborateBoxDerefs` on debug varinfo
Slightly simplifies the `ElaborateBoxDerefs` pass to fix cases where it was applying the wrong projections to debug var infos containing places that deref boxes.
From what I can tell[^1], we don't actually have any tests (or code anywhere, really) that exercise `debug x => *(...: Box<T>)`, and it's very difficult to trigger this in surface Rust, so I wrote a custom MIR test.
What happens is that the pass was turning `*(SOME_PLACE: Box<T>)` into `*(*((((SOME_PLACE).0: Unique<T>).0: NonNull<T>).0: *const T))` in debug var infos. In particular, notice the *double deref*, which was wrong.
This is the root cause of #128554, so this PR fixes#128554 as well. The reason that async closures was affected is because of the way that we compute the [`ByMove` body](https://github.com/rust-lang/rust/blob/master/compiler/rustc_mir_transform/src/coroutine/by_move_body.rs), which resulted in `*(...: Box<T>)` in debug var info. But this really has nothing to do with async closures.
[^1]: Validated by literally replacing the `if elem == PlaceElem::Deref && base_ty.is_box() { ... }` innards with a `panic!()`, which compiled all of stage2 without panicking.
By splitting the `FnSig` within `TyKind::FnPtr` into `FnSigTys` and
`FnHeader`, which can be packed more efficiently. This reduces the size
of the hot `TyKind` type from 32 bytes to 24 bytes on 64-bit platforms.
This reduces peak memory usage by a few percent on some benchmarks. It
also reduces cache misses and page faults similarly, though this doesn't
translate to clear cycles or wall-time improvements on CI.
After deduplication the block conceptually belongs to multiple locations
in the source. Although these blocks are unreachable, in #123341 we did
come across a real side effect, an unreachable block that survives into
the compiled code can cause a debugger to set a breakpoint on the wrong
instruction. Erasing the source information ensures that a debugger will
never be misled into thinking that the unreachable block is worth setting
a breakpoint on, especially after #128627.
Technically we don't need to erase the source information if all the
deduplicated blocks have identical source information, but tracking
that seems like more effort than it's worth.
Jump threading stores values as `u128` (`ScalarInt`) and does its
comparisons for equality as integer comparisons.
This works great for integers. Sadly, not everything is an integer.
Floats famously have wonky equality semantcs, with `NaN!=NaN` and
`0.0 == -0.0`. This does not match our beautiful integer bitpattern
equality and therefore causes things to go horribly wrong.
While jump threading could be extended to support floats by remembering
that they're floats in the value state and handling them properly,
it's signficantly easier to just disable it for now.
Let InstCombine remove Clone shims inside Clone shims
The Clone shims that we generate tend to recurse into other Clone shims, which gets very silly very quickly. Here's our current state: https://godbolt.org/z/E69YeY8eq
So I've added InstSimplify to the shims optimization passes, and improved `is_trivially_pure_clone_copy` so that it can delete those calls inside the shim. This makes the shim way smaller because most of its size is the required ceremony for unwinding.
This change also completely breaks the UI test added for https://github.com/rust-lang/rust/issues/104870. With this PR, that program ICEs in MIR type checking because `is_trivially_pure_clone_copy` and the trait solver disagree on whether `*mut u8` is `Copy`. And adding the requisite `Copy` impl to make them agree makes the test not generate any diagnostics. Considering that I spent most of my time on this PR fixing `#![no_core]` tests, I would prefer to just delete this one. The maintenance burden of `#![no_core]` is uniquely high because when they break they tend to break in very confusing ways.
try-job: x86_64-mingw
Make Clone::clone a lang item
I want to absorb all the logic for picking whether an Instance is LocalCopy or GloballyShared into one place. As part of this, I wanted to identify Clone shims inside `cross_crate_inlinable` and found that rather tricky. `@compiler-errors` suggested that I add a lang item for `Clone::clone` because that would produce other cleanups in the compiler.
That sounds good to me, but I have looked and I've only been able to find one.
r? compiler-errors
Clean up a few minor refs in `format!` macro, as it has a performance cost. Apparently the compiler is unable to inline `format!("{}", &variable)`, and does a run-time double-reference instead (format macro already does one level referencing). Inlining format args prevents accidental `&` misuse.
In the future, branch and MC/DC mappings might have expressions that don't
correspond to any single point in the control-flow graph. That makes it
trickier to keep track of which expressions should expect an `ExpressionUsed`
node.
We therefore sidestep that complexity by only performing `ExpressionUsed`
simplification for expressions associated directly with ordinary `Code`
mappings.
[Coverage][MCDC] Group mcdc tests and fix panic when generating mcdc code for inlined expressions.
### Changes
1. Group all mcdc tests to one directory.
2. Since mcdc instruments different mappings for boolean expressions with normal branch coverage as #125766 introduces, it would be better also trace branch coverage results in mcdc tests.
3. So far rustc does not call `CoverageInfoBuilderMethods::init_coverage` for inlined functions. As a result, it could panic if it tries to instrument mcdc statements for inlined functions due to uninitialized cond bitmaps. We can reproduce this issue by current nightly rustc and [the test](https://github.com/rust-lang/rust/pull/127234/files#diff-c81af6bf4869aa42f5c7334e3e86344475de362f673f54ce439ec75fcb5ac3e5) with flag `--release`. This patch fixes it.
Support tail calls in mir via `TerminatorKind::TailCall`
This is one of the interesting bits in tail call implementation — MIR support.
This adds a new `TerminatorKind` which represents a tail call:
```rust
TailCall {
func: Operand<'tcx>,
args: Vec<Operand<'tcx>>,
fn_span: Span,
},
```
*Structurally* this is very similar to a normal `Call` but is missing a few fields:
- `destination` — tail calls don't write to destination, instead they pass caller's destination to the callee (such that eventual `return` will write to the caller of the function that used tail call)
- `target` — similarly to `destination` tail calls pass the caller's return address to the callee, so there is nothing to do
- `unwind` — I _think_ this is applicable too, although it's a bit confusing
- `call_source` — `become` forbids operators and is not created as a lowering of something else; tail calls always come from HIR (at least for now)
It might be helpful to read the interpreter implementation to understand what `TailCall` means exactly, although I've tried documenting it too.
-----
There are a few `FIXME`-questions still left, ideally we'd be able to answer them during review ':)
-----
r? `@oli-obk`
cc `@scottmcm` `@DrMeepster` `@JakobDegen`
Make jump threading state sparse
Continuation of https://github.com/rust-lang/rust/pull/127024
Both dataflow const-prop and jump threading involve cloning the state vector a lot. This PR replaces the data structure by a sparse vector, considering:
- that jump threading state is typically very sparse (at most 1 or 2 set entries);
- that dataflow const-prop is disabled by default;
- that place/value map is very eager, and prone to creating an overly large state.
The first commit is shared with the previous PR to avoid needless conflicts.
r? `@oli-obk`
Re-implement a type-size based limit
r? lcnr
This PR reintroduces the type length limit added in #37789, which was accidentally made practically useless by the caching changes to `Ty::walk` in #72412, which caused the `walk` function to no longer walk over identical elements.
Hitting this length limit is not fatal unless we are in codegen -- so it shouldn't affect passes like the mir inliner which creates potentially very large types (which we observed, for example, when the new trait solver compiles `itertools` in `--release` mode).
This also increases the type length limit from `1048576 == 2 ** 20` to `2 ** 24`, which covers all of the code that can be reached with craterbot-check. Individual crates can increase the length limit further if desired.
Perf regression is mild and I think we should accept it -- reinstating this limit is important for the new trait solver and to make sure we don't accidentally hit more type-size related regressions in the future.
Fixes#125460
Fix `FnMut::call_mut`/`Fn::call` shim for async closures that capture references
I adjusted async closures to be able to implement `Fn` and `FnMut` *even if* they capture references, as long as those references did not need to borrow data from the closure captures themselves. See #125259.
However, when I did this, I didn't actually relax an assertion in the `build_construct_coroutine_by_move_shim` shim code, which builds the `Fn`/`FnMut`/`FnOnce` implementations for async closures. Therefore, if we actually tried to *call* `FnMut`/`Fn` on async closures, it would ICE.
This PR adjusts this assertion to ensure that we only capture immutable references in closures if they implement `Fn`/`FnMut`. It also adds a bunch of tests and makes more of the async-closure tests into `build-pass` since we often care about these tests actually generating the right closure shims and stuff. I think it might be excessive to *always* use build-pass here, but 🤷 it's not that big of a deal.
Fixes#127019Fixes#127012
r? oli-obk
In 126578 we ended up with more binary size increases than expected.
This change attempts to avoid inlining large things into small things, to avoid that kind of increase, in cases when top-down inlining will still be able to do that inlining later.
Rollup of 7 pull requests
Successful merges:
- #126923 (test: dont optimize to invalid bitcasts)
- #127090 (Reduce merge conflicts from rustfmt's wrapping)
- #127105 (Only update `Eq` operands in GVN if it can update both sides)
- #127150 (Fix x86_64 code being produced for bare-metal LoongArch targets' `compiler_builtins`)
- #127181 (Introduce a `rustc_` attribute to dump all the `DefId` parents of a `DefId`)
- #127182 (Fix error in documentation for IpAddr::to_canonical and Ipv6Addr::to_canonical)
- #127191 (Ensure `out_of_scope_macro_calls` lint is registered)
r? `@ghost`
`@rustbot` modify labels: rollup
Automatically taint InferCtxt when errors are emitted
r? `@nnethercote`
Basically `InferCtxt::dcx` now returns a `DiagCtxt` that refers back to the `Cell<Option<ErrorGuaranteed>>` of the `InferCtxt` and thus when invoking `Diag::emit`, and the diagnostic is an error, we taint the `InferCtxt` directly.
That change on its own has no effect at all, because `InferCtxt` already tracks whether errors have been emitted by recording the global error count when it gets opened, and checking at the end whether the count changed. So I removed that error count check, which had a bit of fallout that I immediately fixed by invoking `InferCtxt::dcx` instead of `TyCtxt::dcx` in a bunch of places.
The remaining new errors are because an error was reported in another query, and never bubbled up. I think they are minor enough for this to be ok, and sometimes it actually improves diagnostics, by not silencing useful diagnostics anymore.
fixes#126485 (cc `@olafes)`
There are more improvements we can do (like tainting in hir ty lowering), but I would rather do that in follow up PRs, because it requires some refactorings.
coverage: Avoid getting extra unexpansion info when we don't need it
Several callers of `unexpand_into_body_span_with_visible_macro` would immediately discard the additional macro-related information, which is wasteful. We can avoid this by having them instead call a simpler method that just returns the span they care about.
This PR also moves the relevant functions out of `coverage::spans::from_mir` and into a new submodule `coverage::unexpand`, so that calling them from `coverage::mappings` is less awkward.
There should be no actual changes to coverage-instrumentation output, as demonstrated by the absence of test updates.
Avoid cloning jump threading state when possible
The current implementation of jump threading passes most of its time cloning its state. This PR attempts to avoid such clones by special-casing the last predecessor when recursing through a terminator.
This is not optimal, but a first step while I refactor the state data structure to be sparse.
The two other commits are drive-by.
Fixes https://github.com/rust-lang/rust/issues/116721
r? `@oli-obk`
These particular callers don't actually use the returned macro information, so
they can use a simpler span-unexpansion function that doesn't return it.
