Add an InstSimplify for repetitive array expressions
I noticed in https://github.com/rust-lang/rust/pull/135068#issuecomment-2569955426 that GVN's implementation of this same transform was quite profitable on the deep-vector benchmark. But of course GVN doesn't run in unoptimized builds, so this is my attempt to write a version of this transform that benefits the deep-vector case and is fast enough to run in InstSimplify.
The benchmark suite indicates that this is effective.
Adds `#[rustc_force_inline]` which is similar to always inlining but
reports an error if the inlining was not possible, and which always
attempts to inline annotated items, regardless of optimisation levels.
It can only be applied to free functions to guarantee that the MIR
inliner will be able to resolve calls.
We already did `Transmute`-then-`PtrToPtr`; this adds the nearly-identical `PtrToPtr`-then-`Transmute`.
It also adds `transmute(Foo(x))` → `transmute(x)`, when `Foo` is a single-field transparent type. That's useful for things like `NonNull { pointer: p }.as_ptr()`.
Found these as I was looking at MCP807-related changes.
rustc_intrinsic: support functions without body
We synthesize a HIR body `loop {}` but such bodyless intrinsics.
Most of the diff is due to turning `ItemKind::Fn` into a brace (named-field) enum variant, because it carries a `bool`-typed field now. This is to remember whether the function has a body. MIR building panics to avoid ever translating the fake `loop {}` body, and the intrinsic logic uses the lack of a body to implicitly mark that intrinsic as must-be-overridden.
I first tried actually having no body rather than generating the fake body, but there's a *lot* of code that assumes that all function items have HIR and MIR, so this didn't work very well. Then I noticed that even `rustc_intrinsic_must_be_overridden` intrinsics have MIR generated (they are filled with an `Unreachable` terminator) so I guess I am not the first to discover this. ;)
r? `@oli-obk`
Begin to implement type system layer of unsafe binders
Mostly TODOs, but there's a lot of match arms that are basically just noops so I wanted to split these out before I put up the MIR lowering/projection part of this logic.
r? oli-obk
Tracking:
- https://github.com/rust-lang/rust/issues/130516
During coverage instrumentation, this variable always holds the coverage graph,
which is a simplified view of the MIR control-flow graph. The new name is
clearer in context, and also shorter.
coverage: Store coverage source regions as `Span` until codegen (take 2)
This is an attempt to re-land #133418:
> Historically, coverage spans were converted into line/column coordinates during the MIR instrumentation pass.
> This PR moves that conversion step into codegen, so that coverage spans spend most of their time stored as Span instead.
> In addition to being conceptually nicer, this also reduces the size of coverage mappings in MIR, because Span is smaller than 4x u32.
That PR was reverted by #133608, because in some circumstances not covered by our test suite we were emitting coverage metadata that was causing `llvm-cov` to exit with an error (#133606).
---
The implementation here is *mostly* the same, but adapted for subsequent changes in the relevant code (e.g. #134163).
I believe that the changes in #134163 should be sufficient to prevent the problem that required the original PR to be reverted. But I haven't been able to reproduce the original breakage in a regression test, and the `llvm-cov` error message is extremely unhelpful, so I can't completely rule out the possibility of this breaking again.
r? jieyouxu (reviewer of the original PR)
Variants::Single: do not use invalid VariantIdx for uninhabited enums
~~Stacked on top of https://github.com/rust-lang/rust/pull/133681, only the last commit is new.~~
Currently, `Variants::Single` for an empty enum contains a `VariantIdx` of 0; looking that up in the enum variant list will ICE. That's quite confusing. So let's fix that by adding a new `Variants::Empty` case for types that have 0 variants.
try-job: i686-msvc
`rustc_span::symbol` defines some things that are re-exported from
`rustc_span`, such as `Symbol` and `sym`. But it doesn't re-export some
closely related things such as `Ident` and `kw`. So you can do `use
rustc_span::{Symbol, sym}` but you have to do `use
rustc_span::symbol::{Ident, kw}`, which is inconsistent for no good
reason.
This commit re-exports `Ident`, `kw`, and `MacroRulesNormalizedIdent`,
and changes many `rustc_span::symbol::` qualifiers in `compiler/` to
`rustc_span::`. This is a 200+ net line of code reduction, mostly
because many files with two `use rustc_span` items can be reduced to
one.
Bounds-check with PtrMetadata instead of Len in MIR
Rather than emitting `Len(*_n)` in array index bounds checks, emit `PtrMetadata(copy _n)` instead -- with some asterisks for arrays and `&mut` that need it to be done slightly differently.
We're getting pretty close to removing `Len` entirely, actually. I think just one more PR after this (for slice drop shims).
r? mir
We don't need `NonNull::as_ptr` debuginfo
In order to stop pessimizing the use of local variables in core, skip debug info for MIR temporaries in tiny (single-BB) functions.
For functions as simple as this -- `Pin::new`, etc -- nobody every actually wants debuginfo for them in the first place. They're more like intrinsics than real functions, and stepping over them is good.
Stop pessimizing the use of local variables in core by skipping debug info for MIR temporaries in tiny (single-BB) functions.
For functions as simple as this -- `Pin::new`, etc -- nobody every actually wants debuginfo for them in the first place. They're more like intrinsics than real functions, and stepping over them is good.
coverage: Use a query to find counters/expressions that must be zero
As of #133446, this query (`coverage_ids_info`) determines which counter/expression IDs are unused. So with only a little extra work, we can take the code that was using that information to determine which coverage counters/expressions must be zero, and move that inside the query as well.
There should be no change in compiler output.
coverage: Prefer to visit nodes whose predecessors have been visited
In coverage instrumentation, we need to traverse the control-flow graph and decide what kind of counter (physical counter or counter-expression) should be used for each node that needs a counter.
The existing traversal order is complex and hard to tweak. This new traversal order tries to be a bit more principled, by always preferring to visit nodes whose predecessors have already been visited, which is a good match for how the counter-creation code ends up dealing with a node's in-edges and out-edges.
For several of the coverage tests, this ends up being a strict improvement in reducing the size of the coverage metadata, and also reducing the number of physical counters needed.
(The new traversal should hopefully also allow some further code simplifications in the future.)
---
This is made possible by the separate simplification pass introduced by #133849. Without that, almost any change to the traversal order ends up increasing the size of the expression table or the number of physical counters.
The words "before" and "after" have an obvious temporal meaning, e.g.
`seek_before_primary_effect`,
`visit_statement_{before,after}_primary_effect`. But "before" is also
used to name the effect that occurs before the primary effect of a
statement/terminator; this is `Effect::Before`. This leads to the
confusing possibility of talking about things happening "before/after
the before event".
This commit removes this awkward overloading of "before" by renaming
`Effect::Before` as `Effect::Early`. It also renames some of the
`Analysis` and `ResultsVisitor` methods to be more consistent.
Here are the before and after names:
- `Effect::{Before,Primary}` -> `Effect::{Early,Primary}`
- `apply_before_statement_effect` -> `apply_early_statement_effect`
- `apply_statement_effect` -> `apply_primary_statement_effect`
- `visit_statement_before_primary_effect` -> `visit_after_early_statement_effect`
- `visit_statement_after_primary_effect` -> `visit_after_primary_statement_effect`
(And s/statement/terminator/ for all the terminator events.)
It uses `MaybeInitializedPlaces` and `MaybeUninitializedPlaces`, but
calls the results `live` and `dead`. This is very confusing given that
there are also analyses called `MaybeLiveLocals` and `MaybeStorageLive`
and `MaybeStorageDead`.
This commit changes it to use `maybe_init` and `maybe_uninit`.
Introduce `MixedBitSet`
`ChunkedBitSet` is good at avoiding excessive memory usage for programs with very large functgions where dataflow bitsets have very large domain sizes. But it's overly heavyweight for small bitsets, because any non-empty `ChunkedBitSet` takes up at least 256 bytes.
This PR introduces `MixedBitSet`, which is a simple bitset that uses `BitSet` for small/medium bitsets and `ChunkedBitSet` for large bitsets. It's a speed and memory usage win.
r? `@Mark-Simulacrum`
This query (`coverage_ids_info`) already determines which counter/expression
IDs are unused, so it only takes a little extra effort to also determine which
counters/expressions must have a value of zero.
It's a performance win because `MixedBitSet` is faster and uses less
memory than `ChunkedBitSet`.
Also reflow some overlong comment lines in
`lint_tail_expr_drop_order.rs`.
remove `Ty::is_copy_modulo_regions`
Using these functions is likely incorrect if an `InferCtxt` is available, I moved this function to `TyCtxt` (and added it to `LateContext`) and added a note to the documentation that one should prefer `Infer::type_is_copy_modulo_regions` instead.
I didn't yet move `is_sized` and `is_freeze`, though I think we should move these as well.
r? `@compiler-errors` cc #132279
The `Borrowed` variant is no longer used. This commit removes it, along
with the `as_results_cursor` method that produces it, and renames
`as_results_cursor_mut` as `as_results_cursor`.
coverage: Store coverage source regions as `Span` until codegen
Historically, coverage spans were converted into line/column coordinates during the MIR instrumentation pass.
This PR moves that conversion step into codegen, so that coverage spans spend most of their time stored as `Span` instead.
In addition to being conceptually nicer, this also reduces the size of coverage mappings in MIR, because `Span` is smaller than 4x u32.
---
There should be no changes to coverage output.
Remove -Zfuel.
I'm not sure this feature is used. I only found 2 references in a google search, both referring to its introduction.
Meanwhile, it's a global mutable state, untracked by incremental compilation, so incompatible with it.
`ResultsCursor` currently owns its `Results`. But sometimes the
`Results` is needed again afterwards. So there is
`ResultsCursor::into_results` for extracting the `Results`, which leads
to some awkwardness.
This commit adds `ResultsHandle`, a `Cow`-like type that can either
borrow or own a a `Results`. `ResultsCursor` now uses it. This is good
because some `ResultsCursor`s really want to own their `Results`, while
others just want to borrow it.
We end with with a few more lines of code, but get some nice cleanups.
- `ResultsCursor::into_results` and `Formatter::into_results` are
removed.
- `write_graphviz_results` now just borrows a `Results`, instead of the
awkward "take ownership of a `Results` and then return it unchanged"
pattern.
This reinstates the cursor flexibility that was lost in #118230 -- which
removed the old `ResultsRefCursor` and `ResultsCloneCursor` types -- but
in a much simpler way. Hooray!
Reduce false positives of tail-expr-drop-order from consumed values (attempt #2)
r? `@nikomatsakis`
Tracked by #123739.
Related to #129864 but not replacing, yet.
Related to #130836.
This is an implementation of the approach suggested in the [Zulip stream](https://rust-lang.zulipchat.com/#narrow/stream/213817-t-lang/topic/temporary.20drop.20order.20changes). A new MIR statement `BackwardsIncompatibleDrop` is added to the MIR syntax. The lint now works by inspecting possibly live move paths before at the `BackwardsIncompatibleDrop` location and the actual drop under the current edition, which should be one before Edition 2024 in practice.
take 2
open up coroutines
tweak the wordings
the lint works up until 2021
We were missing one case, for ADTs, which was
causing `Result` to yield incorrect results.
only include field spans with significant types
deduplicate and eliminate field spans
switch to emit spans to impl Drops
Co-authored-by: Niko Matsakis <nikomat@amazon.com>
collect drops instead of taking liveness diff
apply some suggestions and add explantory notes
small fix on the cache
let the query recurse through coroutine
new suggestion format with extracted variable name
fine-tune the drop span and messages
bugfix on runtime borrows
tweak message wording
filter out ecosystem types earlier
apply suggestions
clippy
check lint level at session level
further restrict applicability of the lint
translate bid into nop for stable mir
detect cycle in type structure
the behavior of the type system not only depends on the current
assumptions, but also the currentnphase of the compiler. This is
mostly necessary as we need to decide whether and how to reveal
opaque types. We track this via the `TypingMode`.
There are a handful of tier 2 and tier 3 targets that cause a LLVM crash
or linker error when generating code that contains `f16` or `f128`. The
cranelift backend also does not support these types. To work around
this, every function in `std` or `core` that contains these types must
be marked `#[inline]` in order to avoid sending any code to the backend
unless specifically requested.
However, this is inconvenient and easy to forget. Introduce a check for
these types in the frontend that automatically inlines any function
signatures that take or return `f16` or `f128`.
Note that this is not a perfect fix because it does not account for the
types being passed by reference or as members of aggregate types, but
this is sufficient for what is currently needed in the standard library.
Fixes: https://github.com/rust-lang/rust/issues/133035
Closes: https://github.com/rust-lang/rust/pull/133037
coverage: Restrict empty-span expansion to only cover `{` and `}`
Coverage instrumentation has some tricky code for converting a coverage-relevant `Span` into a set of start/end line/byte-column coordinates that will be embedded in the CGU's coverage metadata.
A big part of this complexity is special code for handling empty spans, which are expanded into non-empty spans (if possible) because LLVM's coverage reporter does not handle empty spans well.
This PR simplifies that code by restricting it to only apply in two specific situations: when the character after the empty span is `{`, or the character before the empty span is `}`.
(As an added benefit, this means that the expanded spans no longer extend awkwardly beyond the end of a physical line, which was common under the previous implementation.)
Along the way, this PR also removes some unhelpful code for dealing with function source code spread across multiple files. Functions currently can't have coverage spans in multiple files, and if that ever changes (e.g. to properly support expansion regions) then this code will need to be completely overhauled anyway.
Functions currently can't have mappings in multiple files, and if that ever
changes (e.g. to properly support expansion regions), this code will need to be
completely overhauled anyway.
