For structs that cannot be unsized, the layout algorithm sometimes moves
unsized fields to the end of the struct, which circumvented the error
for unexpected unsized fields and returned an unsized layout anyway.
This commit makes it so that the unexpected unsized error is always
returned for structs that cannot be unsized, allowing us to remove an
old hack and fixing some old ICE.
- Replace non-standard names like 's, 'p, 'rg, 'ck, 'parent, 'this, and
'me with vanilla 'a. These are cases where the original name isn't
really any more informative than 'a.
- Replace names like 'cx, 'mir, and 'body with vanilla 'a when the lifetime
applies to multiple fields and so the original lifetime name isn't
really accurate.
- Put 'tcx last in lifetime lists, and 'a before 'b.
Supress niches in coroutines to avoid aliasing violations
As mentioned [here](https://github.com/rust-lang/rust/issues/63818#issuecomment-2264915918), using niches in fields of coroutines that are referenced by other fields is unsound: the discriminant accesses violate the aliasing requirements of the reference pointing to the relevant field. This issue causes [Miri errors in practice](https://github.com/rust-lang/miri/issues/3780).
The "obvious" fix for this is to suppress niches in coroutines. That's what this PR does. However, we have several tests explicitly ensuring that we *do* use niches in coroutines. So I see two options:
- We guard this behavior behind a `-Z` flag (that Miri will set by default). There is no known case of these aliasing violations causing miscompilations. But absence of evidence is not evidence of absence...
- (What this PR does right now.) We temporarily adjust the coroutine layout logic and the associated tests until the proper fix lands. The "proper fix" here is to wrap fields that other fields can point to in [`UnsafePinned`](https://github.com/rust-lang/rust/issues/125735) and make `UnsafePinned` suppress niches; that would then still permit using niches of *other* fields (those that never get borrowed). However, I know that coroutine sizes are already a problem, so I am not sure if this temporary size regression is acceptable.
`@compiler-errors` any opinion? Also who else should be Cc'd here?
Stop storing a special inner body for the coroutine by-move body for async closures
...and instead, just synthesize an item which is treated mostly normally by the MIR pipeline.
This PR does a few things:
* We synthesize a new `DefId` for the by-move body of a closure, which has its `mir_built` fed with the output of the `ByMoveBody` MIR transformation, and some other relevant queries.
* This has the `DefKind::ByMoveBody`, which we use to distinguish it from "real" bodies (that come from HIR) which need to be borrowck'd. Introduce `TyCtxt::is_synthetic_mir` to skip over `mir_borrowck` which is called by `mir_promoted`; borrowck isn't really possible to make work ATM since it heavily relies being called on a body generated from HIR, and is redundant by the construction of the by-move-body.
* Remove the special `PassManager` hacks for handling the inner `by_move_body` stored within the coroutine's mir body. Instead, this body is fed like a regular MIR body, so it's goes through all of the `tcx.*_mir` stages normally (build -> promoted -> ...etc... -> optimized) ✨.
* Remove the `InstanceKind::ByMoveBody` shim, since now we have a "regular" def id, we can just use `InstanceKind::Item`. This also allows us to remove the corresponding hacks from codegen, such as in `fn_sig_for_fn_abi` ✨.
Notable remarks:
* ~~I know it's kind of weird to be using `DefKind::Closure` here, since it's not a distinct closure but just a new MIR body. I don't believe it really matters, but I could also use a different `DefKind`... maybe one that we could use for synthetic MIR bodies in general?~~ edit: We're doing this now.
Document & implement the transmutation modeled by `BikeshedIntrinsicFrom`
Documents that `BikeshedIntrinsicFrom` models transmute-via-union, which is slightly more expressive than the transmute-via-cast implemented by `transmute_copy`. Additionally, we provide an implementation of transmute-via-union as a method on the `BikeshedIntrinsicFrom` trait with additional documentation on the boundary between trait invariants and caller obligations.
Whether or not transmute-via-union is the right kind of transmute to model remains up for discussion [1]. Regardless, it seems wise to document the present behavior.
[1] https://rust-lang.zulipchat.com/#narrow/stream/216762-project-safe-transmute/topic/What.20'kind'.20of.20transmute.20to.20model.3F/near/426331967
Tracking Issue: https://github.com/rust-lang/rust/issues/99571
r? `@compiler-errors`
cc `@scottmcm,` `@Lokathor`
Documents that `BikeshedIntrinsicFrom` models transmute-via-union,
which is slightly more expressive than the transmute-via-cast
implemented by `transmute_copy`. Additionally, we provide an
implementation of transmute-via-union as a method on the
`BikeshedIntrinsicFrom` trait with additional documentation on
the boundary between trait invariants and caller obligations.
Whether or not transmute-via-union is the right kind of transmute
to model remains up for discussion [1]. Regardless, it seems wise
to document the present behavior.
[1] https://rust-lang.zulipchat.com/#narrow/stream/216762-project-safe-transmute/topic/What.20'kind'.20of.20transmute.20to.20model.3F/near/426331967
Stabilize `raw_ref_op` (RFC 2582)
This stabilizes the syntax `&raw const $expr` and `&raw mut $expr`. It has existed unstably for ~4 years now, and has been exposed on stable via the `addr_of` and `addr_of_mut` macros since Rust 1.51 (released more than 3 years ago). I think it has become clear that these operations are here to stay. So it is about time we give them proper primitive syntax. This has two advantages over the macro:
- Being macros, `addr_of`/`addr_of_mut` could in theory do arbitrary magic with the expression on which they work. The only "magic" they actually do is using the argument as a place expression rather than as a value expression. Place expressions are already a subtle topic and poorly understood by many programmers; having this hidden behind a macro using unstable language features makes this even worse. Conversely, people do have an idea of what happens below `&`/`&mut`, so we can make the subtle topic a lot more approachable by connecting to existing intuition.
- The name `addr_of` is quite unfortunate from today's perspective, given that we have accepted provenance as a reality, which means that a pointer is *not* just an address. Strict provenance has a method, `addr`, which extracts the address of a pointer; using the term `addr` in two different ways is quite unfortunate. That's why this PR soft-deprecates `addr_of` -- we will wait a long time before actually showing any warning here, but we should start telling people that the "addr" part of this name is somewhat misleading, and `&raw` avoids that potential confusion.
In summary, this syntax improves developers' ability to conceptualize the operational semantics of Rust, while making a fundamental operation frequently used in unsafe code feel properly built in.
Possible questions to consider, based on the RFC and [this](https://github.com/rust-lang/rust/issues/64490#issuecomment-1163802912) great summary by `@CAD97:`
- Some questions are entirely about the semantics. The semantics are the same as with the macros so I don't think this should have any impact on this syntax PR. Still, for completeness' sake:
- Should `&raw const *mut_ref` give a read-only pointer?
- Tracked at: https://github.com/rust-lang/unsafe-code-guidelines/issues/257
- I think ideally the answer is "no". Stacked Borrows says that pointer is read-only, but Tree Borrows says it is mutable.
- What exactly does `&raw const (*ptr).field` require? Answered in [the reference](https://doc.rust-lang.org/nightly/reference/behavior-considered-undefined.html): the arithmetic to compute the field offset follows the rules of `ptr::offset`, making it UB if it goes out-of-bounds. Making this a safe operation (using `wrapping_offset` rules) is considered too much of a loss for alias analysis.
- Choose a different syntax? I don't want to re-litigate the RFC. The only credible alternative that has been proposed is `&raw $place` instead of `&raw const $place`, which (IIUC) could be achieved by making `raw` a contextual keyword in a new edition. The type is named `*const T`, so the explicit `const` is consistent in that regard. `&raw expr` lacks the explicit indication of immutability. However, `&raw const expr` is quite a but longer than `addr_of!(expr)`.
- Shouldn't we have a completely new, better raw pointer type instead? Yes we all want to see that happen -- but I don't think we should block stabilization on that, given that such a nicer type is not on the horizon currently and given the issues with `addr_of!` mentioned above. (If we keep the `&raw $place` syntax free for this, we could use it in the future for that new type.)
- What about the lint the RFC talked about? It hasn't been implemented yet. Given that the problematic code is UB with or without this stabilization, I don't think the lack of the lint should block stabilization.
- I created an issue to track adding it: https://github.com/rust-lang/rust/issues/127724
- Other points from the "future possibilites of the RFC
- "Syntactic sugar" extension: this has not been implemented. I'd argue this is too confusing, we should stick to what the RFC suggested and if we want to do anything about such expressions, add the lint.
- Encouraging / requiring `&raw` in situations where references are often/definitely incorrect: this has been / is being implemented. On packed fields this already is a hard error, and for `static mut` a lint suggesting raw pointers is being rolled out.
- Lowering of casts: this has been implemented. (It's also an invisible implementation detail.)
- `offsetof` woes: we now have native `offset_of` so this is not relevant any more.
To be done before landing:
- [x] Suppress `unused_parens` lint around `&raw {const|mut}` expressions
- See bottom of https://github.com/rust-lang/rust/pull/127679#issuecomment-2264073752 for rationale
- Implementation: https://github.com/rust-lang/rust/pull/128782
- [ ] Update the Reference.
- https://github.com/rust-lang/reference/pull/1567
Fixes https://github.com/rust-lang/rust/issues/64490
cc `@rust-lang/lang` `@rust-lang/opsem`
try-job: x86_64-msvc
try-job: test-various
try-job: dist-various-1
try-job: armhf-gnu
try-job: aarch64-apple
Move ZST ABI handling to `rustc_target`
Currently, target specific handling of ZST function call ABI (specifically passing them indirectly instead of ignoring them) is handled in `rustc_ty_utils`, whereas all other target specific function call ABI handling is located in `rustc_target`. This PR moves the ZST handling to `rustc_target` so that all the target-specific function call ABI handling is in one place. In the process of doing so, this PR fixes#125850 by ensuring that ZST arguments are always correctly ignored in the x86-64 `"sysv64"` ABI; any code which would be affected by this fix would have ICEd before this PR. Tests are also added using `#[rustc_abi(debug)]` to ensure this behaviour does not regress.
Fixes#125850
Shrink `TyKind::FnPtr`.
By splitting the `FnSig` within `TyKind::FnPtr` into `FnSigTys` and `FnHeader`, which can be packed more efficiently. This reduces the size of the hot `TyKind` type from 32 bytes to 24 bytes on 64-bit platforms. This reduces peak memory usage by a few percent on some benchmarks. It also reduces cache misses and page faults similarly, though this doesn't translate to clear cycles or wall-time improvements on CI.
r? `@compiler-errors`
Normalize struct tail properly for `dyn` ptr-to-ptr casting in new solver
Realized that the new solver didn't handle ptr-to-ptr casting correctly.
r? lcnr
Built on #128694
By splitting the `FnSig` within `TyKind::FnPtr` into `FnSigTys` and
`FnHeader`, which can be packed more efficiently. This reduces the size
of the hot `TyKind` type from 32 bytes to 24 bytes on 64-bit platforms.
This reduces peak memory usage by a few percent on some benchmarks. It
also reduces cache misses and page faults similarly, though this doesn't
translate to clear cycles or wall-time improvements on CI.
Ensure floats are returned losslessly by the Rust ABI on 32-bit x86
Solves #115567 for the (default) `"Rust"` ABI. When compiling for 32-bit x86, this PR changes the `"Rust"` ABI to return floats indirectly instead of in x87 registers (with the exception of single `f32`s, which this PR returns in general purpose registers as they are small enough to fit in one). No change is made to the `"C"` ABI as that ABI requires x87 register usage and therefore will need a different solution.
Remove the unstable `extern "wasm"` ABI (`wasm_abi` feature tracked
in #83788).
As discussed in https://github.com/rust-lang/rust/pull/127513#issuecomment-2220410679
and following, this ABI is a failed experiment that did not end
up being used for anything. Keeping support for this ABI in LLVM 19
would require us to switch wasm targets to the `experimental-mv`
ABI, which we do not want to do.
It should be noted that `Abi::Wasm` was internally used for two
things: The `-Z wasm-c-abi=legacy` ABI that is still used by
default on some wasm targets, and the `extern "wasm"` ABI. Despite
both being `Abi::Wasm` internally, they were not the same. An
explicit `extern "wasm"` additionally enabled the `+multivalue`
feature.
I've opted to remove `Abi::Wasm` in this patch entirely, instead
of keeping it as an ABI with only internal usage. Both
`-Z wasm-c-abi` variants are now treated as part of the normal
C ABI, just with different different treatment in
adjust_for_foreign_abi.
Re-implement a type-size based limit
r? lcnr
This PR reintroduces the type length limit added in #37789, which was accidentally made practically useless by the caching changes to `Ty::walk` in #72412, which caused the `walk` function to no longer walk over identical elements.
Hitting this length limit is not fatal unless we are in codegen -- so it shouldn't affect passes like the mir inliner which creates potentially very large types (which we observed, for example, when the new trait solver compiles `itertools` in `--release` mode).
This also increases the type length limit from `1048576 == 2 ** 20` to `2 ** 24`, which covers all of the code that can be reached with craterbot-check. Individual crates can increase the length limit further if desired.
Perf regression is mild and I think we should accept it -- reinstating this limit is important for the new trait solver and to make sure we don't accidentally hit more type-size related regressions in the future.
Fixes#125460
Fix `FnMut::call_mut`/`Fn::call` shim for async closures that capture references
I adjusted async closures to be able to implement `Fn` and `FnMut` *even if* they capture references, as long as those references did not need to borrow data from the closure captures themselves. See #125259.
However, when I did this, I didn't actually relax an assertion in the `build_construct_coroutine_by_move_shim` shim code, which builds the `Fn`/`FnMut`/`FnOnce` implementations for async closures. Therefore, if we actually tried to *call* `FnMut`/`Fn` on async closures, it would ICE.
This PR adjusts this assertion to ensure that we only capture immutable references in closures if they implement `Fn`/`FnMut`. It also adds a bunch of tests and makes more of the async-closure tests into `build-pass` since we often care about these tests actually generating the right closure shims and stuff. I think it might be excessive to *always* use build-pass here, but 🤷 it's not that big of a deal.
Fixes#127019Fixes#127012
r? oli-obk
Only compute vtable information during codegen
This PR removes vtable information from the `Object` and `TraitUpcasting` candidate sources in the trait solvers, and defers the computation of relevant information to `Instance::resolve`. This is because vtables really aren't a thing in the trait world -- they're an implementation detail in codegen.
Previously it was just easiest to tangle this information together since we were already doing the work of looking at all the supertraits in the trait solver, and specifically because we use traits to represent when it's possible to call a method via a vtable (`Object` candidate) and do upcasting (`Unsize` candidate). but I am somewhat suspicious we're doing a *lot* of extra work, especially in polymorphic contexts, so let's see what perf says.
We already do this for a number of crates, e.g. `rustc_middle`,
`rustc_span`, `rustc_metadata`, `rustc_span`, `rustc_errors`.
For the ones we don't, in many cases the attributes are a mess.
- There is no consistency about order of attribute kinds (e.g.
`allow`/`deny`/`feature`).
- Within attribute kind groups (e.g. the `feature` attributes),
sometimes the order is alphabetical, and sometimes there is no
particular order.
- Sometimes the attributes of a particular kind aren't even grouped
all together, e.g. there might be a `feature`, then an `allow`, then
another `feature`.
This commit extends the existing sorting to all compiler crates,
increasing consistency. If any new attribute line is added there is now
only one place it can go -- no need for arbitrary decisions.
Exceptions:
- `rustc_log`, `rustc_next_trait_solver` and `rustc_type_ir_macros`,
because they have no crate attributes.
- `rustc_codegen_gcc`, because it's quasi-external to rustc (e.g. it's
ignored in `rustfmt.toml`).
