Rollup of 8 pull requests
Successful merges:
- #120484 (Avoid ICE when is_val_statically_known is not of a supported type)
- #120516 (pattern_analysis: cleanup manual impls)
- #120517 (never patterns: It is correct to lower `!` to `_`.)
- #120523 (Improve `io::Read::read_buf_exact` error case)
- #120528 (Store SHOULD_CAPTURE as AtomicU8)
- #120529 (Update data layouts in custom target tests for LLVM 18)
- #120531 (Remove a bunch of `has_errors` checks that have no meaningful or the wrong effect)
- #120533 (Correct paths for hexagon-unknown-none-elf platform doc)
r? `@ghost`
`@rustbot` modify labels: rollup
Avoid ICE when is_val_statically_known is not of a supported type
2 ICE with 1 stone!
1. Implement `llvm.is.constant.ptr` to avoid first ICE in linked issue.
2. return `false` when the argument is not one of `i*`/`f*`/`ptr` to avoid second ICE.
fixes#120480
Replacement of #114390: Add new intrinsic `is_var_statically_known` and optimize pow for powers of two
This adds a new intrinsic `is_val_statically_known` that lowers to [``@llvm.is.constant.*`](https://llvm.org/docs/LangRef.html#llvm-is-constant-intrinsic).` It also applies the intrinsic in the int_pow methods to recognize and optimize the idiom `2isize.pow(x)`. See #114390 for more discussion.
While I have extended the scope of the power of two optimization from #114390, I haven't added any new uses for the intrinsic. That can be done in later pull requests.
Note: When testing or using the library, be sure to use `--stage 1` or higher. Otherwise, the intrinsic will be a noop and the doctests will be skipped. If you are trying out edits, you may be interested in [`--keep-stage 0`](https://rustc-dev-guide.rust-lang.org/building/suggested.html#faster-builds-with---keep-stage).
Fixes#47234Resolves#114390
`@Centri3`
Use `assert_unchecked` instead of `assume` intrinsic in the standard library
Now that a public wrapper for the `assume` intrinsic exists, we can use it in the standard library.
CC #119131
Fix overflow check
Make MIRI choose the path randomly and rename the intrinsic
Add back test
Add miri test and make it operate on `ptr`
Define `llvm.is.constant` for primitives
Update MIRI comment and fix test in stage2
Add const eval test
Clarify that both branches must have the same side effects
guaranteed non guarantee
use immediate type instead
Co-Authored-By: Ralf Jung <post@ralfj.de>
Tune the inlinability of `unwrap`
Fixes#115463
cc `@thomcc`
This tweaks `unwrap` on ~~`Option` &~~ `Result` to be two parts:
- `#[inline(always)]` for checking the discriminant
- `#[cold]` for actually panicking
The idea here is that checking the discriminant on a `Result` ~~or `Option`~~ should always be trivial enough to be worth inlining, even in `opt-level=z`, especially compared to passing it to a function.
As seen in the issue and codegen test, this will hopefully help particularly for things like `.try_into().unwrap()`s that are actually infallible, but in a way that's only visible with the inlining.
EDIT: I've restricted this to `Result` to avoid combining effects
Separate immediate and in-memory ScalarPair representation
Currently, we assume that ScalarPair is always represented using a two-element struct, both as an immediate value and when stored in memory.
This currently works fairly well, but runs into problems with https://github.com/rust-lang/rust/pull/116672, where a ScalarPair involving an i128 type can no longer be represented as a two-element struct in memory. For example, the tuple `(i32, i128)` needs to be represented in-memory as `{ i32, [3 x i32], i128 }` to satisfy alignment requirements. Using `{ i32, i128 }` instead will result in the second element being stored at the wrong offset (prior to LLVM 18).
Resolve this issue by no longer requiring that the immediate and in-memory type for ScalarPair are the same. The in-memory type will now look the same as for normal struct types (and will include padding filler and similar), while the immediate type stays a simple two-element struct type. This also means that booleans in immediate ScalarPair are now represented as i1 rather than i8, just like we do everywhere else.
The core change here is to llvm_type (which now treats ScalarPair as a normal struct) and immediate_llvm_type (which returns the two-element struct that llvm_type used to produce). The rest is fixing things up to no longer assume these are the same. In particular, this switches places that try to get pointers to the ScalarPair elements to use byte-geps instead of struct-geps.
llvm: Allow `noundef` in codegen tests
LLVM 18 will automatically infer `noundef` in some situations. Adjust codegen tests to accept this.
See llvm/llvm-project#76553 for why `noundef` is being generated now.
``@rustbot`` label:+llvm-main
LLVM 18 will automatically infer `noundef` in some situations.
Adjust codegen tests to accept this.
See llvm/llvm-project#76553 for why `noundef` is being generated now.
On LLVM 18 we get slightly different arguments here, so it's easier to
just regex those away. The important details are all still asserted as I
understand things.
Fixes#119193.
@rustbot label: +llvm-main
add more niches to rawvec
Previously RawVec only had a single niche in its `NonNull` pointer. With this change it now has `isize::MAX` niches since half the value-space of the capacity field is never needed, we can't have a capacity larger than isize::MAX.
Currently, we assume that ScalarPair is always represented using
a two-element struct, both as an immediate value and when stored
in memory.
This currently works fairly well, but runs into problems with
https://github.com/rust-lang/rust/pull/116672, where a ScalarPair
involving an i128 type can no longer be represented as a two-element
struct in memory. For example, the tuple `(i32, i128)` needs to be
represented in-memory as `{ i32, [3 x i32], i128 }` to satisfy
alignment requirement. Using `{ i32, i128 }` instead will result in
the second element being stored at the wrong offset (prior to
LLVM 18).
Resolve this issue by no longer requiring that the immediate and
in-memory type for ScalarPair are the same. The in-memory type
will now look the same as for normal struct types (and will include
padding filler and similar), while the immediate type stays a
simple two-element struct type. This also means that booleans in
immediate ScalarPair are now represented as i1 rather than i8,
just like we do everywhere else.
The core change here is to llvm_type (which now treats ScalarPair
as a normal struct) and immediate_llvm_type (which returns the
two-element struct that llvm_type used to produce). The rest is
fixing things up to no longer assume these are the same. In
particular, this switches places that try to get pointers to the
ScalarPair elements to use byte-geps instead of struct-geps.
Sets the accessibility of types and fields in DWARF using
`DW_AT_accessibility` attribute.
`DW_AT_accessibility` (public/protected/private) isn't exactly right for
Rust, but neither is `DW_AT_visibility` (local/exported/qualified), and
there's no way to set `DW_AT_visbility` in LLVM's API.
Signed-off-by: David Wood <david@davidtw.co>
Enable stack probes on aarch64 for LLVM 18
I tested this on `aarch64-unknown-linux-gnu` with LLVM main (~18).
cc #77071, to be closed once we upgrade our LLVM submodule.
Add `-Zfunction-return={keep,thunk-extern}` option
This is intended to be used for Linux kernel RETHUNK builds.
With this commit (optionally backported to Rust 1.73.0), plus a patched Linux kernel to pass the flag, I get a RETHUNK build with Rust enabled that is `objtool`-warning-free and is able to boot in QEMU and load a sample Rust kernel module.
Issue: https://github.com/rust-lang/rust/issues/116853.
This is intended to be used for Linux kernel RETHUNK builds.
With this commit (optionally backported to Rust 1.73.0), plus a
patched Linux kernel to pass the flag, I get a RETHUNK build with
Rust enabled that is `objtool`-warning-free and is able to boot in
QEMU and load a sample Rust kernel module.
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
Add thinlto support to codegen, assembly and coverage tests
Using `--emit=llvm-ir` with thinlto usually result in multiple IR files.
Resolve test case failure issue reported in #113923.
In the future Windows will enable Control-flow Enforcement Technology
(CET aka Shadow Stacks). To protect the path where the context is
updated during exception handling, the binary is required to enumerate
valid unwind entrypoints in a dedicated section which is validated when
the context is being set during exception handling.
The required support for EHCONT has already been merged into LLVM,
long ago. This change adds the Rust codegen option to enable it.
Reference:
* https://reviews.llvm.org/D40223
This also adds a new `ehcont-guard` option to the bootstrap config which
enables EHCont Guard when building std.
Add -Z llvm_module_flag
Allow adding values to the `!llvm.module.flags` metadata for a generated module. The syntax is
`-Z llvm_module_flag=<name>:<type>:<value>:<behavior>`
Currently only u32 values are supported but the type is required to be specified for forward compatibility. The `behavior` element must match one of the named LLVM metadata behaviors.viors.
This flag is expected to be perma-unstable.
This was broken by upstream
llvm/llvm-project@dc6d077396. It's easy
enough to use a regex match to support both, so we do that.
r? @nikic
@rustbot label: +llvm-main
Allow adding values to the `!llvm.module.flags` metadata for a generated
module. The syntax is
`-Z llvm_module_flag=<name>:<type>:<value>:<behavior>`
Currently only u32 values are supported but the type is required to be
specified for forward compatibility. The `behavior` element must match
one of the named LLVM metadata behaviors.viors.
This flag is expected to be perma-unstable.
Hint optimizer about try-reserved capacity
This is #116568, but limited only to the less-common `try_reserve` functions to reduce bloat in debug binaries from debug info, while still addressing the main use-case #116570
Clean up unchecked_math, separate out unchecked_shifts
Tracking issue: #85122
Changes:
1. Remove `const_inherent_unchecked_arith` flag and make const-stability flags the same as the method feature flags. Given the number of other unsafe const fns already stabilised, it makes sense to just stabilise these in const context when they're stabilised.
2. Move `unchecked_shl` and `unchecked_shr` into a separate `unchecked_shifts` flag, since the semantics for them are unclear and they'll likely be stabilised separately as a result.
3. Add an `unchecked_neg` method exclusively to signed integers, under the `unchecked_neg` flag. This is because it's a new API and probably needs some time to marinate before it's stabilised, and while it *would* make sense to have a similar version for unsigned integers since `checked_neg` also exists for those there is absolutely no case where that would be a good idea, IMQHO.
The longer-term goal here is to prepare the `unchecked_math` methods for an FCP and stabilisation since they've existed for a while, their semantics are clear, and people seem in favour of stabilising them.
Decompose singular `matches!` with or-patterns to individual `matches!`
statements to enable branchless code output. The following functions
were changed:
- `is_ascii_alphanumeric`
- `is_ascii_hexdigit`
- `is_ascii_punctuation`
Add codegen tests
Co-authored-by: George Bateman <george.bateman16@gmail.com>
Co-authored-by: scottmcm <scottmcm@users.noreply.github.com>
Copy 1-element arrays as scalars, not vectors
For `[T; 1]` it's silly to copy as `<1 x T>` when we can just copy as `T`.
Inspired by https://github.com/rust-lang/rust/issues/101210#issuecomment-1732470941, which pointed out that `Option<[u8; 1]>` was codegenning worse than `Option<u8>`.
(I'm not sure *why* LLVM doesn't optimize out `<1 x u8>`, but might as well just not emit it in the first place in this codepath.)
---
I think I bit off too much in #116479; let me try just the scalar case first.
r? `@ghost`
Raise minimum supported Apple OS versions
This implements the proposal to raise the minimum supported Apple OS versions as laid out in the now-completed MCP (https://github.com/rust-lang/compiler-team/issues/556).
As of this PR, rustc and the stdlib now support these versions as the baseline:
- macOS: 10.12 Sierra
- iOS: 10
- tvOS: 10
- watchOS: 5 (Unchanged)
In addition to everything this breaks indirectly, these changes also erase the `armv7-apple-ios` target (currently tier 3) because the oldest supported iOS device now uses ARMv7s. Not sure what the policy around tier3 target removal is but shimming it is not an option due to the linker refusing.
[Per comment](https://github.com/rust-lang/compiler-team/issues/556#issuecomment-1297175073), this requires a FCP to merge. cc `@wesleywiser.`
Enable -Zdrop-tracking-mir by default
This PR enables the `drop-tracking-mir` flag by default. This flag was initially implemented in https://github.com/rust-lang/rust/pull/101692.
This flag computes auto-traits on generators based on their analysis MIR, instead of trying to compute on the HIR body. This removes the need for HIR-based drop-tracking, as we can now reuse the same code to compute generator witness types and to compute generator interior fields.
Add codegen test to guard against VecDeque optimization regression
Very small PR that adds a codegen test to guard against regression for the `VecDeque` optimization addressed in #80836. Ensures that Rustc optimizes away the panic when unwrapping the result of `.get(0)` because of the `!is_empty()` condition.
Use the same DISubprogram for each instance of the same inlined function within a caller
# Issue Details:
The call to `panic` within a function like `Option::unwrap` is translated to LLVM as a `tail call` (as it will never return), when multiple calls to the same function like this are inlined LLVM will notice the common `tail call` block (i.e., loading the same panic string + location info and then calling `panic`) and merge them together.
When merging these instructions together, LLVM will also attempt to merge the debug locations as well, but this fails (i.e., debug info is dropped) as Rust emits a new `DISubprogram` at each inline site thus LLVM doesn't recognize that these are actually the same function and so thinks that there isn't a common debug location.
As an example of this, consider the following program:
```rust
#[no_mangle]
fn add_numbers(x: &Option<i32>, y: &Option<i32>) -> i32 {
let x1 = x.unwrap();
let y1 = y.unwrap();
x1 + y1
}
```
When building for x86_64 Windows using 1.72 it generates (note the lack of `.cv_loc` before the call to `panic`, thus it will be attributed to the same line at the `addq` instruction):
```llvm
.cv_loc 0 1 3 0 # src\lib.rs:3:0
addq $40, %rsp
retq
leaq .Lalloc_f570dea0a53168780ce9a91e67646421(%rip), %rcx
leaq .Lalloc_629ace53b7e5b76aaa810d549cc84ea3(%rip), %r8
movl $43, %edx
callq _ZN4core9panicking5panic17h12e60b9063f6dee8E
int3
```
# Fix Details:
Cache the `DISubprogram` emitted for each inlined function instance within a caller so that this can be reused if that instance is encountered again.
Ideally, we would also deduplicate child scopes and variables, however my attempt to do that with #114643 resulted in asserts when building for Linux (#115156) which would require some deep changes to Rust to fix (#115455).
Instead, when using an inlined function as a debug scope, we will also create a new child scope such that subsequent child scopes and variables do not collide (from LLVM's perspective).
After this change the above assembly now (with <https://reviews.llvm.org/D159226> as well) shows the `panic!` was inlined from `unwrap` in `option.rs` at line 935 into the current function in `lib.rs` at line 0 (line 0 is emitted since it is ambiguous which line to use as there were two inline sites that lead to this same code):
```llvm
.cv_loc 0 1 3 0 # src\lib.rs:3:0
addq $40, %rsp
retq
.cv_inline_site_id 6 within 0 inlined_at 1 0 0
.cv_loc 6 2 935 0 # library\core\src\option.rs:935:0
leaq .Lalloc_5f55955de67e57c79064b537689facea(%rip), %rcx
leaq .Lalloc_e741d4de8cb5801e1fd7a6c6795c1559(%rip), %r8
movl $43, %edx
callq _ZN4core9panicking5panic17hde1558f32d5b1c04E
int3
```
Use `preserve_mostcc` for `extern "rust-cold"`
As experimentation in #115242 has shown looks better than `coldcc`. Notably, clang exposes `preserve_most` (https://clang.llvm.org/docs/AttributeReference.html#preserve-most) but not `cold`, so this change should put us on a better-supported path.
And *don't* use a different convention for cold on Windows, because that actually ends up making things worse. (See comment in the code.)
cc tracking issue #97544
Remove some wasm/emscripten ignores
I'm planning on landing a few PRs like this that remove ignores that aren't required. This just covers mir-opt and codegen tests.
As experimentation in 115242 has shown looks better than `coldcc`.
And *don't* use a different convention for cold on Windows, because that actually ends up making things worse.
cc tracking issue 97544
Revert "Use the same DISubprogram for each instance of the same inline function within the caller"
This reverts commit 687bffa493.
Reverting to resolve ICEs reported on nightly.
cc `@dpaoliello`
Fixes#115156
Fix#115150 by encoding f32 and f64 correctly for cross-language CFI. I
missed changing the encoding for f32 and f64 when I introduced the
integer normalization option in #105452 as integer normalization does
not include floating point. `f32` and `f64` should be always encoded as
`f` and `d` since they are both FFI safe when their representation are
the same (i.e., IEEE 754) for both the Rust compiler and Clang.
Add regression test for not `memcpy`ing padding bytes
Closes#56297
See this comparison: https://rust.godbolt.org/z/jjzfonfcE
I don't have any experience with codegen tests, I hope this is correct
Use the same DISubprogram for each instance of the same inlined function within a caller
# Issue Details:
The call to `panic` within a function like `Option::unwrap` is translated to LLVM as a `tail call` (as it will never return), when multiple calls to the same function like this is inlined LLVM will notice the common `tail call` block (i.e., loading the same panic string + location info and then calling `panic`) and merge them together.
When merging these instructions together, LLVM will also attempt to merge the debug locations as well, but this fails (i.e., debug info is dropped) as Rust emits a new `DISubprogram` at each inline site thus LLVM doesn't recognize that these are actually the same function and so thinks that there isn't a common debug location.
As an example of this when building for x86_64 Windows (note the lack of `.cv_loc` before the call to `panic`, thus it will be attributed to the same line at the `addq` instruction):
```
.cv_loc 0 1 23 0 # src\lib.rs:23:0
addq $40, %rsp
retq
leaq .Lalloc_f570dea0a53168780ce9a91e67646421(%rip), %rcx
leaq .Lalloc_629ace53b7e5b76aaa810d549cc84ea3(%rip), %r8
movl $43, %edx
callq _ZN4core9panicking5panic17h12e60b9063f6dee8E
int3
```
# Fix Details:
Cache the `DISubprogram` emitted for each inlined function instance within a caller so that this can be reused if that instance is encountered again, this also requires caching the `DILexicalBlock` and `DIVariable` objects to avoid creating duplicates.
After this change the above assembly now looks like:
```
.cv_loc 0 1 23 0 # src\lib.rs:23:0
addq $40, %rsp
retq
.cv_inline_site_id 5 within 0 inlined_at 1 0 0
.cv_inline_site_id 6 within 5 inlined_at 1 12 0
.cv_loc 6 2 935 0 # library\core\src\option.rs:935:0
leaq .Lalloc_5f55955de67e57c79064b537689facea(%rip), %rcx
leaq .Lalloc_e741d4de8cb5801e1fd7a6c6795c1559(%rip), %r8
movl $43, %edx
callq _ZN4core9panicking5panic17hde1558f32d5b1c04E
int3
```
We've investigated one reason why debugging information often goes wrong at https://reviews.llvm.org/D152095.
> LLVM can't handle IR where subprogram definitions are nested within DICompositeType when doing LTO builds,
> because there's no good way to cross the CU boundary to insert a nested DISubprogram definition in one CU into a type defined in another CU.
