Rollup of 9 pull requests
Successful merges:
- #135340 (Add `explicit_extern_abis` Feature and Enforce Explicit ABIs)
- #139440 (rustc_target: RISC-V: feature addition batch 2)
- #139667 (cfi: Remove #[no_sanitize(cfi)] for extern weak functions)
- #139828 (Don't require rigid alias's trait to hold)
- #139854 (Improve parse errors for stray lifetimes in type position)
- #139889 (Clean UI tests 3 of n)
- #139894 (Fix `opt-dist` CLI flag and make it work without LLD)
- #139900 (stepping into impls for normalization is unproductive)
- #139915 (replace some #[rustc_intrinsic] usage with use of the libcore declarations)
r? `@ghost`
`@rustbot` modify labels: rollup
fix for multiple `#[repr(align(N))]` on functions
tracking issue: https://github.com/rust-lang/rust/issues/82232
fixes https://github.com/rust-lang/rust/issues/132464
The behavior of align is specified at https://doc.rust-lang.org/reference/type-layout.html#r-layout.repr.alignment.align
> For align, if the specified alignment is less than the alignment of the type without the align modifier, then the alignment is unaffected.
So in effect that means that the maximum of the specified alignments should be chosen. That is also the current behavior for `align` on ADTs:
```rust
#![feature(fn_align)]
#[repr(C, align(32), align(64))]
struct Foo {
x: u64,
}
const _: () = assert!(core::mem::align_of::<Foo>() == 64);
// See the godbolt LLVM output: the alignment of this function is 32
#[no_mangle]
#[repr(align(32))]
#[repr(align(64))]
fn foo() {}
// The current logic just picks the first alignment: the alignment of this function is 64
#[no_mangle]
#[repr(align(64))]
#[repr(align(32))]
fn bar() {}
```
https://godbolt.org/z/scco435jEafa859f812/compiler/rustc_middle/src/ty/mod.rs (L1529-L1532)
The https://github.com/rust-lang/rust/issues/132464 issue is really about parsing/representing the attribute, which has already been improved and now uses the "parse, don't validate" attribute approach. That means the behavior is already different from what the issue describes: on current `main`, the first value is chosen. This PR fixes a logic error, where we just did not check for the effect of two or more `align` modifiers. In combination, that fixes the issue.
cc ``@jdonszelmann`` if you do have further thoughs here
enforce unsafe attributes in pre-2024 editions by default
New unsafe attributes should emit an error when used without the `unsafe(...)` in all editions.
The `no_mangle`, `link_section` and `export_name` attributes are exceptions, and can still be used without an unsafe in earlier editions. The only attributes for which this change is relevant right now are `#[ffi_const]` and `#[ffi_pure]`.
This change is required for making `#[unsafe(naked)]` sound in pre-2024 editions.
Move `select_unpredictable` to the `hint` module
There has been considerable discussion in both the ACP (rust-lang/libs-team#468) and tracking issue (#133962) about whether the `bool::select_unpredictable` method should be in `core::hint` instead.
I believe this is the right move for the following reasons:
- The documentation explicitly says that it is a hint, not a codegen guarantee.
- `bool` doesn't have a corresponding `select` method, and I don't think we should be adding one.
- This shouldn't be something that people reach for with auto-completion unless they specifically understand the interactions with branch prediction. Using conditional moves can easily make code *slower* by preventing the CPU from speculating past the condition due to the data dependency.
- Although currently `core::hint` only contains no-ops, this isn't a hard rule (for example `unreachable_unchecked` is a bit of a gray area). The documentation only status that the module contains "hints to compiler that affects how code should be emitted or optimized". This is consistent with what `select_unpredictable` does.
Use the chaining methods on PartialOrd for slices too
#138135 added these doc-hidden trait methods to improve the tuple codegen. This PR adds more implementations and callers so that the codegen for slice (and array) comparisons also improves.
Polymorphize `array::IntoIter`'s iterator impl
Today we emit all the iterator methods for every different array width. That's wasteful since the actual array length never even comes into it -- the indices used are from the separate `alive: IndexRange` field, not even the `N` const param.
This PR switches things so that an `array::IntoIter<T, N>` stores a `PolymorphicIter<[MaybeUninit<T>; N]>`, which we *unsize* to `PolymorphicIter<[MaybeUninit<T>]>` and call methods on that non-`Sized` type for all the iterator methods.
That also necessarily makes the layout consistent between the different lengths of arrays, because of the unsizing. Compare that to today <https://rust.godbolt.org/z/Prb4xMPrb>, where different widths can't even be deduped because the offset to the indices is different for different array widths.
Fix breakage when running compiletest with `--test-args=--edition=2015`
Compiletest has an `--edition` flag to change the default edition tests are run with. Unfortunately no test suite successfully executes when that flag is passed. If the edition is set to something greater than 2015 the breakage is expected, since the test suite currently supports only edition 2015 (Ferrous Systems will open an MCP about fixing that soonish). Surprisingly, the test suite is also broken if `--edition=2015` is passed to compiletest. This PR focuses on fixing the latter.
This PR fixes the two categories of failures happening when `--edition=2015` is passed:
* Some edition-specific tests set their edition through `//@ compile-flags` instead of `//@ edition`. Compiletest doesn't parse the compile flags, so it would see no `//@ edition` and add another `--edition` flag, leading to a rustc error.
* Compiletest would add the edition after `//@ compile-flags`, while some tests depend on flags passed to `//@ compile-flags` being the last flags in the rustc invocation.
Note that for the first category, I opted to manually go and replace all `//@ compile-flags` setting an edition with an explicit `//@ edition`. We could've changed compiletest to instead check whether an edition was set in `//@ compile-flags`, but I thought it was better to enforce a consistent way to set the edition in tests.
I also added the edition to the stamp, so that changing `--edition` results in tests being re-executed.
r? `@jieyouxu`
add `core::intrinsics::simd::{simd_extract_dyn, simd_insert_dyn}`
fixes https://github.com/rust-lang/rust/issues/137372
adds `core::intrinsics::simd::{simd_extract_dyn, simd_insert_dyn}`, which contrary to their non-dyn counterparts allow a non-const index. Many platforms (but notably not x86_64 or aarch64) have dedicated instructions for this operation, which stdarch can emit with this change.
Future work is to also make the `Index` operation on the `Simd` type emit this operation, but the intrinsic can't be used directly. We'll need some MIR shenanigans for that.
r? `@ghost`
Ensure `swap_nonoverlapping` is really always untyped
This replaces #134954, which was arguably overcomplicated.
## Fixes#134713
Actually using the type passed to `ptr::swap_nonoverlapping` for anything other than its size + align turns out to not work, so this goes back to always erasing the types down to just bytes.
(Except in `const`, which keeps doing the same thing as before to preserve `@RalfJung's` fix from #134689)
## Fixes#134946
I'd previously moved the swapping to use auto-vectorization *on bytes*, but someone pointed out on Discord that the tail loop handling from that left a whole bunch of byte-by-byte swapping around. This goes back to manual tail handling to avoid that, then still triggers auto-vectorization on pointer-width values. (So you'll see `<4 x i64>` on `x86-64-v3` for example.)
Tell LLVM about impossible niche tags
I was trying to find a better way of emitting discriminant calculations, but sadly had no luck.
So here's a fairly small PR with the bits that did seem worth bothering:
1. As the [`TagEncoding::Niche` docs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_abi/enum.TagEncoding.html#variant.Niche) describe, it's possible to end up with a dead value in the input that's not already communicated via the range parameter attribute nor the range load metadata attribute. So this adds an `llvm.assume` in non-debug mode to tell LLVM about that. (That way it can tell that the sides of the `select` have disjoint possible values.)
2. I'd written a bunch more tests, or at least made them parameterized, in the process of trying things out, so this checks in those tests to hopefully help future people not trip on the same weird edge cases, like when the tag type is `i8` but yet there's still a variant index and discriminant of `258` which doesn't fit in that tag type because the enum is really weird.
add sret handling for scalar autodiff
r? `@oli-obk`
Fixing one of the todo's which I left in my previous batching PR.
This one handles sret for scalar autodiff. `sret` mostly shows up when we try to return a lot of scalar floats.
People often start testing autodiff which toy functions which just use a few scalars as inputs and outputs, and those were the most likely to be affected by this issue. So this fix should make learning/teaching hopefully a bit easier.
Tracking:
- https://github.com/rust-lang/rust/issues/124509
The default s390x cpu(z10) does not have vector support. Setting
target-cpu at least to z13 enables vectorisation for s390x architecture
and makes the tests pass.
Prevent a test from seeing forbidden numbers in the rustc version
The final CHECK-NOT directive in this test was able to see past the end of the enclosing function, and find the substring `753` or `754` in the git hash in the rustc version number, causing false failures in CI whenever the git hash happens to contain those digits in sequence.
Adding an explicit check for `ret` prevents the CHECK-NOT directive from seeing past the end of the function.
---
Manually tested by adding `// CHECK-NOT: rustc` after the existing CHECK-NOT directives, and demonstrating that the new check prevents it from seeing the rustc version string.
The final CHECK-NOT directive in this test was able to see past the end of the
enclosing function, and find the substring 753 or 754 in the git hash in the
rustc version number, causing false failures in CI.
Adding an explicit check for `ret` prevents the CHECK-NOT directive from seeing
past the end of the function.
Autodiff batching
Enzyme supports batching, which is especially known from the ML side when training neural networks.
There we would normally have a training loop, where in each iteration we would pass in some data (e.g. an image), and a target vector. Based on how close we are with our prediction we compute our loss, and then use backpropagation to compute the gradients and update our weights.
That's quite inefficient, so what you normally do is passing in a batch of 8/16/.. images and targets, and compute the gradients for those all at once, allowing better optimizations.
Enzyme supports batching in two ways, the first one (which I implemented here) just accepts a Batch size,
and then each Dual/Duplicated argument has not one, but N shadow arguments. So instead of
```rs
for i in 0..100 {
df(x[i], y[i], 1234);
}
```
You can now do
```rs
for i in 0..100.step_by(4) {
df(x[i+0],x[i+1],x[i+2],x[i+3], y[i+0], y[i+1], y[i+2], y[i+3], 1234);
}
```
which will give the same results, but allows better compiler optimizations. See the testcase for details.
There is a second variant, where we can mark certain arguments and instead of having to pass in N shadow arguments, Enzyme assumes that the argument is N times longer. I.e. instead of accepting 4 slices with 12 floats each, we would accept one slice with 48 floats. I'll implement this over the next days.
I will also add more tests for both modes.
For any one preferring some more interactive explanation, here's a video of Tim's llvm dev talk, where he presents his work. https://www.youtube.com/watch?v=edvaLAL5RqU
I'll also add some other docs to the dev guide and user docs in another PR.
r? ghost
Tracking:
- https://github.com/rust-lang/rust/issues/124509
- https://github.com/rust-lang/rust/issues/135283
Expose algebraic floating point intrinsics
# Problem
A stable Rust implementation of a simple dot product is 8x slower than C++ on modern x86-64 CPUs. The root cause is an inability to let the compiler reorder floating point operations for better vectorization.
See https://github.com/calder/dot-bench for benchmarks. Measurements below were performed on a i7-10875H.
### C++: 10us ✅
With Clang 18.1.3 and `-O2 -march=haswell`:
<table>
<tr>
<th>C++</th>
<th>Assembly</th>
</tr>
<tr>
<td>
<pre lang="cc">
float dot(float *a, float *b, size_t len) {
#pragma clang fp reassociate(on)
float sum = 0.0;
for (size_t i = 0; i < len; ++i) {
sum += a[i] * b[i];
}
return sum;
}
</pre>
</td>
<td>
<img src="https://github.com/user-attachments/assets/739573c0-380a-4d84-9fd9-141343ce7e68" />
</td>
</tr>
</table>
### Nightly Rust: 10us ✅
With rustc 1.86.0-nightly (8239a37f9) and `-C opt-level=3 -C target-feature=+avx2,+fma`:
<table>
<tr>
<th>Rust</th>
<th>Assembly</th>
</tr>
<tr>
<td>
<pre lang="rust">
fn dot(a: &[f32], b: &[f32]) -> f32 {
let mut sum = 0.0;
for i in 0..a.len() {
sum = fadd_algebraic(sum, fmul_algebraic(a[i], b[i]));
}
sum
}
</pre>
</td>
<td>
<img src="https://github.com/user-attachments/assets/9dcf953a-2cd7-42f3-bc34-7117de4c5fb9" />
</td>
</tr>
</table>
### Stable Rust: 84us ❌
With rustc 1.84.1 (e71f9a9a9) and `-C opt-level=3 -C target-feature=+avx2,+fma`:
<table>
<tr>
<th>Rust</th>
<th>Assembly</th>
</tr>
<tr>
<td>
<pre lang="rust">
fn dot(a: &[f32], b: &[f32]) -> f32 {
let mut sum = 0.0;
for i in 0..a.len() {
sum += a[i] * b[i];
}
sum
}
</pre>
</td>
<td>
<img src="https://github.com/user-attachments/assets/936a1f7e-33e4-4ff8-a732-c3cdfe068dca" />
</td>
</tr>
</table>
# Proposed Change
Add `core::intrinsics::f*_algebraic` wrappers to `f16`, `f32`, `f64`, and `f128` gated on a new `float_algebraic` feature.
# Alternatives Considered
https://github.com/rust-lang/rust/issues/21690 has a lot of good discussion of various options for supporting fast math in Rust, but is still open a decade later because any choice that opts in more than individual operations is ultimately contrary to Rust's design principles.
In the mean time, processors have evolved and we're leaving major performance on the table by not supporting vectorization. We shouldn't make users choose between an unstable compiler and an 8x performance hit.
# References
* https://github.com/rust-lang/rust/issues/21690
* https://github.com/rust-lang/libs-team/issues/532
* https://github.com/rust-lang/rust/issues/136469
* https://github.com/calder/dot-bench
* https://www.felixcloutier.com/x86/vfmadd132ps:vfmadd213ps:vfmadd231ps
try-job: x86_64-gnu-nopt
try-job: x86_64-gnu-aux
gvn: Invalid dereferences for all non-local mutations
Fixes#132353.
This PR removes the computation value by traversing SSA locals through `for_each_assignment_mut`.
Because the `for_each_assignment_mut` traversal skips statements which have side effects, such as dereference assignments, the computation may be unsound. Instead of `for_each_assignment_mut`, we compute values by traversing in reverse postorder.
Because we compute and use the symbolic representation of values on the fly, I invalidate all old values when encountering a dereference assignment. The current approach does not prevent the optimization of a clone to a copy.
In the future, we may add an alias model, or dominance information for dereference assignments, or SSA form to help GVN.
r? cjgillot
cc `@jieyouxu` #132356
cc `@RalfJung` #133474
slice: Remove some uses of unsafe in first/last chunk methods
Remove unsafe `split_at_unchecked` and `split_at_mut_unchecked` in some slice `split_first_chunk`/`split_last_chunk` methods.
Replace those calls with the safe `split_at` and `split_at_checked` where applicable.
Add codegen tests to check for no panics when calculating the last chunk index using `checked_sub` and `split_at`.
Better viewed with whitespace disabled in diff view
---
The unchecked calls are mostly manual implementations of the safe methods, but with the safety condition negated from `mid <= len` to `len < mid`.
```rust
if self.len() < N {
None
} else {
// SAFETY: We manually verified the bounds of the split.
let (first, tail) = unsafe { self.split_at_unchecked(N) };
// Or for the last_chunk methods
let (init, last) = unsafe { self.split_at_unchecked(self.len() - N) };
```
Unsafe is still needed for the pointer array casts. Their safety comments are unmodified.
PassWrapper: adapt for llvm/llvm-project@94122d58fc77079a291a3d008914…
…006cb509d9db
We also have to remove the LLVM argument in cast-target-abi.rs for LLVM
21. I'm not really sure what the best approach here is since that test already uses revisions. We could also fork the test into a copy for LLVM 19-20 and another for LLVM 21, but what I did for now was drop the lint-abort-on-error flag to LLVM figuring that some coverage was better than none, but I'm happy to change this if that was a bad direction.
r? dianqk
````@rustbot```` label llvm-main
