intrinsics fmuladdf{32,64}: expose llvm.fmuladd.* semantics
Add intrinsics `fmuladd{f32,f64}`. This computes `(a * b) + c`, to be fused if the code generator determines that (i) the target instruction set has support for a fused operation, and (ii) that the fused operation is more efficient than the equivalent, separate pair of `mul` and `add` instructions.
https://llvm.org/docs/LangRef.html#llvm-fmuladd-intrinsic
The codegen_cranelift uses the `fma` function from libc, which is a correct implementation, but without the desired performance semantic. I think this requires an update to cranelift to expose a suitable instruction in its IR.
I have not tested with codegen_gcc, but it should behave the same way (using `fma` from libc).
---
This topic has been discussed a few times on Zulip and was suggested, for example, by `@workingjubilee` in [Effect of fma disabled](https://rust-lang.zulipchat.com/#narrow/stream/122651-general/topic/Effect.20of.20fma.20disabled/near/274179331).
Add intrinsics `fmuladd{f16,f32,f64,f128}`. This computes `(a * b) +
c`, to be fused if the code generator determines that (i) the target
instruction set has support for a fused operation, and (ii) that the
fused operation is more efficient than the equivalent, separate pair
of `mul` and `add` instructions.
https://llvm.org/docs/LangRef.html#llvm-fmuladd-intrinsic
MIRI support is included for f32 and f64.
The codegen_cranelift uses the `fma` function from libc, which is a
correct implementation, but without the desired performance semantic. I
think this requires an update to cranelift to expose a suitable
instruction in its IR.
I have not tested with codegen_gcc, but it should behave the same
way (using `fma` from libc).
Use Default visibility for rustc-generated C symbol declarations
Non-default visibilities should only be used for definitions, not declarations, otherwise linking can fail.
This is based on https://github.com/rust-lang/rust/pull/123994.
Issue https://github.com/rust-lang/rust/issues/123427
When I changed `default-hidden-visibility` to `default-visibility` in https://github.com/rust-lang/rust/pull/130005, I updated all places in the code that used `default-hidden-visibility`, replicating the hidden-visibility bug to also happen for protected visibility.
Without this change, trying to build rustc with `-Z default-visibility=protected` fails with a link error.
Non-default visibilities should only be used for definitions, not
declarations, otherwise linking can fail.
Co-authored-by: Collin Baker <collinbaker@chromium.org>
rustc_target: Add sme-b16b16 as an explicit aarch64 target feature
LLVM 20 split out what used to be called b16b16 and correspond to aarch64
FEAT_SVE_B16B16 into sve-b16b16 and sme-b16b16.
Add sme-b16b16 as an explicit feature and update the codegen accordingly.
Resolves https://github.com/rust-lang/rust/pull/129894.
codegen_ssa: consolidate tied target checks
Fixes#105110.
Fixes#105111.
`rustc_codegen_llvm` and `rustc_codegen_gcc` duplicated logic for checking if tied target features were partially enabled. This PR consolidates these checks into `rustc_codegen_ssa` in the `codegen_fn_attrs` query, which also is run pre-monomorphisation for each function, which ensures that this check is run for unused functions, as would be expected.
Also adds a test confirming that enabling one tied feature doesn't imply another - the appropriate error for this was already being emitted. I did a bisect and narrowed it down to two patches it was likely to be - something in #128796, probably #128221 or #128679.
LLVM 20 split out what used to be called b16b16 and correspond to aarch64
FEAT_SVE_B16B16 into sve-b16b16 and sme-b16b16.
Add sme-b16b16 as an explicit feature and update the codegen accordingly.
compiler: Stop reexporting enum-globs from `rustc_target::abi`
Three enums had **all** their variants glob-exported into a distressingly large amount of the tree. Cease to do that, and also cease to glob import the contents of the module that contained them. Redirect relevant imports to their actual source, the `rustc_abi` crate.
No functional changes.
[Coverage][MCDC] Adapt mcdc to llvm 19
Related issue: #126672
Also finish task 4 at #124144
[llvm #82448](https://github.com/llvm/llvm-project/pull/82448) has introduced some break changes into mcdc, causing incompatibility between llvm 18 and 19. This draft adapts to that change and gives up supporting for llvm-18.
Add missing module flags for `-Zfunction-return=thunk-extern`
This fixes a bug in the `-Zfunction-return=thunk-extern` flag. The flag needs to be passed onto LLVM to ensure that functions such as `asan.module_ctor` and `asan.module_dtor` that are created internally in LLVM have the mitigation applied to them.
This was originally discovered [in the Linux kernel](https://lore.kernel.org/all/CANiq72myZL4_poCMuNFevtpYYc0V0embjSuKb7y=C+m3vVA_8g@mail.gmail.com/).
Original flag PR: #116892
PR for similar issue: #129373
Tracking issue: #116853
cc ``@ojeda``
r? ``@wesleywiser``
Like #130865 did for the standard library, we can use `&raw` in the
compiler now that stage0 supports it. Also like the other issue, I did
not make any doc or test changes at this time.
`rustc_codegen_llvm` and `rustc_codegen_gcc` duplicated logic for
checking if tied target features were partially enabled. This commit
consolidates these checks into `rustc_codegen_ssa` in the
`codegen_fn_attrs` query, which also is run pre-monomorphisation for
each function, which ensures that this check is run for unused functions,
as would be expected.
llvm: replace some deprecated functions
`LLVMMDStringInContext` and `LLVMMDNodeInContext` are deprecated, replace them with `LLVMMDStringInContext2` and `LLVMMDNodeInContext2`.
Also replace `Value` with `Metadata` in some function signatures for better consistency.
Bump stage0 to beta-2024-09-22 and rustfmt to nightly-2024-09-22
I'm doing this to apply the changes to version sorting (https://github.com/rust-lang/rustfmt/pull/6284) that have occurred since rustfmt last upgraded (and a few other miscellaneous changes, like changes to expression overflowing: https://github.com/rust-lang/rustfmt/pull/6260). Eagerly updating rustfmt and formatting-the-world will ideally move some of the pressure off of the beta bump which will happen at the beginning of the next release cycle.
You can verify this is correct by checking out the changes, reverting the last commit, reapplying them, and diffing the changes:
```
git fetch git@github.com:compiler-errors/rust.git bump
git checkout -b bump FETCH_HEAD
git reset --hard HEAD~5
./x.py fmt --all
git diff FETCH_HEAD
# ignore the changes to stage0, and rustfmt.toml,
# and test file changes in rustdoc-js-std, run-make.
```
Or just take my word for it? Up to the reviewer.
r? release
Remove macOS 10.10 dynamic linker bug workaround
Rust's current minimum macOS version is 10.12, so the hack can be removed. This PR also updates the `remove_dir_all` docs to reflect that all supported macOS versions are protected against TOCTOU race conditions (the fallback implementation was already removed in #127683).
try-job: dist-x86_64-apple
try-job: dist-aarch64-apple
try-job: dist-apple-various
try-job: aarch64-apple
try-job: x86_64-apple-1
It's crazy to have the integer methods in something close to random
order.
The reordering makes the gaps clear: `const_i64`, `const_i128`,
`const_isize`, and `const_u16`. I guess they just aren't needed.
In `get_fn` there is a complicated set of if/elses to determine if
`hidden` visibility should be applied. There are five calls to
`LLVMRustSetVisibility` and some repetition in the comments.
This commit streamlines it a bit:
- Computes `hidden` and then uses it to determine if a single call to
`LLVMRustSetVisibility` occurs.
- Converts some of the if/elses into boolean expressions.
- Removes the repetitive comments.
Overall this makes it quite a bit shorter, and I find it easier to read.
Supertraits of `BuilderMethods` are all called `XyzBuilderMethods`.
Supertraits of `CodegenMethods` are all called `XyzMethods`. This commit
changes the latter to `XyzCodegenMethods`, for consistency.
It has `Backend` and `Deref` boudns, plus an associated type
`CodegenCx`, and it has a single use. This commit "inlines" it into
`BuilderMethods`, which makes the complicated backend trait situation a
little simpler.
Use -0.0 in `intrinsics::simd::reduce_add_unordered`
-0.0 is the actual neutral additive float, not +0.0, and this matters to codegen.
try-job: aarch64-gnu
deprecate -Csoft-float because it is unsound (and not fixable)
See https://github.com/rust-lang/rust/issues/129893 for details. The general sentiment there seems to be that this flag has no use and sound alternatives exist, so let's add this warning and see if anyone out there disagrees.
Also show a different warning on targets where it does nothing (as documented since https://github.com/rust-lang/rust/pull/36261): it seems to correspond to `-mfloat-abi` in GCC/clang, which is an ARM-specific option. To be really sure it does nothing, only forward the flag to LLVM for eabihf targets. This should not change behavior but makes me sleep better ;)
Don't leave debug locations for constants sitting on the builder indefinitely
Because constants are currently emitted *before* the prologue, leaving the debug location on the IRBuilder spills onto other instructions in the prologue and messes up both line numbers as well as the point LLVM chooses to be the prologue end.
Example LLVM IR (irrelevant IR elided):
Before:
```
define internal { i64, i64 } `@_ZN3tmp3Foo18var_return_opt_try17he02116165b0fc08cE(ptr` align 8 %self) !dbg !347 { start:
%self.dbg.spill = alloca [8 x i8], align 8
%_0 = alloca [16 x i8], align 8
%residual.dbg.spill = alloca [0 x i8], align 1
#dbg_declare(ptr %residual.dbg.spill, !353, !DIExpression(), !357)
store ptr %self, ptr %self.dbg.spill, align 8, !dbg !357
#dbg_declare(ptr %self.dbg.spill, !350, !DIExpression(), !358)
```
After:
```
define internal { i64, i64 } `@_ZN3tmp3Foo18var_return_opt_try17h00b17d08874ddd90E(ptr` align 8 %self) !dbg !347 { start:
%self.dbg.spill = alloca [8 x i8], align 8
%_0 = alloca [16 x i8], align 8
%residual.dbg.spill = alloca [0 x i8], align 1
#dbg_declare(ptr %residual.dbg.spill, !353, !DIExpression(), !357)
store ptr %self, ptr %self.dbg.spill, align 8
#dbg_declare(ptr %self.dbg.spill, !350, !DIExpression(), !358)
```
Note in particular how !357 from %residual.dbg.spill's dbg_declare no longer falls through onto the store to %self.dbg.spill. This fixes argument values at entry when the constant is a ZST (e.g. `<Option as Try>::Residual`). This fixes#130003 (but note that it does *not* fix issues with argument values and non-ZST constants, which emit their own stores that have debug info on them, like #128945).
r? `@michaelwoerister`
Simplify some nested `if` statements
Applies some but not all instances of `clippy::collapsible_if`. Some ended up looking worse afterwards, though, so I left those out. Also applies instances of `clippy::collapsible_else_if`
Review with whitespace disabled please.
Add -Z small-data-threshold
This flag allows specifying the threshold size above which LLVM should not consider placing small objects in a `.sdata` or `.sbss` section.
Support is indicated in the target options via the
small-data-threshold-support target option, which can indicate either an
LLVM argument or an LLVM module flag. To avoid duplicate specifications
in a large number of targets, the default value for support is
DefaultForArch, which is translated to a concrete value according to the
target's architecture.
This flag allows specifying the threshold size above which LLVM should
not consider placing small objects in a .sdata or .sbss section.
Support is indicated in the target options via the
small-data-threshold-support target option, which can indicate either an
LLVM argument or an LLVM module flag. To avoid duplicate specifications
in a large number of targets, the default value for support is
DefaultForArch, which is translated to a concrete value according to the
target's architecture.
s390x: Fix a regression related to backchain feature
In #127506, we introduced a new IBM Z-specific target feature, `backchain`.
This particular `target-feature` was available as a function-level attribute in LLVM 17 and below, so some hacks were used to avoid blowing up LLVM when querying the supported LLVM features.
This led to an unfortunate regression where `cfg!(target-feature = "backchain")` will always return true.
This pull request aims to fix this issue, and a test has been introduced to ensure it will never happen again.
Fixes#129927.
r? `@RalfJung`
Do not request sanitizers for naked functions
Naked functions can only contain inline asm, so any instrumentation inserted by sanitizers is illegal. Don't request it.
Fixes https://github.com/rust-lang/rust/issues/129224.
Because constants are currently emitted *before* the prologue, leaving the
debug location on the IRBuilder spills onto other instructions in the prologue
and messes up both line numbers as well as the point LLVM chooses to be the
prologue end.
Example LLVM IR (irrelevant IR elided):
Before:
define internal { i64, i64 } @_ZN3tmp3Foo18var_return_opt_try17he02116165b0fc08cE(ptr align 8 %self) !dbg !347 {
start:
%self.dbg.spill = alloca [8 x i8], align 8
%_0 = alloca [16 x i8], align 8
%residual.dbg.spill = alloca [0 x i8], align 1
#dbg_declare(ptr %residual.dbg.spill, !353, !DIExpression(), !357)
store ptr %self, ptr %self.dbg.spill, align 8, !dbg !357
#dbg_declare(ptr %self.dbg.spill, !350, !DIExpression(), !358)
After:
define internal { i64, i64 } @_ZN3tmp3Foo18var_return_opt_try17h00b17d08874ddd90E(ptr align 8 %self) !dbg !347 {
start:
%self.dbg.spill = alloca [8 x i8], align 8
%_0 = alloca [16 x i8], align 8
%residual.dbg.spill = alloca [0 x i8], align 1
#dbg_declare(ptr %residual.dbg.spill, !353, !DIExpression(), !357)
store ptr %self, ptr %self.dbg.spill, align 8
#dbg_declare(ptr %self.dbg.spill, !350, !DIExpression(), !358)
Note in particular how !357 from %residual.dbg.spill's dbg_declare no longer
falls through onto the store to %self.dbg.spill. This fixes argument values
at entry when the constant is a ZST (e.g. <Option as Try>::Residual). This
fixes#130003 (but note that it does *not* fix issues with argument values and
non-ZST constants, which emit their own stores that have debug info on them,
like #128945).
Make `Ty::boxed_ty` return an `Option`
Looks like a good place to use Rust's type system.
---
Most of 4ac7bcbaad/compiler/rustc_middle/src/ty/sty.rs (L971-L1963) looks like it could be moved to `TyKind` (then I guess `Ty` should be made to deref to `TyKind`).
Don't emit `expect`/`assume` in opt-level=0
LLVM does not make use of expect/assume calls in `opt-level=0`, so we can simplify IR by not emitting them in this case.
rustc_target: Add various aarch64 features
Add various aarch64 features already supported by LLVM and Linux.
Additionally include some comment fixes to ensure consistency of feature names with the Arm ARM.
Compiler support for features added to stdarch by https://github.com/rust-lang/stdarch/pull/1614.
Tracking issue for unstable aarch64 features is https://github.com/rust-lang/rust/issues/127764.
List of added features:
- FEAT_CSSC
- FEAT_ECV
- FEAT_FAMINMAX
- FEAT_FLAGM2
- FEAT_FP8
- FEAT_FP8DOT2
- FEAT_FP8DOT4
- FEAT_FP8FMA
- FEAT_HBC
- FEAT_LSE128
- FEAT_LSE2
- FEAT_LUT
- FEAT_MOPS
- FEAT_LRCPC3
- FEAT_SVE_B16B16
- FEAT_SVE2p1
- FEAT_WFxT
- FEAT_SME
- FEAT_SME_F16F16
- FEAT_SME_F64F64
- FEAT_SME_F8F16
- FEAT_SME_F8F32
- FEAT_SME_FA64
- FEAT_SME_I16I64
- FEAT_SME_LUTv2
- FEAT_SME2
- FEAT_SME2p1
- FEAT_SSVE_FP8DOT2
- FEAT_SSVE_FP8DOT4
- FEAT_SSVE_FP8FMA
FEAT_FPMR is added in the first commit and then removed in a separate one to highlight it being removed from upstream LLVM 19. The intention is for it to be detectable at runtime through stdarch but not have a corresponding Rust compile-time feature.
LLVM uses the word "code" to refer to a particular kind of coverage mapping.
This unrelated usage of the word is confusing, and makes it harder to introduce
types whose names correspond to the LLVM classification of coverage kinds.
