ABI-required target features: warn when they are missing in base CPU
Part of https://github.com/rust-lang/rust/pull/135408:
instead of adding ABI-required features to the target we build for LLVM, check that they are already there. Crucially we check this after applying `-Ctarget-cpu` and `-Ctarget-feature`, by reading `sess.unstable_target_features`. This means we can tweak the ABI target feature check without changing the behavior for any existing user; they will get warnings but the target features behave as before.
The test changes here show that we are un-doing the "add all required target features" part. Without the full #135408, there is no way to take a way an ABI-required target feature with `-Ctarget-cpu`, so we cannot yet test that part.
Cc ``@workingjubilee``
This improves the codegen for vector `select`, `gather`, `scatter` and
boolean reduction intrinsics and fixesrust-lang/portable-simd#316.
The current behavior of most mask operations during llvm codegen is to
truncate the mask vector to <N x i1>, telling llvm to use the least
significat bit. The exception is the `simd_bitmask` intrinsics, which
already used the most signifiant bit.
Since sse/avx instructions are defined to use the most significant bit,
truncating means that llvm has to insert a left shift to move the bit
into the most significant position, before the mask can actually be
used.
Similarly on aarch64, mask operations like blend work bit by bit,
repeating the least significant bit across the whole lane involves
shifting it into the sign position and then comparing against zero.
By shifting before truncating to <N x i1>, we tell llvm that we only
consider the most significant bit, removing the need for additional
shift instructions in the assembly.
Add `#[optimize(none)]`
cc #54882
This extends the `optimize` attribute to add `none`, which corresponds to the LLVM `OptimizeNone` attribute.
Not sure if an MCP is required for this, happy to file one if so.
Separate Builder methods from tcx
As part of the autodiff upstreaming we noticed, that it would be nice to have various builder methods available without the TypeContext, which prevents the normal CodegenCx to be passed around between threads.
We introduce a SimpleCx which just owns the llvm module and llvm context, to encapsulate them.
The previous CodegenCx now implements deref and forwards access to the llvm module or context to it's SimpleCx sub-struct. This gives us a bit more flexibility, because now we can pass (or construct) the SimpleCx in locations where we don't have enough information to construct a CodegenCx, or are not able to pass it around due to the tcx lifetimes (and it not implementing send/sync).
This also introduces an SBuilder, similar to the SimpleCx. The SBuilder uses a SimpleCx, whereas the existing Builder uses the larger CodegenCx. I will push updates to make implementations generic (where possible) to be implemented once and work for either of the two. I'll also clean up the leftover code.
`call` is a bit tricky, because it requires a tcx, I probably need to duplicate it after all.
Tracking:
- https://github.com/rust-lang/rust/issues/124509
This reflects the fact that we can't compute meaningful info for a function
that wasn't instrumented and therefore doesn't have `function_coverage_info`.
Make our `DIFlags` match `LLVMDIFlags` in the LLVM-C API
In order to be able to use a mixture of LLVM-C and C++ bindings for debuginfo, our Rust-side `DIFlags` needs to have the same layout as LLVM-C's `LLVMDIFlags`, and we also need to be able to convert it to the `DIFlags` accepted by LLVM's C++ API.
Internally, LLVM converts between the two types with a simple cast. We can't necessarily rely on that always being true, and LLVM doesn't expose a conversion function, so we have two potential options:
- Convert each bit/subvalue individually
- Statically assert that doing a cast is actually fine
As long as both types do remain the same under the hood (which seems likely), the static-assert-and-cast approach is easier and faster. If the static assertions ever start failing against some future version of LLVM, we'll have to switch over to the convert-each-subvalue approach, which is a bit more error-prone.
---
Extracted from #134009, though this PR ended up choosing the static-assert-and-cast approach over the convert-each-subvalue approach.
remove support for the (unstable) #[start] attribute
As explained by `@Noratrieb:`
`#[start]` should be deleted. It's nothing but an accidentally leaked implementation detail that's a not very useful mix between "portable" entrypoint logic and bad abstraction.
I think the way the stable user-facing entrypoint should work (and works today on stable) is pretty simple:
- `std`-using cross-platform programs should use `fn main()`. the compiler, together with `std`, will then ensure that code ends up at `main` (by having a platform-specific entrypoint that gets directed through `lang_start` in `std` to `main` - but that's just an implementation detail)
- `no_std` platform-specific programs should use `#![no_main]` and define their own platform-specific entrypoint symbol with `#[no_mangle]`, like `main`, `_start`, `WinMain` or `my_embedded_platform_wants_to_start_here`. most of them only support a single platform anyways, and need cfg for the different platform's ways of passing arguments or other things *anyways*
`#[start]` is in a super weird position of being neither of those two. It tries to pretend that it's cross-platform, but its signature is a total lie. Those arguments are just stubbed out to zero on ~~Windows~~ wasm, for example. It also only handles the platform-specific entrypoints for a few platforms that are supported by `std`, like Windows or Unix-likes. `my_embedded_platform_wants_to_start_here` can't use it, and neither could a libc-less Linux program.
So we have an attribute that only works in some cases anyways, that has a signature that's a total lie (and a signature that, as I might want to add, has changed recently, and that I definitely would not be comfortable giving *any* stability guarantees on), and where there's a pretty easy way to get things working without it in the first place.
Note that this feature has **not** been RFCed in the first place.
*This comment was posted [in May](https://github.com/rust-lang/rust/issues/29633#issuecomment-2088596042) and so far nobody spoke up in that issue with a usecase that would require keeping the attribute.*
Closes https://github.com/rust-lang/rust/issues/29633
try-job: x86_64-gnu-nopt
try-job: x86_64-msvc-1
try-job: x86_64-msvc-2
try-job: test-various
When LLVM's location discriminator value limit is exceeded, emit locations with dummy spans instead of dropping them entirely
Dropping them fails `-Zverify-llvm-ir`.
Fixes#135332.
r? `@jieyouxu`
Revert most of #133194 (except the test and the comment fixes). Then refix
not emitting locations at all when the correct location discriminator value
exceeds LLVM's capacity.
Add gpu-kernel calling convention
The amdgpu-kernel calling convention was reverted in commit f6b21e90d1 (#120495 and https://github.com/rust-lang/rust-analyzer/pull/16463) due to inactivity in the amdgpu target.
Introduce a `gpu-kernel` calling convention that translates to `ptx_kernel` or `amdgpu_kernel`, depending on the target that rust compiles for.
Tracking issue: #135467
amdgpu target tracking issue: #135024
The amdgpu-kernel calling convention was reverted in commit
f6b21e90d1 due to inactivity in the amdgpu
target.
Introduce a `gpu-kernel` calling convention that translates to
`ptx_kernel` or `amdgpu_kernel`, depending on the target that rust
compiles for.
Rename `BitSet` to `DenseBitSet`
r? `@Mark-Simulacrum` as you requested this in https://github.com/rust-lang/rust/pull/134438#discussion_r1890659739 after such a confusion.
This PR renames `BitSet` to `DenseBitSet` to make it less obvious as the go-to solution for bitmap needs, as well as make its representation (and positives/negatives) clearer. It also expands the comments there to hopefully make it clearer when it's not a good fit, with some alternative bitsets types.
(This migrates the subtrees cg_gcc and clippy to use the new name in separate commits, for easier review by their respective owners, but they can obvs be squashed)
add `-Zmin-function-alignment`
tracking issue: https://github.com/rust-lang/rust/issues/82232
This PR adds the `-Zmin-function-alignment=<align>` flag, that specifies a minimum alignment for all* functions.
### Motivation
This feature is requested by RfL [here](https://github.com/rust-lang/rust/issues/128830):
> i.e. the equivalents of `-fmin-function-alignment` ([GCC](https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html#index-fmin-function-alignment_003dn), Clang does not support it) / `-falign-functions` ([GCC](https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html#index-falign-functions), [Clang](https://clang.llvm.org/docs/ClangCommandLineReference.html#cmdoption-clang1-falign-functions)).
>
> For the Linux kernel, the behavior wanted is that of GCC's `-fmin-function-alignment` and Clang's `-falign-functions`, i.e. align all functions, including cold functions.
>
> There is [`feature(fn_align)`](https://github.com/rust-lang/rust/issues/82232), but we need to do it globally.
### Behavior
The `fn_align` feature does not have an RFC. It was decided at the time that it would not be necessary, but maybe we feel differently about that now? In any case, here are the semantics of this flag:
- `-Zmin-function-alignment=<align>` specifies the minimum alignment of all* functions
- the `#[repr(align(<align>))]` attribute can be used to override the function alignment on a per-function basis: when `-Zmin-function-alignment` is specified, the attribute's value is only used when it is higher than the value passed to `-Zmin-function-alignment`.
- the target may decide to use a higher value (e.g. on x86_64 the minimum that LLVM generates is 16)
- The highest supported alignment in rust is `2^29`: I checked a bunch of targets, and they all emit the `.p2align 29` directive for targets that align functions at all (some GPU stuff does not have function alignment).
*: Only with `build-std` would the minimum alignment also be applied to `std` functions.
---
cc `@ojeda`
r? `@workingjubilee` you were active on the tracking issue
Adds `#[rustc_force_inline]` which is similar to always inlining but
reports an error if the inlining was not possible, and which always
attempts to inline annotated items, regardless of optimisation levels.
It can only be applied to free functions to guarantee that the MIR
inliner will be able to resolve calls.
llvm: Ignore error value that is always false
See llvm/llvm-project#121851
For LLVM 20+, this function (`renameModuleForThinLTO`) has no return value. For prior versions of LLVM, this never failed, but had a signature which allowed an error value people were handling.
`@rustbot` label: +llvm-main
r? `@nikic`
Wait a moment before approving while the llvm-main infrastructure picks it up.
Add support for wasm exception handling to Emscripten target
This is a draft because we need some additional setting for the Emscripten target to select between the old exception handling and the new exception handling. I don't know how to add a setting like that, would appreciate advice from Rust folks. We could maybe choose to use the new exception handling if `Ctarget-feature=+exception-handling` is passed? I tried this but I get errors from llvm so I'm not doing it right.
See llvm/llvm-project#121851
For LLVM 20+, this function (`renameModuleForThinLTO`) has no return
value. For prior versions of LLVM, this never failed, but had a
signature which allowed an error value people were handling.
Add a notion of "some ABIs require certain target features"
I think I finally found the right shape for the data and checks that I recently added in https://github.com/rust-lang/rust/pull/133099, https://github.com/rust-lang/rust/pull/133417, https://github.com/rust-lang/rust/pull/134337: we have a notion of "this ABI requires the following list of target features, and it is incompatible with the following list of target features". Both `-Ctarget-feature` and `#[target_feature]` are updated to ensure we follow the rules of the ABI. This removes all the "toggleability" stuff introduced before, though we do keep the notion of a fully "forbidden" target feature -- this is needed to deal with target features that are actual ABI switches, and hence are needed to even compute the list of required target features.
We always explicitly (un)set all required and in-conflict features, just to avoid potential trouble caused by the default features of whatever the base CPU is. We do this *before* applying `-Ctarget-feature` to maintain backward compatibility; this poses a slight risk of missing some implicit feature dependencies in LLVM but has the advantage of not breaking users that deliberately toggle ABI-relevant target features. They get a warning but the feature does get toggled the way they requested.
For now, our logic supports x86, ARM, and RISC-V (just like the previous logic did). Unsurprisingly, RISC-V is the nicest. ;)
As a side-effect this also (unstably) allows *enabling* `x87` when that is harmless. I used the opportunity to mark SSE2 as required on x86-64, to better match the actual logic in LLVM and because all x86-64 chips do have SSE2. This infrastructure also prepares us for requiring SSE on x86-32 when we want to use that for our ABI (and for float semantics sanity), see https://github.com/rust-lang/rust/issues/133611, but no such change is happening in this PR.
r? `@workingjubilee`
[Debuginfo] Force enum `DISCR_*` to `static const u64` to allow for inspection via LLDB
see [here](https://rust-lang.zulipchat.com/#narrow/channel/317568-t-compiler.2Fwg-debugging/topic/Revamping.20Debuginfo/near/486614878) for more info.
This change mainly helps `*-msvc` debugged with LLDB. Currently, LLDB cannot inspect `static` struct fields, so the intended visualization for enums is only borderline functional, and niche enums with ranges of discriminant cannot be determined at all .
LLDB *can* inspect `static const` values (though for whatever reason, non-enum/non-u64 consts don't work).
This change adds the `LLVMRustDIBuilderCreateQualifiedType` to the rust FFI layer to wrap the discr type with a `const` modifier, as well as forcing all generated integer enum `DISCR_*` values to be u64's. Those values will only ever be used by debugger visualizers anyway, so it shouldn't be a huge deal, but I left a fixme comment for it just in case.. The `tag` also still properly reflects the discriminant type, so no information is lost.
Range metadata was disabled for amdgpu due to a backend bug. I did not
encounter any problems when removing the workaround to enable range
metadata (tried compiling `core` and `alloc`), so I assume this has
been fixed in LLVM in the last years.
Remove the workaround to re-enable range metadata.
Pointers for variables all need to be in the same address space for
correct compilation. Therefore ensure that even if a global variable is
created in a different address space, it is casted to the default
address space before its value is used.
This is necessary for the amdgpu target and others where the default
address space for global variables is not 0.
For example `core` does not compile in debug mode when not casting the
address space to the default one because it tries to emit the following
(simplified) LLVM IR, containing a type mismatch:
```llvm
@alloc_0 = addrspace(1) constant <{ [6 x i8] }> <{ [6 x i8] c"bit.rs" }>, align 1
@alloc_1 = addrspace(1) constant <{ ptr }> <{ ptr addrspace(1) @alloc_0 }>, align 8
; ^ here a struct containing a `ptr` is needed, but it is created using a `ptr addrspace(1)`
```
For this to compile, we need to insert a constant `addrspacecast` before
we use a global variable:
```llvm
@alloc_0 = addrspace(1) constant <{ [6 x i8] }> <{ [6 x i8] c"bit.rs" }>, align 1
@alloc_1 = addrspace(1) constant <{ ptr }> <{ ptr addrspacecast (ptr addrspace(1) @alloc_0 to ptr) }>, align 8
```
As vtables are global variables as well, they are also created with an
`addrspacecast`. In the SSA backend, after a vtable global is created,
metadata is added to it. To add metadata, we need the non-casted global
variable. Therefore we strip away an addrspacecast if there is one, to
get the underlying global.
coverage: Store coverage source regions as `Span` until codegen (take 2)
This is an attempt to re-land #133418:
> Historically, coverage spans were converted into line/column coordinates during the MIR instrumentation pass.
> This PR moves that conversion step into codegen, so that coverage spans spend most of their time stored as Span instead.
> In addition to being conceptually nicer, this also reduces the size of coverage mappings in MIR, because Span is smaller than 4x u32.
That PR was reverted by #133608, because in some circumstances not covered by our test suite we were emitting coverage metadata that was causing `llvm-cov` to exit with an error (#133606).
---
The implementation here is *mostly* the same, but adapted for subsequent changes in the relevant code (e.g. #134163).
I believe that the changes in #134163 should be sufficient to prevent the problem that required the original PR to be reverted. But I haven't been able to reproduce the original breakage in a regression test, and the `llvm-cov` error message is extremely unhelpful, so I can't completely rule out the possibility of this breaking again.
r? jieyouxu (reviewer of the original PR)
In codegen, a used function with `FunctionCoverageInfo` but no mappings has
historically indicated a bug. However, that will no longer be the case after
moving some fallible span-processing steps into codegen.
Re-export more `rustc_span::symbol` things from `rustc_span`.
`rustc_span::symbol` defines some things that are re-exported from `rustc_span`, such as `Symbol` and `sym`. But it doesn't re-export some closely related things such as `Ident` and `kw`. So you can do `use rustc_span::{Symbol, sym}` but you have to do `use rustc_span::symbol::{Ident, kw}`, which is inconsistent for no good reason.
This commit re-exports `Ident`, `kw`, and `MacroRulesNormalizedIdent`, and changes many `rustc_span::symbol::` qualifiers to `rustc_span::`. This is a 300+ net line of code reduction, mostly because many files with two `use rustc_span` items can be reduced to one.
r? `@jieyouxu`
`rustc_span::symbol` defines some things that are re-exported from
`rustc_span`, such as `Symbol` and `sym`. But it doesn't re-export some
closely related things such as `Ident` and `kw`. So you can do `use
rustc_span::{Symbol, sym}` but you have to do `use
rustc_span::symbol::{Ident, kw}`, which is inconsistent for no good
reason.
This commit re-exports `Ident`, `kw`, and `MacroRulesNormalizedIdent`,
and changes many `rustc_span::symbol::` qualifiers in `compiler/` to
`rustc_span::`. This is a 200+ net line of code reduction, mostly
because many files with two `use rustc_span` items can be reduced to
one.
coverage: Dismantle `map_data.rs` by moving its responsibilities elsewhere
This is a series of incremental changes that combine to let us get rid of `coverageinfo/map_data.rs`, by moving all of its responsibilities into more appropriate places.
Some of the notable consequences are:
- We once again build the per-CGU file table on the fly while preparing individual covfun records, instead of building the whole table up-front. The up-front approach was introduced by #117042 to work around various other problems in generating the covmap/covfun records, but subsequent cleanups have made that approach no longer necessary.
- Expression conversion and mapping-region conversion are now performed directly in `mapgen::covfun`, which should make future changes easier.
- We no longer insert unused function instances into the same map that is also used to track used function instances. This helps to decouple the handling of used vs unused functions.
---
There should be no meaningful change to compiler output. The file table is no longer sorted, because reordering it would invalidate the file indices stored in individual covfun records, but the table order should still be deterministic (albeit arbitrary).
There are some subsequent cleanups that I intend to investigate, but this is enough change for one PR.
This patch dismantles what was left of `FunctionCoverage` in `map_data.rs`,
replaces `function_coverage_map` with a set, and overhauls how we prepare
covfun records for unused functions.
The checks in `is_eligible_for_coverage` include `is_fn_like`, but will also
exclude various function-like things that cannot possibly have coverage
instrumentation.
reject aarch64 target feature toggling that would change the float ABI
~~Stacked on top of https://github.com/rust-lang/rust/pull/133099. Only the last two commits are new.~~
The first new commit lays the groundwork for separately controlling whether a feature may be enabled or disabled. The second commit uses that to make it illegal to *disable* the `neon` feature (which is only possible via `-Ctarget-feature`, and so the new check just adds a warning). Enabling the `neon` feature remains allowed on targets that don't disable `neon` or `fp-armv8`, which is all our built-in targets. This way, the entire PR is not a breaking change.
Fixes https://github.com/rust-lang/rust/issues/131058 for hardfloat targets (together with https://github.com/rust-lang/rust/pull/133102 which fixed it for softfloat targets).
Part of https://github.com/rust-lang/rust/issues/116344.
coverage: Tidy up creation of covmap and covfun records
This is a small follow-up to #134163 that mostly just inlines and renames some variables, and adds a few comments.
It also slightly defers the creation of the LLVM value that holds the filename table, to just before the value is needed.
---
try-job: x86_64-mingw-2
try-job: dist-x86_64-linux
forbid toggling x87 and fpregs on hard-float targets
Part of https://github.com/rust-lang/rust/issues/116344, follow-up to https://github.com/rust-lang/rust/pull/129884:
The `x87` target feature on x86 and the `fpregs` target feature on ARM must not be disabled on a hardfloat target, as that would change the float ABI. However, *enabling* `fpregs` on ARM is [explicitly requested](https://github.com/rust-lang/rust/issues/130988) as it seems to be useful. Therefore, we need to refine the distinction of "forbidden" target features and "allowed" target features: all (un)stable target features can determine on a per-target basis whether they should be allowed to be toggled or not. `fpregs` then checks whether the current target has the `soft-float` feature, and if yes, `fpregs` is permitted -- otherwise, it is not. (Same for `x87` on x86).
Also fixes https://github.com/rust-lang/rust/issues/132351. Since `fpregs` and `x87` can be enabled on some builds and disabled on others, it would make sense that one can query it via `cfg`. Therefore, I made them behave in `cfg` like any other unstable target feature.
The first commit prepares the infrastructure, but does not change behavior. The second commit then wires up `fpregs` and `x87` with that new infrastructure.
r? `@workingjubilee`
In the LLVM-C API, boolean values are passed as `typedef int LLVMBool`, but our
Rust-side typedef was using `c_uint` instead.
Signed and unsigned integers have the same ABI on most platforms, but that
isn't universally true, so we should prefer to be consistent with LLVM.
Pass end position of span through inline ASM cookie
Before this PR, only the start position of the span was passed though the inline ASM cookie to diagnostics. LLVM 19 has full support for 64-bit inline ASM cookies; this PR uses that to pass the end position of the span in the upper 32 bits, meaning inline ASM diagnostics now point at the entire line the error occurred on, not just the first character of it.
codegen `#[naked]` functions using global asm
tracking issue: https://github.com/rust-lang/rust/issues/90957Fixes#124375
This implements the approach suggested in the tracking issue: use the existing global assembly infrastructure to emit the body of `#[naked]` functions. The main advantage is that we now have full control over what gets generated, and are no longer dependent on LLVM not sneakily messing with our output (inlining, adding extra instructions, etc).
I discussed this approach with `@Amanieu` and while I think the general direction is correct, there is probably a bunch of stuff that needs to change or move around here. I'll leave some inline comments on things that I'm not sure about.
Combined with https://github.com/rust-lang/rust/pull/127853, if both accepted, I think that resolves all steps from the tracking issue.
r? `@Amanieu`
coverage: Rearrange the code for embedding per-function coverage metadata
This is a series of refactorings to the code that prepares and embeds per-function coverage metadata records (“covfun records”) in the `__llvm_covfun` linker section of the final binary. The `llvm-cov` tool reads this metadata from the binary when preparing a coverage report.
Beyond general cleanup, a big motivation behind these changes is to pave the way for re-landing an updated version of #133418.
---
There should be no change in compiler output, as demonstrated by the absence of (meaningful) changes to coverage tests.
The first patch is just moving code around, so I suggest looking at the other patches to see the actual changes.
---
try-job: x86_64-gnu
try-job: x86_64-msvc
try-job: aarch64-apple
rustc_target: ppc64 target string fixes for LLVM 20
LLVM continues to clean these up, and we continue to make this consistent. This is similar to 9caced7bad, e985396145, and
a10e744faf.
```@rustbot``` label: +llvm-main
Add the `power8-crypto` target feature
Add the `power8-crypto` target feature. This will enable adding some new PPC intrinsics in stdarch (specifically AES, SHA and CLMUL intrinsics). The implied target feature is from [here](https://github.com/llvm/llvm-project/blob/main/llvm/lib/Target/PowerPC/PPC.td)
```@rustbot``` label A-target-feature O-PowerPC
LLVM continues to clean these up, and we continue to make this
consistent. This is similar to 9caced7bad,
e985396145, and
a10e744faf.
`@rustbot` label: +llvm-main
coverage: Use a query to find counters/expressions that must be zero
As of #133446, this query (`coverage_ids_info`) determines which counter/expression IDs are unused. So with only a little extra work, we can take the code that was using that information to determine which coverage counters/expressions must be zero, and move that inside the query as well.
There should be no change in compiler output.
A bunch of cleanups
These are all extracted from a branch I have to get rid of driver queries. Most of the commits are not directly necessary for this, but were found in the process of implementing the removal of driver queries.
Previous PR: https://github.com/rust-lang/rust/pull/132410
This query (`coverage_ids_info`) already determines which counter/expression
IDs are unused, so it only takes a little extra effort to also determine which
counters/expressions must have a value of zero.
It was inconsistently done (sometimes even within a single function) and
most of the rest of the compiler uses fatal errors instead, which need
to be caught using catch_with_exit_code anyway. Using fatal errors
instead of ErrorGuaranteed everywhere in the driver simplifies things a
bit.
LLVM does not include an implementation of the va_arg instruction for
Xtensa. From what I understand, this is a conscious decision and
instead language frontends are encouraged to implement it themselves.
The rationale seems to be that loading values correctly requires
language and ABI-specific knowledge that LLVM lacks.
This is true of most architectures, and rustc already provides
implementation for a number of them. This commit extends the support to
include Xtensa.
See https://lists.llvm.org/pipermail/llvm-dev/2017-August/116337.html
for some discussion on the topic.
Unfortunately there does not seem to be a reference document for the
semantics of the va_list and va_arg on Xtensa. The most reliable source
is the GCC implementation, which this commit tries to follow. Clang also
provides its own compatible implementation.
This was tested for all the types that rustc allows in variadics.
Co-authored-by: Brian Tarricone <brian@tarricone.org>
Co-authored-by: Jonathan Bastien-Filiatrault <joe@x2a.org>
Co-authored-by: Paul Lietar <paul@lietar.net>
coverage: Use a query to identify which counter/expression IDs are used
Given that we already have a query to identify the highest-numbered counter ID in a MIR body, we can extend that query to also build bitsets of used counter/expression IDs. That lets us avoid some messy coverage bookkeeping during the main MIR traversal for codegen.
This does mean that we fail to treat some IDs as used in certain MIR-inlining scenarios, but I think that's fine, because it means that the results will be consistent across all instantiations of a function.
---
There's some more cleanup I want to do in the function coverage collector, since it isn't really collecting anything any more, but I'll leave that for future work.
Respect verify-llvm-ir option in the backend
We are currently unconditionally verifying the LLVM IR in the backend (twice), ignoring the value of the verify-llvm-ir option. This has substantial compile-time impact for debug builds.
Support input/output in vector registers of PowerPC inline assembly
This extends currently clobber-only vector registers (`vreg`) support to allow passing `#[repr(simd)]` types as input/output.
| Architecture | Register class | Target feature | Allowed types |
| ------------ | -------------- | -------------- | -------------- |
| PowerPC | `vreg` | `altivec` | `i8x16`, `i16x8`, `i32x4`, `f32x4` |
| PowerPC | `vreg` | `vsx` | `f32`, `f64`, `i64x2`, `f64x2` |
In addition to floats and `core::simd` types listed above, `core::arch` types and custom `#[repr(simd)]` types of the same size and type are also allowed. All allowed types and relevant target features are currently unstable.
r? `@Amanieu`
`@rustbot` label +O-PowerPC +A-inline-assembly
This is currently handled automatically by the fact that codegen doesn't visit
coverage statements in unused functions, but that will no longer be the case
when unused IDs are identified by a separate query instead.
Enable -Zshare-generics for inline(never) functions
This avoids inlining cross-crate generic items when possible that are
already marked inline(never), implying that the author is not intending
for the function to be inlined by callers. As such, having a local copy
may make it easier for LLVM to optimize but mostly just adds to binary
bloat and codegen time. In practice our benchmarks indicate this is
indeed a win for larger compilations, where the extra cost in dynamic
linking to these symbols is diminished compared to the advantages in
fewer copies that need optimizing in each binary.
It might also make sense it expand this with other heuristics (e.g.,
`#[cold]`) in the future, but this seems like a good starting point.
FWIW, I expect that doing cleanup in where we make the decision
what should/shouldn't be shared is also a good idea. Way too
much code needed to be tweaked to check this. But I'm hoping
to leave that for a follow-up PR rather than blocking this on it.
This reduces code sizes and better respects programmer intent when
marking inline(never). Previously such a marking was essentially ignored
for generic functions, as we'd still inline them in remote crates.
coverage: Store coverage source regions as `Span` until codegen
Historically, coverage spans were converted into line/column coordinates during the MIR instrumentation pass.
This PR moves that conversion step into codegen, so that coverage spans spend most of their time stored as `Span` instead.
In addition to being conceptually nicer, this also reduces the size of coverage mappings in MIR, because `Span` is smaller than 4x u32.
---
There should be no changes to coverage output.
Support input/output in vector registers of s390x inline assembly (under asm_experimental_reg feature)
This extends currently clobber-only vector registers (`vreg`) support to allow passing `#[repr(simd)]` types, floats (f32/f64/f128), and integers (i32/i64/i128) as input/output.
This is unstable and gated under new `#![feature(asm_experimental_reg)]` (tracking issue: https://github.com/rust-lang/rust/issues/133416). If the feature is not enabled, only clober is supported as before.
| Architecture | Register class | Target feature | Allowed types |
| ------------ | -------------- | -------------- | -------------- |
| s390x | `vreg` | `vector` | `i32`, `f32`, `i64`, `f64`, `i128`, `f128`, `i8x16`, `i16x8`, `i32x4`, `i64x2`, `f32x4`, `f64x2` |
This matches the list of types that are supported by the vector registers in LLVM:
https://github.com/llvm/llvm-project/blob/llvmorg-19.1.0/llvm/lib/Target/SystemZ/SystemZRegisterInfo.td#L301-L313
In addition to `core::simd` types and floats listed above, custom `#[repr(simd)]` types of the same size and type are also allowed. All allowed types other than i32/f32/i64/f64/i128, and relevant target features are currently unstable.
Currently there is no SIMD type for s390x in `core::arch`, but this is tracked in https://github.com/rust-lang/rust/issues/130869.
cc https://github.com/rust-lang/rust/issues/130869 about vector facility support in s390x
cc https://github.com/rust-lang/rust/issues/125398 & https://github.com/rust-lang/rust/issues/116909 about f128 support in asm
`@rustbot` label +O-SystemZ +A-inline-assembly
Fix asm goto with outputs and move it to a separate feature gate
Tracking issue: #119364
This PR addresses 3 aspects of asm goto with outputs:
* Codegen is fixed. My initial implementation has an oversight which cause the output to be only stored in fallthrough path, but not in label blocks.
* Outputs can now be used with `options(noreturn)` if a label block is given.
* All of this is moved to a new feature gate, because we likely want to stabilise `asm_goto` before asm goto with outputs.
`@rustbot` labels: +A-inline-assembly +F-asm
When labels are present, the `noreturn` option really means that asm block
won't fallthrough -- if labels are present, then outputs can still be
meaningfully used.
A used function with no mappings has historically indicated a bug, but that
will no longer be the case after moving some fallible span-processing steps
into codegen.
Allow disabling ASan instrumentation for globals
AddressSanitizer adds instrumentation to global variables unless the [`no_sanitize_address`](https://llvm.org/docs/LangRef.html#global-attributes) attribute is set on them.
This commit extends the existing `#[no_sanitize(address)]` attribute to set this; previously it only had the desired effect on functions.
(cc https://github.com/rust-lang/rust/issues/39699)
The maximum discriminator value LLVM can currently encode is 2^12. If macro use
results in more than 2^12 calls to the same function attributed to the same
callsite, and those calls are MIR-inlined, we will require more than the maximum
discriminator value to completely represent the debug information. Once we reach
that point drop the debug info instead.
The test relies on the fact that inlining more than 2^12 calls at the same
callsite will trigger a panic (and after the following commit, a warning) due to
LLVM limitations but with collapse_debuginfo the callsites should not be the
same.
the behavior of the type system not only depends on the current
assumptions, but also the currentnphase of the compiler. This is
mostly necessary as we need to decide whether and how to reveal
opaque types. We track this via the `TypingMode`.
CFI: Append debug location to CFI blocks
Currently we're not appending debug locations to the inserted CFI blocks. This shows up in #132615 and #100783. This change fixes that by passing down the debug location to the CFI type-test generation and appending it to the blocks.
Credits also belong to `@jakos-sec` who worked with me on this.
Add a default implementation for CodegenBackend::link
As a side effect this should add raw-dylib support to cg_gcc as the default ArchiveBuilderBuilder that is used implements create_dll_import_lib. I haven't tested if the raw-dylib support actually works however.
coverage: Restrict empty-span expansion to only cover `{` and `}`
Coverage instrumentation has some tricky code for converting a coverage-relevant `Span` into a set of start/end line/byte-column coordinates that will be embedded in the CGU's coverage metadata.
A big part of this complexity is special code for handling empty spans, which are expanded into non-empty spans (if possible) because LLVM's coverage reporter does not handle empty spans well.
This PR simplifies that code by restricting it to only apply in two specific situations: when the character after the empty span is `{`, or the character before the empty span is `}`.
(As an added benefit, this means that the expanded spans no longer extend awkwardly beyond the end of a physical line, which was common under the previous implementation.)
Along the way, this PR also removes some unhelpful code for dealing with function source code spread across multiple files. Functions currently can't have coverage spans in multiple files, and if that ever changes (e.g. to properly support expansion regions) then this code will need to be completely overhauled anyway.
After link_binary the temporary files referenced by CodegenResults are
deleted, so calling link_binary again with the same CodegenResults
should not be allowed.
LLVM does not expect to ever see multiple dbg_declares for the same variable at the same
location with different values. proc-macros make it possible for arbitrary code,
including multiple calls that get inlined, to happen at any given location in the source
code. Add discriminators when that happens so these locations are different to LLVM.
This may interfere with the AddDiscriminators pass in LLVM, which is added by the
unstable flag -Zdebug-info-for-profiling.
Fixes#131944
Rollup of 5 pull requests
Successful merges:
- #132552 (Add v9, v8plus, and leoncasa target feature to sparc and use v8plus in create_object_file)
- #132745 (pointee_info_at: fix logic for recursing into enums)
- #132777 (try_question_mark_nop: update test for LLVM 20)
- #132785 (rustc_target: more target string fixes for LLVM 20)
- #132794 (Use a separate dir for r-a builds consistently in helix config)
r? `@ghost`
`@rustbot` modify labels: rollup
Trim and tidy includes in `rustc_llvm`
These includes tend to accumulate over time, and are usually only removed when something breaks in a new LLVM version, so it's nice to clean them up manually once in a while.
General strategy used for this PR:
- Remove all includes from `LLVMWrapper.h` that aren't needed by the header itself, transplanting them to individual source files as necessary.
- For each source file, temporarily remove each include if doing so doesn't cause a compile error.
- If a “required” include looks like it shouldn't be needed, try replacing it with its sub-includes, then trim that list.
- After doing all of the above, go back and re-add any removed include if the file does actually use things defined in that header, even if the header happens to also be included by something else.
Functions currently can't have mappings in multiple files, and if that ever
changes (e.g. to properly support expansion regions), this code will need to be
completely overhauled anyway.
We already had a dedicated `LocalFileId` index type, but previously we used a
raw `u32` for global file IDs, because index types were harder to pass through
FFI.
Simplify FFI calls for `-Ztime-llvm-passes` and `-Zprint-codegen-stats`
The existing code for these unstable LLVM-infodump flags was jumping through hoops to pass an allocated C string across the FFI boundary, when it's much simpler to just write to a `&RustString` instead.
coverage: Extract safe FFI wrapper functions to `llvm_cov`
This PR takes all of the inline `unsafe` calls in coverage codegen, and all the safe wrapper functions in `coverageinfo/mod.rs`, and moves them to a new `llvm_cov` submodule that is dedicated to safe FFI wrapper functions. This reduces the mixing of abstraction levels in the rest of coverage codegen.
As a follow-up, this PR also tidies up the names and signatures of several of the coverage FFI functions.
Set "symbol name" in raw-dylib import libraries to the decorated name
`windows-rs` received a bug report that mixing raw-dylib generated and the Windows SDK import libraries was causing linker failures: <https://github.com/microsoft/windows-rs/issues/3285>
The root cause turned out to be #124958, that is we are not including the decorated name in the import library and so the import name type is also not being correctly set.
This change modifies the generation of import libraries to set the "symbol name" to the fully decorated name and correctly marks the import as being data vs function.
Note that this also required some changes to how the symbol is named within Rust: for MSVC we now need to use the decorated name but for MinGW we still need to use partially decorated (or undecorated) name.
Fixes#124958
Passing i686 MSVC and MinGW build: <https://github.com/rust-lang/rust/actions/runs/11000433888?pr=130586>
r? `@ChrisDenton`
Add a new `wide-arithmetic` feature for WebAssembly
This commit adds a new rustc target feature named `wide-arithmetic` for WebAssembly targets. This corresponds to the [wide-arithmetic] proposal for WebAssembly which adds new instructions catered towards accelerating integer arithmetic larger than 64-bits. This proposal to WebAssembly is not standard yet so this new feature is flagged as an unstable target feature. Additionally Rust's LLVM version doesn't support this new feature yet since support will first be added in LLVM 20, so the feature filtering logic for LLVM is updated to handle this.
I'll also note that I'm not currently planning to add wasm-specific intrinsics to `std::arch::wasm32` at this time. The currently proposed instructions are all accessible through `i128` or `u128`-based operations which Rust already supports, so intrinsic shouldn't be necessary to get access to these new instructions.
[wide-arithmetic]: https://github.com/WebAssembly/wide-arithmetic
rustc_codegen_llvm: Add a new 'pc' option to branch-protection
Add a new 'pc' option to -Z branch-protection for aarch64 that enables the use of PC as a diversifier in PAC branch protection code.
When the pauth-lr target feature is enabled in combination with -Z branch-protection=pac-ret,pc, the new 9.5-a instructions (pacibsppc, retaasppc, etc) will be generated.
mark some target features as 'forbidden' so they cannot be (un)set with -Ctarget-feature
The context for this is https://github.com/rust-lang/rust/issues/116344: some target features change the way floats are passed between functions. Changing those target features is unsound as code compiled for the same target may now use different ABIs.
So this introduces a new concept of "forbidden" target features (on top of the existing "stable " and "unstable" categories), and makes it a hard error to (un)set such a target feature. For now, the x86 and ARM feature `soft-float` is on that list. We'll have to make some effort to collect more relevant features, and similar features from other targets, but that can happen after the basic infrastructure for this landed. (These features are being collected in https://github.com/rust-lang/rust/issues/131799.)
I've made this a warning for now to give people some time to speak up if this would break something.
MCP: https://github.com/rust-lang/compiler-team/issues/780
Support clobber_abi and vector registers (clobber-only) in PowerPC inline assembly
This supports `clobber_abi` which is one of the requirements of stabilization mentioned in #93335.
This basically does a similar thing I did in https://github.com/rust-lang/rust/pull/130630 to implement `clobber_abi` for s390x, but for powerpc/powerpc64/powerpc64le.
- This also supports vector registers (as `vreg`) as clobber-only, which need to support clobbering of them to implement `clobber_abi`.
- `vreg` should be able to accept `#[repr(simd)]` types as input/output if the unstable `altivec` target feature is enabled, but `core::arch::{powerpc,powerpc64}` vector types, `#[repr(simd)]`, and `core::simd` are all unstable, so the fact that this is currently a clobber-only should not be considered a blocker of clobber_abi implementation or stabilization. So I have not implemented it in this PR.
- See https://github.com/rust-lang/rust/pull/131551 (which is based on this PR) for a PR to implement this.
- (I'm not sticking to whether that PR should be a separate PR or part of this PR, so I can merge that PR into this PR if needed.)
Refs:
- PPC32 SysV: Section "Function Calling Sequence" in [System V Application Binary Interface PowerPC Processor Supplement](https://refspecs.linuxfoundation.org/elf/elfspec_ppc.pdf)
- PPC64 ELFv1: Section 3.2 "Function Calling Sequence" in [64-bit PowerPC ELF Application Binary Interface Supplement](https://refspecs.linuxfoundation.org/ELF/ppc64/PPC-elf64abi.html#FUNC-CALL)
- PPC64 ELFv2: Section 2.2 "Function Calling Sequence" in [64-Bit ELF V2 ABI Specification](https://openpowerfoundation.org/specifications/64bitelfabi/)
- AIX: [Register usage and conventions](https://www.ibm.com/docs/en/aix/7.3?topic=overview-register-usage-conventions), [Special registers in the PowerPC®](https://www.ibm.com/docs/en/aix/7.3?topic=overview-special-registers-in-powerpc), [AIX vector programming](https://www.ibm.com/docs/en/aix/7.3?topic=concepts-aix-vector-programming)
- Register definition in LLVM: https://github.com/llvm/llvm-project/blob/llvmorg-19.1.0/llvm/lib/Target/PowerPC/PPCRegisterInfo.td#L189
If I understand the above four ABI documentations correctly, except for the PPC32 SysV's VR (Vector Registers) and 32-bit AIX (currently not supported by rustc)'s r13, there does not appear to be important differences in terms of implementing `clobber_abi`:
- The above four ABIs are consistent about FPR (0-13: volatile, 14-31: nonvolatile), CR (0-1,5-7: volatile, 2-4: nonvolatile), XER (volatile), and CTR (volatile).
- As for GPR, only the registers we are treating as reserved are slightly different
- r0, r3-r12 are volatile
- r1(sp, reserved), r14-31 are nonvolatile
- r2(reserved) is TOC pointer in PPC64 ELF/AIX, system-reserved register in PPC32 SysV (AFAIK used as thread pointer in Linux/BSDs)
- r13(reserved for non-32-bit-AIX) is thread pointer in PPC64 ELF, small data area pointer register in PPC32 SysV, "reserved under 64-bit environment; not restored across system calls[^r13]" in AIX)
- As for FPSCR, volatile in PPC64 ELFv1/AIX, some fields are volatile only in certain situations (rest are volatile) in PPC32 SysV/PPC64 ELFv2.
- As for VR (Vector Registers), it is not mentioned in PPC32 SysV, v0-v19 are volatile in both in PPC64 ELF/AIX, v20-v31 are nonvolatile in PPC64 ELF, reserved or nonvolatile depending on the ABI ([vec-extabi vs vec-default in LLVM](https://reviews.llvm.org/D89684), we are [using vec-extabi](https://github.com/rust-lang/rust/pull/131341#discussion_r1797693299)) in AIX:
> When the default Vector enabled mode is used, these registers are reserved and must not be used.
> In the extended ABI vector enabled mode, these registers are nonvolatile and their values are preserved across function calls
I left [FIXME comment about PPC32 SysV](https://github.com/rust-lang/rust/pull/131341#discussion_r1790496095) and added ABI check for AIX.
- As for VRSAVE, it is not mentioned in PPC32 SysV, nonvolatile in PPC64 ELFv1, reserved in PPC64 ELFv2/AIX
- As for VSCR, it is not mentioned in PPC32 SysV/PPC64 ELFv1, some fields are volatile only in certain situations (rest are volatile) in PPC64 ELFv2, volatile in AIX
We are currently treating r1-r2, r13 (non-32-bit-AIX), r29-r31, LR, CTR, and VRSAVE as reserved.
We are currently not processing anything about FPSCR and VSCR, but I feel those are things that should be processed by `preserves_flags` rather than `clobber_abi` if we need to do something about them. (However, PPCRegisterInfo.td in LLVM does not seem to define anything about them.)
Replaces #111335 and #124279
cc `@ecnelises` `@bzEq` `@lu-zero`
r? `@Amanieu`
`@rustbot` label +O-PowerPC +A-inline-assembly
[^r13]: callee-saved, according to [LLVM](6a6af0246b/llvm/lib/Target/PowerPC/PPCCallingConv.td (L322)) and [GCC](a9173a50e7/gcc/config/rs6000/rs6000.h (L859)).
Reduce dependence on the target name
The target name can be anything with custom target specs. Matching on fields inside the target spec is much more robust than matching on the target name.
Also remove the unused is_builtin target spec field.
