Fix#90546 by filtering out global value function pointer types from the
type tests, and adding the LowerTypeTests pass to the rustc LTO
optimization pipelines.
Add a new `compare_bytes` intrinsic instead of calling `memcmp` directly
As discussed in #113435, this lets the backends be the place that can have the "don't call the function if n == 0" logic, if it's needed for the target. (I didn't actually *add* those checks, though, since as I understood it we didn't actually need them on known targets?)
Doing this also let me make it `const` (unstable), which I don't think `extern "C" fn memcmp` can be.
cc `@RalfJung` `@Amanieu`
This deduplicates some logic and makes it easier to follow what wrappers
are produced. In the future it may allow moving the code to determine
which wrappers to create to cg_ssa.
cg_llvm: stop identifying ADTs in LLVM IR
This is an extension of https://github.com/rust-lang/rust/pull/94107. It may be a minor perf win.
Fixes#96242.
Now that we use opaque pointers, ADTs can no longer be recursive, so we
do not need to name them. Previously, this would be necessary if you had
a struct like
```rs
struct Foo(Box<Foo>, u64, u64);
```
which would be represented with something like
```ll
%Foo = type { %Foo*, i64, i64 }
```
which is now just
```ll
{ ptr, i64, i64 }
```
r? `@tmiasko`
Coverage FFI types were historically split across two modules, because some of
them were needed by code in `rustc_codegen_ssa`.
Now that all of the coverage codegen code has been moved into
`rustc_codegen_llvm` (#113355), it's possible to move all of the FFI types into
a single module, making it easier to see all of them at once.
Filter out short-lived LLVM diagnostics before they reach the rustc handler
During profiling I saw remark passes being unconditionally enabled: for example `Machine Optimization Remark Emitter`.
The diagnostic remarks enabled by default are [from missed optimizations and opt analyses](https://github.com/rust-lang/rust/pull/113339#discussion_r1259480303). They are created by LLVM, passed to the diagnostic handler on the C++ side, emitted to rust, where they are unpacked, C++ strings are converted to rust, etc.
Then they are discarded in the vast majority of the time (i.e. unless some kind of `-Cremark` has enabled some of these passes' output to be printed).
These unneeded allocations are very short-lived, basically only lasting between the LLVM pass emitting them and the rust handler where they are discarded. So it doesn't hugely impact max-rss, and is only a slight reduction in instruction count (cachegrind reports a reduction between 0.3% and 0.5%) _on linux_. It's possible that targets without `jemalloc` or with a worse allocator, may optimize these less.
It is however significant in the aggregate, looking at the total number of allocated bytes:
- it's the biggest source of allocations according to dhat, on the benchmarks I've tried e.g. `syn` or `cargo`
- allocations on `syn` are reduced by 440MB, 17% (from 2440722647 bytes total, to 2030461328 bytes)
- allocations on `cargo` are reduced by 6.6GB, 19% (from 35371886402 bytes total, to 28723987743 bytes)
Some of these diagnostics objects [are allocated in LLVM](https://github.com/rust-lang/rust/pull/113339#discussion_r1252387484) *before* they're emitted to our diagnostic handler, where they'll be filtered out. So we could remove those in the future, but that will require changing a few LLVM call-sites upstream, so I left a FIXME.
now that remarks are filtered before cg_llvm's diagnostic handler callback
is called, we don't need to do the filtering post c++-to-rust conversion
of the diagnostic.
cleanup: remove pointee types
This can't be merged until the oldest LLVM version we support uses opaque pointers, which will be the case after #114148. (Also note `-Cllvm-args="-opaque-pointers=0"` can technically be used in LLVM 15, though I don't think we should support that configuration.)
I initially hoped this would provide some minor perf win, but in https://github.com/rust-lang/rust/pull/105412#issuecomment-1341224450 it had very little impact, so this is only valuable as a cleanup.
As a followup, this will enable #96242 to be resolved.
r? `@ghost`
`@rustbot` label S-blocked
Operand types are now tracked explicitly, so there is no need to reserve ID 0
for the special always-zero counter.
As part of the renumbering, this change fixes an off-by-one error in the way
counters were counted by the `coverageinfo` query. As a result, functions
should now have exactly the number of counters they actually need, instead of
always having an extra counter that is never used.
Operand types are now tracked explicitly, so there is no need for expression
IDs to avoid counter IDs by descending from `u32::MAX`. Instead they can just
count up from 0, and can be used directly as indices when necessary.
Because the three kinds of operand are now distinguished explicitly, we no
longer need fiddly code to disambiguate counter IDs and expression IDs based on
the total number of counters/expressions in a function.
This does increase the size of operands from 4 bytes to 8 bytes, but that
shouldn't be a big deal since they are mostly stored inside boxed structures,
and the current coverage code is not particularly size-optimized anyway.
Now that we use opaque pointers, ADTs can no longer be recursive, so we
do not need to name them. Previously, this would be necessary if you had
a struct like
```rs
struct Foo(Box<Foo>, u64, u64);
```
which would be represented with something like
```ll
%Foo = type { %Foo*, i64, i64 }
```
which is now just
```ll
{ ptr, i64, i64 }
```
This section name is always constant for a given target, but obtaining it from
LLVM requires a few intermediate allocations. There's no need to do so
repeatedly from inside a per-function loop.
Both GCC and Clang write by default a `.comment` section with compiler
information:
```txt
$ gcc -c -xc /dev/null && readelf -p '.comment' null.o
String dump of section '.comment':
[ 1] GCC: (GNU) 11.2.0
$ clang -c -xc /dev/null && readelf -p '.comment' null.o
String dump of section '.comment':
[ 1] clang version 14.0.1 (https://github.com/llvm/llvm-project.git c62053979489ccb002efe411c3af059addcb5d7d)
```
They also implement the `-Qn` flag to avoid doing so:
```txt
$ gcc -Qn -c -xc /dev/null && readelf -p '.comment' null.o
readelf: Warning: Section '.comment' was not dumped because it does not exist!
$ clang -Qn -c -xc /dev/null && readelf -p '.comment' null.o
readelf: Warning: Section '.comment' was not dumped because it does not exist!
```
So far, `rustc` only does it for WebAssembly targets and only
when debug info is enabled:
```txt
$ echo 'fn main(){}' | rustc --target=wasm32-unknown-unknown --emit=llvm-ir -Cdebuginfo=2 - && grep llvm.ident rust_out.ll
!llvm.ident = !{!27}
```
In the RFC part of this PR it was decided to always add
the information, which gets us closer to other popular compilers.
An opt-out flag like GCC and Clang may be added later on if deemed
necessary.
Implementation-wise, this covers both `ModuleLlvm::new()` and
`ModuleLlvm::new_metadata()` cases by moving the addition to
`context::create_module` and adds a few test cases.
ThinLTO also sees the `llvm.ident` named metadata duplicated (in
temporary outputs), so this deduplicates it like it is done for
`wasm.custom_sections`. The tests also check this duplication does
not take place.
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
Prototype: Add unstable `-Z reference-niches` option
MCP: rust-lang/compiler-team#641
Relevant RFC: rust-lang/rfcs#3204
This prototype adds a new `-Z reference-niches` option, controlling the range of valid bit-patterns for reference types (`&T` and `&mut T`), thereby enabling new enum niching opportunities. Like `-Z randomize-layout`, this setting is crate-local; as such, references to built-in types (primitives, tuples, ...) are not affected.
The possible settings are (here, `MAX` denotes the all-1 bit-pattern):
| `-Z reference-niches=` | Valid range |
|:---:|:---:|
| `null` (the default) | `1..=MAX` |
| `size` | `1..=(MAX- size)` |
| `align` | `align..=MAX.align_down_to(align)` |
| `size,align` | `align..=(MAX-size).align_down_to(align)` |
------
This is very WIP, and I'm not sure the approach I've taken here is the best one, but stage 1 tests pass locally; I believe this is in a good enough state to unleash this upon unsuspecting 3rd-party code, and see what breaks.
Support `--print KIND=PATH` command line syntax
As is already done for `--emit KIND=PATH` and `-L KIND=PATH`.
In the discussion of #110785, it was pointed out that `--print KIND=PATH` is nicer than trying to apply the single global `-o` path to `--print`'s output, because in general there can be multiple print requests within a single rustc invocation, and anyway `-o` would already be used for a different meaning in the case of `link-args` and `native-static-libs`.
I am interested in using `--print cfg=PATH` in Buck2. Currently Buck2 works around the lack of support for `--print KIND=PATH` by [indirecting through a Python wrapper script](d43cf3a51a/prelude/rust/tools/get_rustc_cfg.py) to redirect rustc's stdout into the location dictated by the build system.
From skimming Cargo's usages of `--print`, it definitely seems like it would benefit from `--print KIND=PATH` too. Currently it is working around the lack of this by inserting `--crate-name=___ --print=crate-name` so that it can look for a line containing `___` as a delimiter between the 2 other `--print` informations it actually cares about. This is commented as a "HACK" and "abuse". 31eda6f7c3/src/cargo/core/compiler/build_context/target_info.rs (L242) (FYI `@weihanglo` as you dealt with this recently in https://github.com/rust-lang/cargo/pull/11633.)
Mentioning reviewers active in #110785: `@fee1-dead` `@jyn514` `@bjorn3`
Resurrect: rustc_llvm: Add a -Z `print-codegen-stats` option to expose LLVM statistics.
This resurrects PR https://github.com/rust-lang/rust/pull/104000, which has sat idle for a while. And I want to see the effect of stack-move optimizations on LLVM (like https://reviews.llvm.org/D153453) :).
I have applied the changes requested by `@oli-obk` and `@nagisa` https://github.com/rust-lang/rust/pull/104000#discussion_r1014625377 and https://github.com/rust-lang/rust/pull/104000#discussion_r1014642482 in the latest commits.
r? `@oli-obk`
-----
LLVM has a neat [statistics](https://llvm.org/docs/ProgrammersManual.html#the-statistic-class-stats-option) feature that tracks how often optimizations kick in. It's very handy for optimization work. Since we expose the LLVM pass timings, I thought it made sense to expose the LLVM statistics too.
-----
(Edit: fix broken link
(Edit2: fix segmentation fault and use malloc
If `rustc` is built with
```toml
[llvm]
assertions = true
```
Then you can see like
```
rustc +stage1 -Z print-codegen-stats -C opt-level=3 tmp.rs
===-------------------------------------------------------------------------===
... Statistics Collected ...
===-------------------------------------------------------------------------===
3 aa - Number of MayAlias results
193 aa - Number of MustAlias results
531 aa - Number of NoAlias results
...
```
And the current default build emits only
```
$ rustc +stage1 -Z print-codegen-stats -C opt-level=3 tmp.rs
===-------------------------------------------------------------------------===
... Statistics Collected ...
===-------------------------------------------------------------------------===
$
```
This might be better to emit the message to tell assertion flag necessity, but now I can't find how to do that...
Add the `no-builtins` attribute to functions when `no_builtins` is applied at the crate level.
**When `no_builtins` is applied at the crate level, we should add the `no-builtins` attribute to each function to ensure it takes effect in LTO.**
This is also the reason why no_builtins does not take effect in LTO as mentioned in #35540.
Now, `#![no_builtins]` should be similar to `-fno-builtin` in clang/gcc, see https://clang.godbolt.org/z/z4j6Wsod5.
Next, we should make `#![no_builtins]` participate in LTO again. That makes sense, as LTO also takes into consideration function-level instruction optimizations, such as the MachineOutliner. More importantly, when a user writes a large `#![no_builtins]` crate, they would like this crate to participate in LTO as well.
We should also add a function-level no_builtins attribute to allow users to have more control over it. This is similar to Clang's `__attribute__((no_builtin))` feature, see https://clang.godbolt.org/z/Wod6KK6eq. Before implementing this feature, maybe we should discuss whether to support more fine-grained control, such as `__attribute__((no_builtin("memcpy")))`.
Related discussions:
- #109821
- #35540
Next (a separate pull request?):
- [ ] Revert #35637
- [ ] Add a function-level `no_builtin` attribute?
Better diagnostics for dlltool errors.
When dlltool fails, show the full command that was executed. In particular, llvm-dlltool is not very helpful, printing a generic usage message rather than what actually went wrong, so stdout and stderr aren't of much use when troubleshooting.
When dlltool fails, show the full command that was executed. In
particular, llvm-dlltool is not very helpful, printing a generic usage
message rather than what actually went wrong, so stdout and stderr
aren't of much use when troubleshooting.
LLVM has a neat [statistics] feature that tracks how often optimizations kick
in. It's very handy for optimization work. Since we expose the LLVM pass
timings, I thought it made sense to expose the LLVM statistics too.
[statistics]: https://llvm.org/docs/ProgrammersManual.html#the-statistic-class-stats-option
Remove `LLVMRustCoverageHashCString`
Coverage has two FFI functions for computing the hash of a byte string. One takes a ptr/len pair (`LLVMRustCoverageHashByteArray`), and the other takes a NUL-terminated C string (`LLVMRustCoverageHashCString`).
But on closer inspection, the C string version is unnecessary. The calling-side code converts a Rust `&str` into a `CString`, and the C++ code then immediately turns it back into a ptr/len string before actually hashing it. So we can just call the ptr/len version directly instead.
---
This PR also fixes a bug in the C++ declaration of `LLVMRustCoverageHashByteArray`. It should be `size_t`, since that's what is declared and passed on the Rust side, and it's what `StrRef`'s constructor expects to receive on the callee side.
Coverage has two FFI functions for computing the hash of a byte string. One
takes a ptr/len pair, and the other takes a NUL-terminated C string.
But on closer inspection, the C string version is unnecessary. The calling-side
code converts a Rust `&str` into a C string, and the C++ code then immediately
turns it back into a ptr/len string before actually hashing it.
The function body immediately treats it as a slice anyway, so this just makes
it possible to call the hash function with arbitrary read-only byte slices.
Move `TyCtxt::mk_x` to `Ty::new_x` where applicable
Part of rust-lang/compiler-team#616
turns out there's a lot of places we construct `Ty` this is a ridiculously huge PR :S
r? `@oli-obk`
Revert the lexing of `c"…"` string literals
Fixes \[after beta-backport\] #113235.
Further progress is tracked in #113333.
This PR *manually* reverts parts of #108801 (since a git-revert would've been too coarse-grained & messy)
and git-reverts #111647.
CC `@fee1-dead` (#108801) `@klensy` (#111647)
r? `@compiler-errors`
`@rustbot` label F-c_str_literals beta-nominated
Add `-Zremark-dir` unstable flag to write LLVM optimization remarks to YAML
This PR adds an option for `rustc` to emit LLVM optimization remarks to a set of YAML files, which can then be digested by existing tools, like https://github.com/OfekShilon/optview2. When `-Cremark-dir` is passed, and remarks are enabled (`-Cremark=all`), the remarks will be now written to the specified directory, **instead** of being printed to standard error output. The files are named based on the CGU from which they are being generated.
Currently, the remarks are written using the LLVM streaming machinery, directly in the diagnostics handler. It seemed easier than going back to Rust and then form there back to C++ to use the streamer from the diagnostics handler. But there are many ways to implement this, of course, so I'm open to suggestions :)
I included some comments with questions into the code. Also, I'm not sure how to test this.
r? `@tmiasko`
Support for native WASM exceptions
### Motivation
Currently, rustc does not support native WASM exceptions. It does support JavaScript based exceptions for the wasm32-emscripten-target, but this requires back&forth with javascript for many calls, which is very slow.
Native wasm support for exceptions is quite common: Clang+LLVM implemented them years ago, and all major browsers support them by now. They enable zero-cost exceptions, at least with regard to runtime-performance-cost. They may increase startup-time and code size, though.
### Important: This PR does not change default behaviour
Exceptions usually add a lot of code in form of unwinding blocks, increasing the binary size. Most users probably do not want that, especially which regard to web development.
Therefore, wasm exceptions play a similar role as WASM-threads: rustc should support them, like clang does, but users who want to use it have to use some command-line magic like rustflags to opt in.
### What does this PR do?
As stated above, the default behaviour is not changed. It is already possible to opt-in into wasm exceptions using the command line. Unfortunately, the LLVM IR is invalid and the LLVM backend crashes.
```
rustc <sourcefile>
--target wasm32-unknown-unknown
-C panic=unwind
-C llvm-args=-wasm-enable-eh
-C target-feature=+exception-handling
```
As it turns out, LLVM is quite picky when it comes to IR for exception handling. If the IR does not look exactly like it should, some LLVM-assertions fail and the code generation crashes.
This PR adds the necessary modifications to the code generator to make it work. It also adds `exception-handling` as a wasm target feature.
### What this PR does not / what is missing
This PR is not a full fledges solution. It is the first step. A few parts are still missing; however, it is already useable (see next section).
Currently missing:
* The std library has to be adapted. Currently, only [no_std] crates work
* Usually, nested exceptions abort the program (i.e. a panic during the cleanup of another panic). This is currently not done yet.
- Currently, code inside cleanup handlers does not unwind
- To fix this requires a little more work: The code generator currently maintains a single terminate block per function for this. Unfortunately, WASM requires funclet based exception handling. Therefore, we need to create a terminate block per funclet. This is probably not a big problem, but I want to keep this PR simple.
### How to use the compiler given this PR?
This PR does not add any command line flags or features. It uses those which are already there. To compile with exceptions enabled, you need
* to set the panic strategy to unwind, i.e. `-C panic=unwind`
* to enable the exception-handling target feature, i.e. `-C target-feature=+exception-handling`
* to tell LLVM about the exception handling, i.e. `-C llvm-args=-wasm-enable-eh`
Since the standard library has not been adapted, you can only use it in [no_std] crates as of now. The intrinsic `core::intrinsics::r#try` works. To throw exceptions, you need the ```@llvm.wasm.throw``` intrinsic.
I created a sample application which works for me: https://github.com/mirkootter/rust-wasm-demos
This example can be run at https://webassembly.sh
After the last commit, they contain `Option<&OperandBundleDef<'a>>` but
the values are always `Some(_)`. This commit removes the needless
`Option` wrapper. This also simplifies the type signatures of
`LLVMRustBuild{Invoke,Call}`, which were relying on the fact that the
represention of `Option<&T>` is the same as `&T` for non-`None` values.
They never have a length of more than two. So this commit changes them
to `SmallVec<[_; 2]>`.
Also, we possibly push `None` values and then filter those `None` values
out again with `retain`. So this commit removes the `retain` and instead
only pushes the values if they are `Some(_)`.
I don't know why `SmallStr` was used here; some ad hoc profiling showed
this code is not that hot, the string is usually empty, and when it's
not empty it's usually very short. However, the use of a
`SmallStr<1024>` does result in 1024 byte `memcpy` call on each
execution, which shows up when I do `memcpy` profiling. So using a
normal string makes the code both simpler and very slightly faster.
`lookup_debug_loc` finds a file, line, and column, which requires two
binary searches. But this call site only needs the file.
This commit replaces the call with `lookup_source_file`, which does a
single binary search.
`lookup_debug_loc` calls `SourceMap::lookup_line`, which does a binary
search over the files, and then a binary search over the lines within
the found file. It then calls `SourceFile::line_begin_pos`, which redoes
the binary search over the lines within the found file.
This commit removes the second binary search over the lines, instead
getting the line starting pos directly using the result of the first
binary search over the lines.
(And likewise for `get_span_loc`, in the cranelift backend.)
Because tiny CGUs make compilation less efficient *and* result in worse
generated code.
We don't do this when the number of CGUs is explicitly given, because
there are times when the requested number is very important, as
described in some comments within the commit. So the commit also
introduces a `CodegenUnits` type that distinguishes between default
values and user-specified values.
This change has a roughly neutral effect on walltimes across the
rustc-perf benchmarks; there are some speedups and some slowdowns. But
it has significant wins for most other metrics on numerous benchmarks,
including instruction counts, cycles, binary size, and max-rss. It also
reduces parallelism, which is good for reducing jobserver competition
when multiple rustc processes are running at the same time. It's smaller
benchmarks that benefit the most; larger benchmarks already have CGUs
that are all larger than the minimum size.
Here are some example before/after CGU sizes for opt builds.
- html5ever
- CGUs: 16, mean size: 1196.1, sizes: [3908, 2992, 1706, 1652, 1572,
1136, 1045, 948, 946, 938, 579, 471, 443, 327, 286, 189]
- CGUs: 4, mean size: 4396.0, sizes: [6706, 3908, 3490, 3480]
- libc
- CGUs: 12, mean size: 35.3, sizes: [163, 93, 58, 53, 37, 8, 2 (x6)]
- CGUs: 1, mean size: 424.0, sizes: [424]
- tt-muncher
- CGUs: 5, mean size: 1819.4, sizes: [8508, 350, 198, 34, 7]
- CGUs: 1, mean size: 9075.0, sizes: [9075]
Note that CGUs of size 100,000+ aren't unusual in larger programs.
Removed use of iteration through a HashMap/HashSet in rustc_incremental and replaced with IndexMap/IndexSet
This allows for the `#[allow(rustc::potential_query_instability)]` in rustc_incremental to be removed, moving towards fixing #84447 (although a LOT more modules have to be changed to fully resolve it). Only HashMaps/HashSets that are being iterated through have been modified (although many structs and traits outside of rustc_incremental had to be modified as well, as they had fields/methods that involved a HashMap/HashSet that would be iterated through)
I'm making a PR for just 1 module changed to test for performance regressions and such, for future changes I'll either edit this PR to reflect additional modules being converted, or batch multiple modules of changes together and make a PR for each group of modules.
use c literals in compiler and library
Use c literals #108801 in compiler and library
currently blocked on:
* <strike>rustfmt: don't know how to format c literals</strike> nope, nightly one works.
* <strike>bootstrap</strike>
r? `@ghost`
`@rustbot` blocked
These tend to have special handling in a bunch of places anyway, so the variant helps remember that. And I think it's easier to grok than non-Scalar Aggregates sometimes being `Immediates` (like I got wrong and caused 109992). As a minor bonus, it means we don't need to generate poison LLVM values for them to pass around in `OperandValue::Immediate`s.
Optimize scalar and scalar pair representations loaded from ByRef in llvm
in https://github.com/rust-lang/rust/pull/105653 I noticed that we were generating suboptimal LLVM IR if we had a `ConstValue::ByRef` that could be represented by a `ScalarPair`. Before https://github.com/rust-lang/rust/pull/105653 this is probably rare, but after it, every slice will go down this suboptimal code path that requires LLVM to untangle a bunch of indirections and translate static allocations that are only used once to read a scalar pair from.
Adds support for LLVM [SafeStack] which provides backward edge control
flow protection by separating the stack into two parts: data which is
only accessed in provable safe ways is allocated on the normal stack
(the "safe stack") and all other data is placed in a separate allocation
(the "unsafe stack").
SafeStack support is enabled by passing `-Zsanitizer=safestack`.
[SafeStack]: https://clang.llvm.org/docs/SafeStack.html
Support #[global_allocator] without the allocator shim
This makes it possible to use liballoc/libstd in combination with `--emit obj` if you use `#[global_allocator]`. This is what rust-for-linux uses right now and systemd may use in the future. Currently they have to depend on the exact implementation of the allocator shim to create one themself as `--emit obj` doesn't create an allocator shim.
Note that currently the allocator shim also defines the oom error handler, which is normally required too. Once `#![feature(default_alloc_error_handler)]` becomes the only option, this can be avoided. In addition when using only fallible allocator methods and either `--cfg no_global_oom_handling` for liballoc (like rust-for-linux) or `--gc-sections` no references to the oom error handler will exist.
To avoid this feature being insta-stable, you will have to define `__rust_no_alloc_shim_is_unstable` to avoid linker errors.
(Labeling this with both T-compiler and T-lang as it originally involved both an implementation detail and had an insta-stable user facing change. As noted above, the `__rust_no_alloc_shim_is_unstable` symbol requirement should prevent unintended dependence on this unstable feature.)
Rather than returning an array of features from to_llvm_features, return a structure that contains
the dependencies. This also contains metadata on how the features depend on each other to allow for
the correct enabling and disabling.
Some features that are tied together only make sense to be folded
together when enabling the feature. For example on AArch64 sve and
neon are tied together, however it doesn't make sense to disable neon
when disabling sve.
In #91608 the fp-armv8 feature was removed as it's tied to the neon
feature. However disabling neon didn't actually disable the use of
floating point registers and instructions, for this `-fp-armv8` is
required.
Fix dependency tracking for debugger visualizers
This PR fixes dependency tracking for debugger visualizer files by changing the `debugger_visualizers` query to an `eval_always` query that scans the AST while it is still available. This way the set of visualizer files is already available when dep-info is emitted. Since the query is turned into an `eval_always` query, dependency tracking will now reliably detect changes to the visualizer script files themselves.
TODO:
- [x] perf.rlo
- [x] Needs a bit more documentation in some places
- [x] Needs regression test for the incr. comp. case
Fixes https://github.com/rust-lang/rust/issues/111226
Fixes https://github.com/rust-lang/rust/issues/111227
Fixes https://github.com/rust-lang/rust/issues/111295
r? `@wesleywiser`
cc `@gibbyfree`
Only depend on CFG_VERSION in rustc_interface
This avoids having to rebuild the whole compiler on each commit when `omit-git-hash = false`.
cc https://github.com/rust-lang/rust/issues/76720 - this won't fix it, and I'm not suggesting we turn this on by default, but it will make it less painful for people who do have `omit-git-hash` on as a workaround.
Remove the ThinLTO CU hack
This reverts #46722, commit e0ab5d5feb.
Since #111167, commit 10b69dde3f, we are
generating DWARF subprograms in a way that is meant to be more compatible
with LLVM's expectations, so hopefully we don't need this workaround
rewriting CUs anymore.
Remove misleading target feature aliases
Fixes#100752. This is a follow up to #103750. These aliases could not be completely removed until rust-lang/stdarch#1355 landed.
cc `@Amanieu`
CFI: Fix SIGILL reached via trait objects
Fix#106547 by transforming the concrete self into a reference to a trait object before emitting type metadata identifiers for trait methods.
You will need to add the following as replacement for the old __rust_*
definitions when not using the alloc shim.
#[no_mangle]
static __rust_no_alloc_shim_is_unstable: u8 = 0;
This makes it possible to use liballoc/libstd in combination with
`--emit obj` if you use `#[global_allocator]`. Making it work for the
default libstd allocator would require weak functions, which are not
well supported on all systems.
This reverts #46722, commit e0ab5d5feb.
Since #111167, commit 10b69dde3f, we are
generating DWARF subprograms in a way that is meant to be more compatible
with LLVM's expectations, so hopefully we don't need this workaround
rewriting CUs anymore.
Mark s390x condition code register as clobbered in inline assembly
Various s390x instructions (arithmetic operations, logical operations, comparisons, etc. see also "Condition Codes" section in [z/Architecture Reference Summary](https://www.ibm.com/support/pages/zarchitecture-reference-summary)) modify condition code register `cc`, but AFAIK there is currently no way to mark it as clobbered in `asm!`.
`cc` register definition in LLVM:
https://github.com/llvm/llvm-project/blob/main/llvm/lib/Target/SystemZ/SystemZRegisterInfo.td#L320
This PR also updates asm_experimental_arch docs in the unstable-book to mention s390x registers.
cc `@uweigand`
r? `@Amanieu`
Output LLVM optimization remark kind in `-Cremark` output
Since https://github.com/rust-lang/rust/pull/90833, the optimization remark kind has not been printed. Therefore it wasn't possible to easily determine from the log (in a programmatic way) which remark kind was produced. I think that the most interesting remarks are the missed ones, which can lead users to some code optimization.
Maybe we could also change the format closer to the "old" one:
```
note: optimization remark for tailcallelim at /checkout/src/libcore/num/mod.rs:1:0: marked this call a tail call candidate
```
I wanted to programatically parse the remarks so that they could work e.g. with https://github.com/OfekShilon/optview2. However, now that I think about it, probably the proper solution is to tell rustc to output them to YAML and then use the YAML as input for the opt remark visualization tools. The flag for enabling this does not seem to work though (https://github.com/rust-lang/rust/issues/96705#issuecomment-1117632322).
Still I think that it's good to output the remark kind anyway, it's an important piece of information.
r? ```@tmiasko```
debuginfo: split method declaration and definition
When we're adding a method to a type DIE, we only want a DW_AT_declaration
there, because LLVM LTO can't unify type definitions when a child DIE is a
full subprogram definition. Now the subprogram definition gets added at the
CU level with a specification link back to the abstract declaration.
Both GCC and Clang write debuginfo this way for C++ class methods.
Fixes#109730.
Fixes#109934.
Stabilize raw-dylib, link_ordinal, import_name_type and -Cdlltool
This stabilizes the `raw-dylib` feature (#58713) for all architectures (i.e., `x86` as it is already stable for all other architectures).
Changes:
* Permit the use of the `raw-dylib` link kind for x86, the `link_ordinal` attribute and the `import_name_type` key for the `link` attribute.
* Mark the `raw_dylib` feature as stable.
* Stabilized the `-Zdlltool` argument as `-Cdlltool`.
* Note the path to `dlltool` if invoking it failed (we don't need to do this if `dlltool` returns an error since it prints its path in the error message).
* Adds tests for `-Cdlltool`.
* Adds tests for being unable to find the dlltool executable, and dlltool failing.
* Fixes a bug where we were checking the exit code of dlltool to see if it failed, but dlltool always returns 0 (indicating success), so instead we need to check if anything was written to `stderr`.
NOTE: As previously noted (https://github.com/rust-lang/rust/pull/104218#issuecomment-1315895618) using dlltool within rustc is temporary, but this is not the first time that Rust has added a temporary tool use and argument: https://github.com/rust-lang/rust/pull/104218#issuecomment-1318720482
Big thanks to ``````@tbu-`````` for the first version of this PR (#104218)
Add cross-language LLVM CFI support to the Rust compiler
This PR adds cross-language LLVM Control Flow Integrity (CFI) support to the Rust compiler by adding the `-Zsanitizer-cfi-normalize-integers` option to be used with Clang `-fsanitize-cfi-icall-normalize-integers` for normalizing integer types (see https://reviews.llvm.org/D139395).
It provides forward-edge control flow protection for C or C++ and Rust -compiled code "mixed binaries" (i.e., for when C or C++ and Rust -compiled code share the same virtual address space). For more information about LLVM CFI and cross-language LLVM CFI support for the Rust compiler, see design document in the tracking issue #89653.
Cross-language LLVM CFI can be enabled with -Zsanitizer=cfi and -Zsanitizer-cfi-normalize-integers, and requires proper (i.e., non-rustc) LTO (i.e., -Clinker-plugin-lto).
Thank you again, ``@bjorn3,`` ``@nikic,`` ``@samitolvanen,`` and the Rust community for all the help!
When we're adding a method to a type DIE, we only want a DW_AT_declaration
there, because LLVM LTO can't unify type definitions when a child DIE is a
full subprogram definition. Now the subprogram definition gets added at the
CU level with a specification link back to the abstract declaration.
This commit adds cross-language LLVM Control Flow Integrity (CFI)
support to the Rust compiler by adding the
`-Zsanitizer-cfi-normalize-integers` option to be used with Clang
`-fsanitize-cfi-icall-normalize-integers` for normalizing integer types
(see https://reviews.llvm.org/D139395).
It provides forward-edge control flow protection for C or C++ and Rust
-compiled code "mixed binaries" (i.e., for when C or C++ and Rust
-compiled code share the same virtual address space). For more
information about LLVM CFI and cross-language LLVM CFI support for the
Rust compiler, see design document in the tracking issue #89653.
Cross-language LLVM CFI can be enabled with -Zsanitizer=cfi and
-Zsanitizer-cfi-normalize-integers, and requires proper (i.e.,
non-rustc) LTO (i.e., -Clinker-plugin-lto).
Report allocation errors as panics
OOM is now reported as a panic but with a custom payload type (`AllocErrorPanicPayload`) which holds the layout that was passed to `handle_alloc_error`.
This should be review one commit at a time:
- The first commit adds `AllocErrorPanicPayload` and changes allocation errors to always be reported as panics.
- The second commit removes `#[alloc_error_handler]` and the `alloc_error_hook` API.
ACP: https://github.com/rust-lang/libs-team/issues/192Closes#51540Closes#51245
Support AIX-style archive type
Reading facility of AIX big archive has been supported by `object` since 0.30.0.
Writing facility of AIX big archive has already been supported by `ar_archive_writer`, but we need to bump the version to support the new archive type enum.
Add `rustc_fluent_macro` to decouple fluent from `rustc_macros`
Fluent, with all the icu4x it brings in, takes quite some time to compile. `fluent_messages!` is only needed in further downstream rustc crates, but is blocking more upstream crates like `rustc_index`. By splitting it out, we allow `rustc_macros` to be compiled earlier, which speeds up `x check compiler` by about 5 seconds (and even more after the needless dependency on `serde_json` is removed from `rustc_data_structures`).
Encode hashes as bytes, not varint
In a few places, we store hashes as `u64` or `u128` and then apply `derive(Decodable, Encodable)` to the enclosing struct/enum. It is more efficient to encode hashes directly than try to apply some varint encoding. This PR adds two new types `Hash64` and `Hash128` which are produced by `StableHasher` and replace every use of storing a `u64` or `u128` that represents a hash.
Distribution of the byte lengths of leb128 encodings, from `x build --stage 2` with `incremental = true`
Before:
```
( 1) 373418203 (53.7%, 53.7%): 1
( 2) 196240113 (28.2%, 81.9%): 3
( 3) 108157958 (15.6%, 97.5%): 2
( 4) 17213120 ( 2.5%, 99.9%): 4
( 5) 223614 ( 0.0%,100.0%): 9
( 6) 216262 ( 0.0%,100.0%): 10
( 7) 15447 ( 0.0%,100.0%): 5
( 8) 3633 ( 0.0%,100.0%): 19
( 9) 3030 ( 0.0%,100.0%): 8
( 10) 1167 ( 0.0%,100.0%): 18
( 11) 1032 ( 0.0%,100.0%): 7
( 12) 1003 ( 0.0%,100.0%): 6
( 13) 10 ( 0.0%,100.0%): 16
( 14) 10 ( 0.0%,100.0%): 17
( 15) 5 ( 0.0%,100.0%): 12
( 16) 4 ( 0.0%,100.0%): 14
```
After:
```
( 1) 372939136 (53.7%, 53.7%): 1
( 2) 196240140 (28.3%, 82.0%): 3
( 3) 108014969 (15.6%, 97.5%): 2
( 4) 17192375 ( 2.5%,100.0%): 4
( 5) 435 ( 0.0%,100.0%): 5
( 6) 83 ( 0.0%,100.0%): 18
( 7) 79 ( 0.0%,100.0%): 10
( 8) 50 ( 0.0%,100.0%): 9
( 9) 6 ( 0.0%,100.0%): 19
```
The remaining 9 or 10 and 18 or 19 are `u64` and `u128` respectively that have the high bits set. As far as I can tell these are coming primarily from `SwitchTargets`.
Fluent, with all the icu4x it brings in, takes quite some time to
compile. `fluent_messages!` is only needed in further downstream rustc
crates, but is blocking more upstream crates like `rustc_index`. By
splitting it out, we allow `rustc_macros` to be compiled earlier, which
speeds up `x check compiler` by about 5 seconds (and even more after the
needless dependency on `serde_json` is removed from
`rustc_data_structures`).
`-Cdebuginfo=1` was never line tables only and
can't be due to backwards compatibility issues.
This was clarified and an option for line tables only
was added. Additionally an option for line info
directives only was added, which is well needed for
some targets. The debug info options should now
behave the same as clang's debug info options.
Update `ty::VariantDef` to use `IndexVec<FieldIdx, FieldDef>`
And while doing the updates for that, also uses `FieldIdx` in `ProjectionKind::Field` and `TypeckResults::field_indices`.
There's more places that could use it (like `rustc_const_eval` and `LayoutS`), but I tried to keep this PR from exploding to *even more* places.
Part 2/? of https://github.com/rust-lang/compiler-team/issues/606
And while doing the updates for that, also uses `FieldIdx` in `ProjectionKind::Field` and `TypeckResults::field_indices`.
There's more places that could use it (like `rustc_const_eval` and `LayoutS`), but I tried to keep this PR from exploding to *even more* places.
Part 2/? of https://github.com/rust-lang/compiler-team/issues/606
Partial stabilization of `once_cell`
This PR aims to stabilize a portion of the `once_cell` feature:
- `core::cell::OnceCell`
- `std::cell::OnceCell` (re-export of the above)
- `std::sync::OnceLock`
This will leave `LazyCell` and `LazyLock` unstabilized, which have been moved to the `lazy_cell` feature flag.
Tracking issue: https://github.com/rust-lang/rust/issues/74465 (does not fully close, but it may make sense to move to a new issue)
Future steps for separate PRs:
- ~~Add `#[inline]` to many methods~~ #105651
- Update cranelift usage of the `once_cell` crate
- Update rust-analyzer usage of the `once_cell` crate
- Update error messages discussing once_cell
## To be stabilized API summary
```rust
// core::cell (in core/cell/once.rs)
pub struct OnceCell<T> { .. }
impl<T> OnceCell<T> {
pub const fn new() -> OnceCell<T>;
pub fn get(&self) -> Option<&T>;
pub fn get_mut(&mut self) -> Option<&mut T>;
pub fn set(&self, value: T) -> Result<(), T>;
pub fn get_or_init<F>(&self, f: F) -> &T where F: FnOnce() -> T;
pub fn into_inner(self) -> Option<T>;
pub fn take(&mut self) -> Option<T>;
}
impl<T: Clone> Clone for OnceCell<T>;
impl<T: Debug> Debug for OnceCell<T>
impl<T> Default for OnceCell<T>;
impl<T> From<T> for OnceCell<T>;
impl<T: PartialEq> PartialEq for OnceCell<T>;
impl<T: Eq> Eq for OnceCell<T>;
```
```rust
// std::sync (in std/sync/once_lock.rs)
impl<T> OnceLock<T> {
pub const fn new() -> OnceLock<T>;
pub fn get(&self) -> Option<&T>;
pub fn get_mut(&mut self) -> Option<&mut T>;
pub fn set(&self, value: T) -> Result<(), T>;
pub fn get_or_init<F>(&self, f: F) -> &T where F: FnOnce() -> T;
pub fn into_inner(self) -> Option<T>;
pub fn take(&mut self) -> Option<T>;
}
impl<T: Clone> Clone for OnceLock<T>;
impl<T: Debug> Debug for OnceLock<T>;
impl<T> Default for OnceLock<T>;
impl<#[may_dangle] T> Drop for OnceLock<T>;
impl<T> From<T> for OnceLock<T>;
impl<T: PartialEq> PartialEq for OnceLock<T>
impl<T: Eq> Eq for OnceLock<T>;
impl<T: RefUnwindSafe + UnwindSafe> RefUnwindSafe for OnceLock<T>;
unsafe impl<T: Send> Send for OnceLock<T>;
unsafe impl<T: Sync + Send> Sync for OnceLock<T>;
impl<T: UnwindSafe> UnwindSafe for OnceLock<T>;
```
No longer planned as part of this PR, and moved to the `rust_cell_try` feature gate:
```rust
impl<T> OnceCell<T> {
pub fn get_or_try_init<F, E>(&self, f: F) -> Result<&T, E> where F: FnOnce() -> Result<T, E>;
}
impl<T> OnceLock<T> {
pub fn get_or_try_init<F, E>(&self, f: F) -> Result<&T, E> where F: FnOnce() -> Result<T, E>;
}
```
I am new to this process so would appreciate mentorship wherever needed.
Move `mir::Field` → `abi::FieldIdx`
The first PR for https://github.com/rust-lang/compiler-team/issues/606
This is just the move-and-rename, because it's plenty big already. Future PRs will start using `FieldIdx` more broadly, and concomitantly removing `FieldIdx::new`s.
The first PR for https://github.com/rust-lang/compiler-team/issues/606
This is just the move-and-rename, because it's plenty big-and-bitrotty already. Future PRs will start using `FieldIdx` more broadly, and concomitantly removing `FieldIdx::new`s.
Update ar_archive_writer to 0.1.3
This updates object to 0.30 and fixes a bug where the symbol table would be omitted for archives where there are object files yet none that export any symbol. This bug could lead to linker errors for crates like rustc_std_workspace_core which don't contain any code of their own but exist solely for their dependencies. This is likely the cause of the linker issues I was experiencing on Webassembly. It has been shown to cause issues on other platforms too.
cc rust-lang/ar_archive_writer#5
debuginfo: Get pointer size/align from tcx.data_layout instead of layout_of
This avoids some type interning and a query execution. It also just makes the code simpler.
Use poison instead of undef
In cases where it is legal, we should prefer poison values over undef values.
This replaces undef with poison for aggregate construction and for uninhabited types. There are more places where we can likely use poison, but I wanted to stay conservative to start with.
In particular the aggregate case is important for newer LLVM versions, which are not able to handle an undef base value during early optimization due to poison-propagation concerns.
r? `@cuviper`
This updates object to 0.30 and fixes a bug where the symbol table
would be omitted for archives where there are object files yet none
that export any symbol. This bug could lead to linker errors for crates
like rustc_std_workspace_core which don't contain any code of their own
but exist solely for their dependencies. This is likely the cause of
the linker issues I was experiencing on Webassembly. It has been shown
to cause issues on other platforms too.
cc rust-lang/ar_archive_writer#5
Add `-Z time-passes-format` to allow specifying a JSON output for `-Z time-passes`
This adds back the `-Z time` option as that is useful for [my rustc benchmark tool](https://github.com/Zoxc/rcb), reverting https://github.com/rust-lang/rust/pull/102725. It now uses nanoseconds and bytes as the units so it is renamed to `time-precise`.
Fix cross-compiling with dlltool for raw-dylib
Fix for #103939
Issue Details:
When attempting to cross-compile using the `raw-dylib` feature and the GNU toolchain, rustc would attempt to find a cross-compiling version of dlltool (e.g., `i686-w64-mingw32-dlltool`). The has two issues 1) on Windows dlltool is always `dlltool` (no cross-compiling named versions exist) and 2) it only supported compiling to i686 and x86_64 resulting in ARM 32 and 64 compiling as x86_64.
Fix Details:
* On Windows always use the normal `dlltool` binary.
* Add the ARM64 cross-compiling dlltool name (support for this is coming: https://sourceware.org/bugzilla/show_bug.cgi?id=29964)
* Provide the `-m` argument to dlltool to indicate the target machine type.
(This is the first of two PRs to fix the remaining issues for the `raw-dylib` feature (#58713) that is blocking stabilization (#104218))
Updates `interpret`, `codegen_ssa`, and `codegen_cranelift` to consume the new cast instead of the intrinsic.
Includes `CastTransmute` for custom MIR building, to be able to test the extra UB.
In cases where it is legal, we should prefer poison values over
undef values.
This replaces undef with poison for aggregate construction and
for uninhabited types. There are more places where we can likely
use poison, but I wanted to stay conservative to start with.
In particular the aggregate case is important for newer LLVM
versions, which are not able to handle an undef base value during
early optimization due to poison-propagation concerns.
tidy: enforce comment blocks to have an even number of backticks
After PR #108694, most unmatched backticks in `compiler/` comments have been eliminated. This PR adds a tidy lint to ensure no new unmatched backticks are added, and either addresses the lint in the remaining instances it found, or allows it.
Very often, backtick containing sections wrap around lines, for example:
```Rust
// This function takes a tuple `(Vec<String>,
// Box<[u8]>)` and transforms it into `Vec<u8>`.
```
The lint is implemented to work on top of blocks, counting each line with a `//` into a block, and counting if there are an odd or even number of backticks in the entire block, instead of looking at just a single line.
This makes it easier to open the messages file while developing on features.
The commit was the result of automatted changes:
for p in compiler/rustc_*; do mv $p/locales/en-US.ftl $p/messages.ftl; rmdir $p/locales; done
for p in compiler/rustc_*; do sed -i "s#\.\./locales/en-US.ftl#../messages.ftl#" $p/src/lib.rs; done
Add `round_ties_even` to `f32` and `f64`
Tracking issue: #96710
Redux of #82273. See also #55107
Adds a new method, `round_ties_even`, to `f32` and `f64`, that rounds the float to the nearest integer , rounding halfway cases to the number with an even least significant bit. Uses the `roundeven` LLVM intrinsic to do this.
Of the five IEEE 754 rounding modes, this is the only one that doesn't already have a round-to-integer function exposed by Rust (others are `round`, `floor`, `ceil`, and `trunc`). Ties-to-even is also the rounding mode used for int-to-float and float-to-float `as` casts, as well as float arithmentic operations. So not having an explicit rounding method for it seems like an oversight.
Bikeshed: this PR currently uses `round_ties_even` for the name of the method. But maybe `round_ties_to_even` is better, or `round_even`, or `round_to_even`?
rustc_middle: Remove trait `DefIdTree`
This trait was a way to generalize over both `TyCtxt` and `Resolver`, but now `Resolver` has access to `TyCtxt`, so this trait is no longer necessary.
Remove legacy PM leftovers
This drops two leftovers of legacy PM usage:
* We don't need to initialize passes anymore.
* The pass listing was still using legacy PM passes. Replace it with the corresponding new PM listing.
Name LLVM anonymous constants by a hash of their contents
This makes the names stable between different versions of a crate unlike the `AllocId` naming, making LLVM IR comparisons with `llvm-diff` more practical.
(This is a large commit. The changes to
`compiler/rustc_middle/src/ty/context.rs` are the most important ones.)
The current naming scheme is a mess, with a mix of `_intern_`, `intern_`
and `mk_` prefixes, with little consistency. In particular, in many
cases it's easy to use an iterator interner when a (preferable) slice
interner is available.
The guiding principles of the new naming system:
- No `_intern_` prefixes.
- The `intern_` prefix is for internal operations.
- The `mk_` prefix is for external operations.
- For cases where there is a slice interner and an iterator interner,
the former is `mk_foo` and the latter is `mk_foo_from_iter`.
Also, `slice_interners!` and `direct_interners!` can now be `pub` or
non-`pub`, which helps enforce the internal/external operations
division.
It's not perfect, but I think it's a clear improvement.
The following lists show everything that was renamed.
slice_interners
- const_list
- mk_const_list -> mk_const_list_from_iter
- intern_const_list -> mk_const_list
- substs
- mk_substs -> mk_substs_from_iter
- intern_substs -> mk_substs
- check_substs -> check_and_mk_substs (this is a weird one)
- canonical_var_infos
- intern_canonical_var_infos -> mk_canonical_var_infos
- poly_existential_predicates
- mk_poly_existential_predicates -> mk_poly_existential_predicates_from_iter
- intern_poly_existential_predicates -> mk_poly_existential_predicates
- _intern_poly_existential_predicates -> intern_poly_existential_predicates
- predicates
- mk_predicates -> mk_predicates_from_iter
- intern_predicates -> mk_predicates
- _intern_predicates -> intern_predicates
- projs
- intern_projs -> mk_projs
- place_elems
- mk_place_elems -> mk_place_elems_from_iter
- intern_place_elems -> mk_place_elems
- bound_variable_kinds
- mk_bound_variable_kinds -> mk_bound_variable_kinds_from_iter
- intern_bound_variable_kinds -> mk_bound_variable_kinds
direct_interners
- region
- intern_region (unchanged)
- const
- mk_const_internal -> intern_const
- const_allocation
- intern_const_alloc -> mk_const_alloc
- layout
- intern_layout -> mk_layout
- adt_def
- intern_adt_def -> mk_adt_def_from_data (unusual case, hard to avoid)
- alloc_adt_def(!) -> mk_adt_def
- external_constraints
- intern_external_constraints -> mk_external_constraints
Other
- type_list
- mk_type_list -> mk_type_list_from_iter
- intern_type_list -> mk_type_list
- tup
- mk_tup -> mk_tup_from_iter
- intern_tup -> mk_tup
Remove type-traversal trait aliases
#107924 moved the type traversal (folding and visiting) traits into the type library, but created trait aliases in `rustc_middle` to minimise both the API churn for trait consumers and the arising boilerplate. As mentioned in that PR, an alternative approach of defining subtraits with blanket implementations of the respective supertraits was also considered at that time but was ruled out as not adding much value.
Unfortunately, it has since emerged that rust-analyzer has difficulty with these trait aliases at present, resulting in a degraded contributor experience (see the recent [r-a has become useless](https://rust-lang.zulipchat.com/#narrow/stream/182449-t-compiler.2Fhelp/topic/r-a.20has.20become.20useless) topic on the #t-compiler/help Zulip stream).
This PR removes the trait aliases, and accordingly the underlying type library traits are now used directly; they are parameterised by `TyCtxt<'tcx>` rather than just the `'tcx` lifetime, and imports have been updated to reflect the fact that the trait aliases' explicitly named traits are no longer automatically brought into scope. These changes also roll-back the (no-longer required) workarounds to #107747 that were made in b409329c62.
Since this PR is just a find+replace together with the changes necessary for compilation & tidy to pass, it's currently just one mega-commit. Let me know if you'd like it broken up.
r? `@oli-obk`
Extend `CodegenBackend` trait with a function returning the translation
resources from the codegen backend, which can be added to the complete
list of resources provided to the emitter.
Signed-off-by: David Wood <david.wood@huawei.com>
Instead of loading the Fluent resources for every crate in
`rustc_error_messages`, each crate generates typed identifiers for its
own diagnostics and creates a static which are pulled together in the
`rustc_driver` crate and provided to the diagnostic emitter.
Signed-off-by: David Wood <david.wood@huawei.com>
Add `kernel-address` sanitizer support for freestanding targets
This PR adds support for KASan (kernel address sanitizer) instrumentation in freestanding targets. I included the minimal set of `x86_64-unknown-none`, `riscv64{imac, gc}-unknown-none-elf`, and `aarch64-unknown-none` but there's likely other targets it can be added to. (`linux_kernel_base.rs`?) KASan uses the address sanitizer attributes but has the `CompileKernel` parameter set to `true` in the pass creation.
Add the attributes to functions according to the settings.
"xray-always" overrides "xray-never", and they both override
"xray-ignore-loops" and "xray-instruction-threshold", but we'll
let lints deal with warnings about silly attribute combinations.
Rollup of 8 pull requests
Successful merges:
- #107022 (Implement `SpecOptionPartialEq` for `cmp::Ordering`)
- #107100 (Use proper `InferCtxt` when probing for associated types in astconv)
- #107103 (Use new solver in `evaluate_obligation` query (when new solver is enabled))
- #107190 (Recover from more const arguments that are not wrapped in curly braces)
- #107306 (Correct suggestions for closure arguments that need a borrow)
- #107339 (internally change regions to be covariant)
- #107344 (Minor tweaks in the new solver)
- #107373 (Don't merge vtables when full debuginfo is enabled.)
Failed merges:
r? `@ghost`
`@rustbot` modify labels: rollup
Don't merge vtables when full debuginfo is enabled.
This PR makes the compiler not emit the `unnamed_addr` attribute for vtables when full debuginfo is enabled, so that they don't get merged even if they have the same contents. This allows debuggers to more reliably map from a dyn pointer to the self-type of a trait object by looking at the vtable's debuginfo.
The PR only changes the behavior of the LLVM backend as other backends don't emit vtable debuginfo (as far as I can tell).
The performance impact of this change should be small as [measured](https://github.com/rust-lang/rust/pull/103514#issuecomment-1290833854) in a previous PR.
...and remove it from `PointeeInfo`, which isn't meant for this.
There are still various places (marked with FIXMEs) that assume all pointers
have the same size and alignment. Fixing this requires parsing non-default
address spaces in the data layout string, which will be done in a followup.
Various cleanups around pre-TyCtxt queries and functions
part of #105462
based on https://github.com/rust-lang/rust/pull/106776 (everything starting at [0e2b39f](0e2b39fd1f) is new in this PR)
r? `@petrochenkov`
I think this should be most of the uncontroversial part of #105462.
Rollup of 7 pull requests
Successful merges:
- #104505 (Remove double spaces after dots in comments)
- #106784 (prevent E0512 from emitting [type error] by checking the references_error)
- #106834 (new trait solver: only consider goal changed if response is not identity)
- #106889 (Mention the lack of `windows_mut` in `windows`)
- #106963 (Use `scope_expr_id` from `ProbeCtxt`)
- #106970 (Switch to `EarlyBinder` for `item_bounds` query)
- #106980 (Hide `_use_mk_alias_ty_instead` in `<AliasTy as Debug>::fmt`)
Failed merges:
r? `@ghost`
`@rustbot` modify labels: rollup
Previously, it was only put on scalars with range validity invariants
like bool, was uninit was obviously invalid for those.
Since then, we have normatively declared all uninit primitives to be
undefined behavior and can therefore put `noundef` on them.
The remaining concern was the `mem::uninitialized` function, which cause
quite a lot of UB in the older parts of the ecosystem. This function now
doesn't return uninit values anymore, making users of it safe from this
change.
The only real sources of UB where people could encounter uninit
primitives are `MaybeUninit::uninit().assume_init()`, which has always
be clear in the docs about being UB and from heap allocations (like
reading from the spare capacity of a vec. This is hopefully rare enough
to not break anything.
Migrate `codegen_ssa` to diagnostics structs - [Part 3]
Completes migrating `codegen_ssa` module except 2 outstanding errors that depend on other crates:
1. [`rustc_middle::mir::interpret::InterpError`](b6097f2e1b/compiler/rustc_middle/src/mir/interpret/error.rs (L475)): I saw `rustc_middle` is unassigned, I am open to take this work.
2. `codegen_llvm`'s use of `fn span_invalid_monomorphization_error`, which I started to replace in the [last commit](9a31b3cdda) of this PR, but would like to know the team's preference on how we should keep replacing the other macros:
2.1. Update macros to expect a `Diagnostic`
2.2. Remove macros and expand the code on each use.
See [some examples of the different options in this experimental commit](64aee83e80)
_Part 2 - https://github.com/rust-lang/rust/pull/103792_
r? ``@davidtwco``
Cc ``@compiler-errors``
Remove wrapper functions for some unstable options
They are trivial and just forward to the option. Like most other options, we can just access it directly.
Allow building std with cranelift
- Don't pass llvm-specific args when using cranelift
- Don't use `asm` in compiler_builtins when using cranelift
r? `@bjorn3` cc `@Mark-Simulacrum`
Correct branch-protection ModFlagBehavior for Aarch64 on LLVM-15
When building with Fat LTO and BTI enabled on aarch64, the BTI is set to `Module::Min` for alloc shim but is set to `Module::Error` for the crate. This was fine when we were using LLVM-14 but LLVM-15 changes it's behaviour to support for compiling with different `mbranch-protection` flags.
Refer:
b0343a38a5
fixes https://github.com/rust-lang/rust/issues/102162
When building with Fat LTO and BTI enabled on aarch64, the BTI is set to
`Module::Min` for alloc shim but is set to `Module::Error` for the
crate. This was fine when we were using LLVM-14 but LLVM-15 changes it's
behaviour to support for compiling with different `mbranch-protection`
flags.
Refer:
b0343a38a5
Following the s390x ELF ABI and based on the clang implementation,
provide appropriate definitions of va_list in library/core/src/ffi/mod.rs
and va_arg handling in compiler/rustc_codegen_llvm/src/va_arg.rs.
Fixes the following test cases on s390x:
src/test/run-make-fulldeps/c-link-to-rust-va-list-fn
src/test/ui/abi/variadic-ffi.rs
Fixes https://github.com/rust-lang/rust/issues/84628.
Use struct types during codegen in less places
This makes it easier to use cg_ssa from a backend like Cranelift that doesn't have any struct types at all. After this PR struct types are still used for function arguments and return values. Removing those usages is harder but should still be doable.
Don't internalize __llvm_profile_counter_bias
Currently, LLVM profiling runtime counter relocation cannot be used by rust during LTO because symbols are being internalized before all symbol information is known.
This mode makes LLVM emit a __llvm_profile_counter_bias symbol which is referenced by the profiling initialization, which itself is pulled in by the rust driver here [1].
It is enabled with -Cllvm-args=-runtime-counter-relocation for platforms which are opt-in to this mode like Linux. On these platforms there will be no link error, rather just surprising behavior for a user which request runtime counter relocation. The profiling runtime will not see that symbol go on as if it were never there. On Fuchsia, the profiling runtime must have this symbol which will cause a hard link error.
As an aside, I don't have enough context as to why rust's LTO model is how it is. AFAICT, the internalize pass is only safe to run at link time when all symbol information is actually known, this being an example as to why. I think special casing this symbol as a known one that LLVM can emit which should not have it's visbility de-escalated should be fine given how seldom this pattern of defining an undefined symbol to get initilization code pulled in is. From a quick grep, __llvm_profile_runtime is the only symbol that rustc does this for.
[1] 0265a3e93b/compiler/rustc_codegen_ssa/src/back/linker.rs (L598)
Add LLVM KCFI support to the Rust compiler
This PR adds LLVM Kernel Control Flow Integrity (KCFI) support to the Rust compiler. It initially provides forward-edge control flow protection for operating systems kernels for Rust-compiled code only by aggregating function pointers in groups identified by their return and parameter types. (See llvm/llvm-project@cff5bef.)
Forward-edge control flow protection for C or C++ and Rust -compiled code "mixed binaries" (i.e., for when C or C++ and Rust -compiled code share the same virtual address space) will be provided in later work as part of this project by identifying C char and integer type uses at the time types are encoded (see Type metadata in the design document in the tracking issue #89653).
LLVM KCFI can be enabled with -Zsanitizer=kcfi.
Thank you again, `@bjorn3,` `@eddyb,` `@nagisa,` and `@ojeda,` for all the help!
This commit adds LLVM Kernel Control Flow Integrity (KCFI) support to
the Rust compiler. It initially provides forward-edge control flow
protection for operating systems kernels for Rust-compiled code only by
aggregating function pointers in groups identified by their return and
parameter types. (See llvm/llvm-project@cff5bef.)
Forward-edge control flow protection for C or C++ and Rust -compiled
code "mixed binaries" (i.e., for when C or C++ and Rust -compiled code
share the same virtual address space) will be provided in later work as
part of this project by identifying C char and integer type uses at the
time types are encoded (see Type metadata in the design document in the
tracking issue #89653).
LLVM KCFI can be enabled with -Zsanitizer=kcfi.
Co-authored-by: bjorn3 <17426603+bjorn3@users.noreply.github.com>
Currently, LLVM profiling runtime counter relocation cannot be
used by rust during LTO because symbols are being internalized
before all symbol information is known.
This mode makes LLVM emit a __llvm_profile_counter_bias symbol
which is referenced by the profiling initialization, which itself
is pulled in by the rust driver here [1].
It is enabled with -Cllvm-args=-runtime-counter-relocation for
platforms which are opt-in to this mode like Linux. On these
platforms there will be no link error, rather just surprising
behavior for a user which request runtime counter relocation.
The profiling runtime will not see that symbol go on as if it
were never there. On Fuchsia, the profiling runtime must have
this symbol which will cause a hard link error.
As an aside, I don't have enough context as to why rust's LTO
model is how it is. AFAICT, the internalize pass is only safe
to run at link time when all symbol information is actually
known, this being an example as to why. I think special casing
this symbol as a known one that LLVM can emit which should not
have it's visbility de-escalated should be fine given how
seldom this pattern of defining an undefined symbol to get
initilization code pulled in is. From a quick grep,
__llvm_profile_runtime is the only symbol that rustc does this
for.
[1] 0265a3e93b/compiler/rustc_codegen_ssa/src/back/linker.rs (L598)
This ensures that the error is printed even for unused variables,
as well as unifying the handling between the LLVM and GCC backends.
This also fixes unusual behavior around exported Rust-defined variables
with linkage attributes. With the previous behavior, it appears to be
impossible to define such a variable such that it can actually be imported
and used by another crate. This is because on the importing side, the
variable is required to be a pointer, but on the exporting side, the
type checker rejects static variables of pointer type because they do
not implement `Sync`. Even if it were possible to import such a type, it
appears that code generation on the importing side would add an unexpected
additional level of pointer indirection, which would break type safety.
This highlighted that the semantics of linkage on Rust-defined variables
is different to linkage on foreign items. As such, we now model the
difference with two different codegen attributes: linkage for Rust-defined
variables, and import_linkage for foreign items.
This change gives semantics to the test
src/test/ui/linkage-attr/auxiliary/def_illtyped_external.rs which was
previously expected to fail to compile. Therefore, convert it into a
test that is expected to successfully compile.
The update to the GCC backend is speculative and untested.
Rollup of 9 pull requests
Successful merges:
- #104199 (Keep track of the start of the argument block of a closure)
- #105050 (Remove useless borrows and derefs)
- #105153 (Create a hacky fail-fast mode that stops tests at the first failure)
- #105164 (Restore `use` suggestion for `dyn` method call requiring `Sized`)
- #105193 (Disable coverage instrumentation for naked functions)
- #105200 (Remove useless filter in unused extern crate check.)
- #105201 (Do not call fn_sig on non-functions.)
- #105208 (Add AmbiguityError for inconsistent resolution for an import)
- #105214 (update Miri)
Failed merges:
r? `@ghost`
`@rustbot` modify labels: rollup
codegen-llvm: never combine DSOLocal and DllImport
Prevent DllImport from being attached to DSOLocal definitions in the LLVM IR. The combination makes no sense, since definitions local to the compilation unit will never be imported from external objects.
Additionally, LLVM will refuse the IR if it encounters the combination (introduced in [1]):
```
if (GV.hasDLLImportStorageClass())
Assert(!GV.isDSOLocal(),
"GlobalValue with DLLImport Storage is dso_local!", &GV);
```
Right now, codegen-llvm will only apply DllImport to constants and rely on call-stubs for functions. Hence, we simply extend the codegen of constants to skip DllImport for any local definitions.
This was discovered when switching the EFI targets to the static relocation model [2]. With this fixed, we can start another attempt at this.
[1] 509132b368
[2] https://github.com/rust-lang/rust/issues/101656
Print all features with --print target-features
This fixes `rustc --print target-features` with respect to aliases and tied features.
Before this change, the print command assumed that each LLVM feature corresponds exactly to one rustc feature. In the case of aliases and tied features, this assumption failed and some features (such as aarch64's "pacg") were missing. With this change, every target feature is listed.
Prevent DllImport from being attached to DSOLocal definitions in the
LLVM IR. The combination makes no sense, since definitions local to the
compilation unit will never be imported from external objects.
Additionally, LLVM will refuse the IR if it encounters the
combination (introduced in [1]):
if (GV.hasDLLImportStorageClass())
Assert(!GV.isDSOLocal(),
"GlobalValue with DLLImport Storage is dso_local!", &GV);
Right now, codegen-llvm will only apply DllImport to constants and rely
on call-stubs for functions. Hence, we simply extend the codegen of
constants to skip DllImport for any local definitions.
This was discovered when switching the EFI targets to the static
relocation model [2]. With this fixed, we can start another attempt at
this.
[1] 509132b368
[2] https://github.com/rust-lang/rust/issues/101656
After https://github.com/llvm/llvm-project/commit/8689f5e landed, LLVM takes the intersection of v8a and v8r as default.
This commit brings back v8a support by explicitly specifying v8a in the feature list.
This should solve #97724.
Use `as_deref` in compiler (but only where it makes sense)
This simplifies some code :3
(there are some changes that are not exacly `as_deref`, but more like "clever `Option`/`Result` method use")
Mark functions created for `raw-dylib` on x86 with DllImport storage class
Fix for #104453
## Issue Details
On x86 Windows, LLVM uses 'L' as the prefix for any private global symbols (`PrivateGlobalPrefix`), so when the `raw-dylib` feature creates an undecorated function symbol that begins with an 'L' LLVM misinterprets that as a private global symbol that it created and so fails the compilation at a later stage since such a symbol must have a definition.
## Fix Details
Mark the function we are creating for `raw-dylib` with `DllImport` storage class (this was already being done for MSVC at a later point for `callee::get_fn` but not for GNU (due to "backwards compatibility")): this will cause LLVM to prefix the name with `__imp_` and so it won't mistake it for a private global symbol.
Pass 128-bit C-style enum enumerator values to LLVM
Pass the full 128 bits of C-style enum enumerators through to LLVM. This means that debuginfo for C-style repr128 enums is now emitted correctly for DWARF platforms (as compared to not being correctly emitted on any platform).
Tracking issue: #56071
Improve generating Custom entry function
This commit is aimed at making compiler-generated entry functions (Basically just C `main` right now) more generic so other targets can do similar things for custom entry. This was initially implemented as part of https://github.com/rust-lang/rust/pull/100316.
Currently, this moves the entry function name and Call convention to the target spec.
Signed-off-by: Ayush Singh <ayushsingh1325@gmail.com>
Fix some misleading target feature aliases
This is the first half of a fix for #100752. It looks like these aliases were added in #78361 and slipped under the radar, as these features are not AVX512. These features _do_ add AVX512 instructions when used _in combination_ with AVX512F, but without AVX512F, these features still provide 128-bit and 256-bit vector instructions. A user might be mislead into thinking these features imply AVX512F (which is true of the actual AVX512 features). This PR allows using the names as defined by LLVM, which matches Intel documentation.
A future PR should change the `std::arch` intrinsics to use these names, and finally remove these aliases from rustc.
r? ```@workingjubilee```
cc ```@Amanieu```
For the next commit, `FunctionCx::codegen_*_terminator` need to take a
`&mut Bx` instead of consuming a `Bx`. This triggers a cascade of
similar changes across multiple functions. The resulting code is more
concise and replaces many `&mut bx` expressions with `bx`.
Perform simple scalar replacement of aggregates (SROA) MIR opt
This is a re-open of https://github.com/rust-lang/rust/pull/85796
I copied the debuginfo implementation (first commit) from `@eddyb's` own SROA PR.
This pass replaces plain field accesses by simple locals when possible.
To be eligible, the replaced locals:
- must not be enums or unions;
- must not be used whole;
- must not have their address taken.
The storage and deinit statements are duplicated on each created local.
cc `@tmiasko` who reviewed the former version of this PR.
interpret: support for per-byte provenance
Also factors the provenance map into its own module.
The third commit does the same for the init mask. I can move it in a separate PR if you prefer.
Fixes https://github.com/rust-lang/miri/issues/2181
r? `@oli-obk`
llvm: dwo only emitted when object code emitted
Fixes#103932.
`CompiledModule` should not think a DWARF object was emitted when a bitcode-only compilation has happened, this can confuse archive file creation (which expects to create an archive containing non-existent dwo files).
r? ``````@michaelwoerister``````
This commit is aimed at making compiler generated entry functions
(Basically just C `main` right now) more generic so other targets can do
similar things for custom entry. This was initially implemented as part
of https://github.com/rust-lang/rust/pull/100316.
Currently, this moves the entry function name and Call convention to the
target spec.
Signed-off-by: Ayush Singh <ayushsingh1325@gmail.com>
Fix Access Violation when using lld & ThinLTO on windows-msvc
Users report an AV at runtime of the compiled binary when using lld and ThinLTO on windows-msvc. The AV occurs when accessing a static value which is defined in one crate but used in another. Based on the disassembly of the cross-crate use, it appears that the use is not correctly linked with the definition and is instead assigned a garbage pointer value.
If we look at the symbol tables for each crates' obj file, we can see what is happening:
*lib.obj*:
```
COFF SYMBOL TABLE
...
00E 00000000 SECT2 notype External | _ZN10reproducer7memrchr2FN17h612b61ca0e168901E
...
```
*bin.obj*:
```
COFF SYMBOL TABLE
...
010 00000000 UNDEF notype External | __imp__ZN10reproducer7memrchr2FN17h612b61ca0e168901E
...
```
The use of the symbol has the "import" style symbol name but the declaration doesn't generate any symbol with the same name. As a result, linking the files generates a warning from lld:
> rust-lld: warning: bin.obj: locally defined symbol imported: reproducer::memrchr::FN::h612b61ca0e168901 (defined in lib.obj) [LNK4217]
and the symbol reference remains undefined at runtime leading to the AV.
To fix this, we just need to detect that we are performing ThinLTO (and thus, static linking) and omit the `dllimport` attribute on the extern item in LLVM IR.
Fixes#81408
`CompiledModule` should not think a DWARF object was emitted when a
bitcode-only compilation has happened, this can confuse archive file
creation (which expects to create an archive containing non-existent dwo
files).
Signed-off-by: David Wood <david.wood@huawei.com>
LLVM 16: Update RISCV data layout
The RISCV data layout was changed in 974e2e690b.
This updates all `riscv64*` targets, though I don't really know what the difference between the `gc` and `imac` ones is.
Passes `x test codegen` at LLVM head and with the currently bundled LLVM version. Without this patch, some tests fail with:
> error: internal compiler error: compiler/rustc_codegen_llvm/src/context.rs:192:13: data-layout for target `riscv64gc-unknown-none-elf`, `e-m:e-p:64:64-i64:64-i128:128-n64-S128`, differs from LLVM target's `riscv64` default layout, `e-m:e-p:64:64-i64:64-i128:128-n32:64-S128
Moved type_array function to rustc_codegen_ssa::BaseTypeMethods trait.
This allows using normal alloca function to create arrays as suggested in
https://github.com/rust-lang/rust/pull/104022.
Signed-off-by: Ayush Singh <ayushsingh1325@gmail.com>
LLVM 16: Switch to using MemoryEffects
This adapts the compiler to the changes required by 304f1d59ca.
AFAICT, `WriteOnly` isn't used by the compiler, all `ReadNone` uses were migrated and the remaining use of `ReadOnly` is only for function parameters.
To simplify the FFI, this PR uses an enum to represent `MemoryEffects` across the FFI boundary, which then gets mapped to the matching static factory method when constructing the attribute.
Fixes#103961.
`@rustbot` label +llvm-main
r? `@nikic`
asm: Work around LLVM bug on AArch64
Upstream issue: https://github.com/llvm/llvm-project/issues/58384
LLVM gets confused if we assign a 32-bit value to a 64-bit register, so pass the 32-bit register name to LLVM in that case.
asm: Match clang behavior for inlateout fixed register operands
We have 2 options for representing LLVM constraints for `inlateout` operands on a fixed register (e.g. `r0`): `={r0},0` or `={r0},{r0}`.
This PR changes the behavior to the latter, which matches the behavior of Clang since https://reviews.llvm.org/D87279.
Users report an AV at runtime of the compiled binary when using lld and
ThinLTO on windows-msvc. The AV occurs when accessing a static value
which is defined in one crate but used in another. Based on the
disassembly of the cross-crate use, it appears that the use is not
correctly linked with the definition and is instead assigned a garbage
pointer value.
If we look at the symbol tables for each crates' obj file, we can see
what is happening:
*lib.obj*:
```
COFF SYMBOL TABLE
...
00E 00000000 SECT2 notype External | _ZN10reproducer7memrchr2FN17h612b61ca0e168901E
...
```
*bin.obj*:
```
COFF SYMBOL TABLE
...
010 00000000 UNDEF notype External | __imp__ZN10reproducer7memrchr2FN17h612b61ca0e168901E
...
```
The use of the symbol has the "import" style symbol name but the
declaration doesn't generate any symbol with the same name. As a result,
linking the files generates a warning from lld:
> rust-lld: warning: bin.obj: locally defined symbol imported: reproducer::memrchr::FN::h612b61ca0e168901 (defined in lib.obj) [LNK4217]
and the symbol reference remains undefined at runtime leading to the AV.
To fix this, we just need to detect that we are performing ThinLTO (and
thus, static linking) and omit the `dllimport` attribute on the extern
item in LLVM IR.
The new implementation doesn't use weak lang items and instead changes
`#[alloc_error_handler]` to an attribute macro just like
`#[global_allocator]`.
The attribute will generate the `__rg_oom` function which is called by
the compiler-generated `__rust_alloc_error_handler`. If no `__rg_oom`
function is defined in any crate then the compiler shim will call
`__rdl_oom` in the alloc crate which will simply panic.
This also fixes link errors with `-C link-dead-code` with
`default_alloc_error_handler`: `__rg_oom` was previously defined in the
alloc crate and would attempt to reference the `oom` lang item, even if
it didn't exist. This worked as long as `__rg_oom` was excluded from
linking since it was not called.
This is a prerequisite for the stabilization of
`default_alloc_error_handler` (#102318).
Don't use usub.with.overflow intrinsic
The canonical form of a usub.with.overflow check in LLVM are separate sub + icmp instructions, rather than a usub.with.overflow intrinsic. Using usub.with.overflow will generally result in worse optimization potential.
The backend will attempt to form usub.with.overflow when it comes to actual instruction selection. This is not fully reliable, but I believe this is a better tradeoff than using the intrinsic in IR.
Fixes#103285.
