rustbuild: Build tests with LLD if `use-lld = true` was passed
Addresses https://github.com/rust-lang/rust/pull/76127#discussion_r479932392.
Our test suite is generally ready to run with an explicitly specified linker (https://github.com/rust-lang/rust/pull/45191),
so LLD specified with `use-lld = true` works as well.
Only 4 tests fail (on `x86_64-pc-windows-msvc`):
```
ui/panic-runtime/lto-unwind.rs
run-make-fulldeps/debug-assertions
run-make-fulldeps/foreign-exceptions
run-make-fulldeps/test-harness
```
All of them are legitimate issues with LLD (or at least with combination Rust+LLD) and manifest in segfaults on access to TLS (https://github.com/rust-lang/rust/pull/76127#issuecomment-683473325). UPD: These issues are caused by https://github.com/rust-lang/rust/issues/72145 and appear because I had `-Ctarget-cpu=native` set.
UPD: Further commits build tests with LLD for non-MSVC targets and propagate LLD to more places when `use-lld` is enabled.
Move `rustllvm` into `compiler/rustc_llvm`
The `rustllvm` directory is not self-contained, it contains C++ code built by a build script of the `rustc_llvm` crate which is then linked into that crate.
So it makes sense to make `rustllvm` a part of `rustc_llvm` and move it into its directory.
I replaced `rustllvm` with more obvious `llvm-wrapper` as the subdirectory name, but something like `llvm-adapter` would work as well, other suggestions are welcome.
To make things more confusing, the Rust side of FFI functions defined in `rustllvm` can be found in `rustc_codegen_llvm` rather than in `rustc_llvm`. Perhaps they need to be moved as well, but this PR doesn't do that.
The presence of multiple LLVM-related directories in `src` (`llvm-project`, `rustllvm`, `librustc_llvm`, `librustc_codegen_llvm` and their predecessors) historically confused me and made me wonder about their purpose.
With this PR we will have LLVM itself (`llvm-project`), a FFI crate (`rustc_llvm`, kind of `llvm-sys`) and a codegen backend crate using LLVM through the FFI crate (`rustc_codegen_llvm`).
- Use prettier syntax for codegen-units defaults
- Remove comment about parallelism that only made sense for specific
values of codegen-units
- Be more specific about what `verbose-tests` does
Ninja substantially improves LLVM build time. On a 96-way system, using
Make took 248s, and using Ninja took 161s, a 35% improvement.
We already require a variety of tools to build Rust. If someone wants to
build without Ninja (for instance, to minimize the set of packages
required to bootstrap a new target), they can easily set `ninja=false`
in `config.toml`. Our defaults should help people build Rust (and LLVM)
faster, to speed up development.
Improve defaults in x.py
- Make the default stage dependent on the subcommand
- Don't build stage1 rustc artifacts with x.py build --stage 1. If this is what you want, use x.py build --stage 2 instead, which gives you a working libstd.
- Change default debuginfo when debug = true from 2 to 1
I tried to fix CI to use `--stage 2` everywhere it currently has no stage, but I might have missed a spot.
This does not update much of the documentation - most of it is in https://github.com/rust-lang/rustc-dev-guide/ or https://github.com/rust-lang/rust-forge and will need a separate PR.
See individual commits for a detailed rationale of each change.
See also the MCP: https://github.com/rust-lang/compiler-team/issues/326
r? @Mark-Simulacrum , but anyone is free to give an opinion.
From [a conversation in discord](https://discordapp.com/channels/442252698964721669/443151243398086667/719200989269327882):
> Linking seems to consume all available RAM, leading to the OS to swap memory to disk and slowing down everything in the process
Compiling itself doesn't seem to take up as much RAM, and I'm only looking to check whether a minimal testcase can be compiled by rustc, where the runtime performance isn't much of an issue
> do you have debug = true or debuginfo-level = 2 in config.toml?
> if so I think that results in over 2GB of debuginfo nowadays and is likely the culprit
> which might mean we're giving out bad advice :(
Anecdotally, this sped up my stage 1 build from 15 to 10 minutes.
This still adds line numbers, it only removes variable and type information.
- Improve wording for debuginfo description
Co-authored-by: Teymour Aldridge <42674621+teymour-aldridge@users.noreply.github.com>
bootstrap: Configurable musl libdir
Make it possible to customize the location of musl libdir using
musl-libdir in config.toml, e.g., to use lib64 instead of lib.
If config.toml `profiler = false`, the test/mir-opt/instrument_coverage
test is ignored. Otherwise, this patch ensures the profiler_runtime is
loaded when -Zinstrument-coverage is enabled. Confirmed that this works
for MacOS.
Provide separate option for std debug asserts
On local one-off benchmarking of libcore metadata-only, debug asserts in std are a significant hit (15s to 20s). Provide an option for compiler developers to disable them. A build with a nightly compiler is around 10s, for reference.
This allows configuring the directory for build artifacts, instead of having it always be ./build. This means you can set it to a constant location, letting you reuse the same cache while working in several different directories.
The configuration lives in config.toml under build.build-dir. By default, it keeps the existing default of ./build, but it can be configured to any relative or absolute path. Additionally, it allows making outputs relative to the root of the git repository using $ROOT.
Make LLVM version suffix independent of rustc version on dev channel
Remove rustc version from LLVM version suffix on dev channel,
avoiding the need for full rebuilds when switching between
branches with different LLVM submodule & rustc version.
Note: To avoid full rebuild, on subsequent LLVM submodule update, copy the
current value of `LLVM_VERSION_SUFFIX` from `build/*/llvm/build/CMakeCache.txt`,
to `version-suffix` in `config.toml`.
Remove rustc version from LLVM version suffix on dev channel, avoiding
the need for full rebuilds when moving between commits with different
LLVM submodule & rustc version.
Enable Control Flow Guard in rustbuild
Now that Rust supports Control Flow Guard (#68180), add a config.toml option to build the standard library with CFG enabled.
r? @nagisa
allow rustfmt key in [build] section
Permit using `rustfmt` in `config.toml`. It will allow to not download `rustfmt` binary, which is not possible for at least some tiers-3 platforms.
Fixes: #67624
r? @Mark-Simulacrum
This commit builds on #65501 continue to simplify the build system and
compiler now that we no longer have multiple LLVM backends to ship by
default. Here this switches the compiler back to what it once was long
long ago, which is linking LLVM directly to the compiler rather than
dynamically loading it at runtime. The `codegen-backends` directory of
the sysroot no longer exists and all relevant support in the build
system is removed. Note that `rustc` still supports a dynamically loaded
codegen backend as it did previously, it just no longer supports
dynamically loaded codegen backends in its own sysroot.
Additionally as part of this the `librustc_codegen_llvm` crate now once
again explicitly depends on all of its crates instead of implicitly
loading them through the sysroot. This involved filling out its
`Cargo.toml` and deleting all the now-unnecessary `extern crate`
annotations in the header of the crate. (this in turn required adding a
number of imports for names of macros too).
The end results of this change are:
* Rustbuild's build process for the compiler as all the "oh don't forget
the codegen backend" checks can be easily removed.
* Building `rustc_codegen_llvm` is much simpler since it's simply
another compiler crate.
* Managing the dependencies of `rustc_codegen_llvm` is much simpler since
it's "just another `Cargo.toml` to edit"
* The build process should be a smidge faster because there's more
parallelism in the main rustc build step rather than splitting
`librustc_codegen_llvm` out to its own step.
* The compiler is expected to be slightly faster by default because the
codegen backend does not need to be dynamically loaded.
* Disabling LLVM as part of rustbuild is still supported, supporting
multiple codegen backends is still supported, and dynamic loading of a
codegen backend is still supported.
With #65251 landed there's no need to build two LLVM backends and ship
them with rustc, every target we have now uses the same LLVM backend!
This removes the `src/llvm-emscripten` submodule and additionally
removes all support from rustbuild for building the emscripten LLVM
backend. Multiple codegen backend support is left in place for now, and
this is intended to be an easy 10-15 minute win on CI times by avoiding
having to build LLVM twice.
