Cleanup option parsing and config.toml.example
- Add an assertion that `link-shared = true` when `thin-lto = true`.
Previously, link-shared would be silently overwritten.
- Get rid of `Option<bool>` in bootstrap/config.rs. Set defaults
immediately instead of delaying until later in bootstrap. This makes
it easier to find what the default value is.
- Remove redundant `config.x = false` when the default was already false
- Set defaults for `bindir` in `default_opts()` instead of `parse()`
- Update `download-ci-llvm = if-supported` option to match bootstrap.py
- Remove redundant check for link_shared. Previously, it was checked twice.
- Update various options in config.toml.example to their defaults.
Previously, some options showed an example value instead of the
default value.
- Fix incorrect defaults in config.toml.example
+ `use-libcxx` defaults to false
+ Add missing `check-stage = 0`
+ Update several defaults to be conditional (e.g. `if incremental { 10 } else { 100 }`)
- Remove redundant defaults in prose
- Use the same comment for the default and target-dependent `musl-root`
- Fix typos
- Link to `cc_detect` for `cc` and `cxx`, since the logic is ... complicated.
- Update more defaults to better reflect how they actually get set
- Remove ignored `gpg-password-file` option
This stopped being used in
7704d35,
but was never removed from config.toml.
- Remove unused flags from `config.toml`
+ Disallow `infodir` and `localstatedir` in `config.toml`
+ Allow the flags in `./configure`, but give a warning that they will be
ignored.
+ Fix incorrect comment that `datadir` will be ignored.
Example output:
```
$ ./configure --set install.infodir=xxx
configure: processing command line
configure:
configure: install.infodir := xxx
configure: build.configure-args := ['--set', 'install.infodir=xxx']
warning: infodir will be ignored
configure:
configure: writing `config.toml` in current directory
configure:
configure: run `python /home/joshua/rustc3/x.py --help`
configure:
```
- Update CHANGELOG
cc https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/bootstrap.20defaults
- Add an assertion that `link-shared = true` when `thin-lto = true`.
Previously, link-shared would be silently overwritten.
- Get rid of `Option<bool>` in bootstrap/config.rs. Set defaults
immediately instead of delaying until later in bootstrap. This makes
it easier to find what the default value is.
- Remove redundant `config.x = false` when the default was already false
- Set defaults for `bindir` in `default_opts()` instead of `parse()`
- Update `download-ci-llvm = if-supported` option to match bootstrap.py
- Remove redundant check for link_shared. Previously, it was checked twice.
- Update various options in config.toml.example to their defaults.
Previously, some options showed an example value instead of the
default value.
- Fix incorrect defaults in config.toml.example
+ `use-libcxx` defaults to false
+ Add missing `check-stage = 0`
+ Update several defaults to be conditional (e.g. `if incremental { 10 } else { 100 }`)
- Remove redundant defaults in prose
- Use the same comment for the default and target-dependent `musl-root`
- Fix typos
- Link to `cc_detect` for `cc` and `cxx`, since the logic is ... complicated.
- Update more defaults to better reflect how they actually get set
- Remove ignored `gpg-password-file` option
This stopped being used in
7704d35acc,
but was never removed from config.toml.
- Remove unused flags from `config.toml`
+ Disallow `infodir` and `localstatedir` in `config.toml`
+ Allow the flags in `./configure`, but give a warning that they will be
ignored.
+ Fix incorrect comment that `datadir` will be ignored.
Example output:
```
$ ./configure --set install.infodir=xxx
configure: processing command line
configure:
configure: install.infodir := xxx
configure: build.configure-args := ['--set', 'install.infodir=xxx']
warning: infodir will be ignored
configure:
configure: writing `config.toml` in current directory
configure:
configure: run `python /home/joshua/rustc3/x.py --help`
configure:
```
- Update CHANGELOG
- Add "as an example" where appropriate
- Link to an issue instead of to ephemeral chats
Add `download-rustc = "if-unchanged"`
This allows keeping the setting to a fixed value without having to
toggle it when you want to work on the compiler instead of on tools.
This sets `BOOTSTRAP_DOWNLOAD_RUSTC` in bootstrap.py so rustbuild doesn't have to try and replicate its logic.
Helps with https://github.com/rust-lang/rust/issues/81930.
r? `@Mark-Simulacrum` cc `@camelid`
Adds bootstrap rules to support installing rust-demangler.
When compiling with `-Z instrument-coverage`, the coverage reports are
generated by `llvm-cov`. `llvm-cov` includes a built-in demangler for
C++, and an option to supply an alternate demangler. For Rust, we have
`rust-demangler`, currently used in `rustc` coverage tests.
Fuchsia's toolchain for Rust is built via `./x.py install`. Fuchsia is
adding support for Rust coverage, and we need to include the
`rust-demangler` in the installed `bin` directory.
Configured rust-demangler as an in-tree extended tool.
Added tests to support `./x.py test rust-demangler`.
Install with extended tools by default only if `profiler = true`.
config.toml parsing error improvements
Improve error messages for musl-libdir and wasi-root keys. Previously
the parser would panic with `unwrap()`. Now it prints
Target "wasm32-wasi" does not have a "wasi-root" key
(and similar for the `musl-libdir` field, which is used in target that
use musl)
Also update comments around wasi-root field to make it clear that the
field is only valid in wasm32-wasi target and needs to be moved to a
`[target.wasm32-wasi]` section to be valid.
Fixes#82317
---
r? `@Mark-Simulacrum`
Improve error messages for musl-libdir and wasi-root keys. Previously
the parser would panic with `unwrap()`. Now it prints
Target "wasm32-wasi" does not have a "wasi-root" key
(and similar for the `musl-libdir` field, which is used in target that
use musl)
Also update comments around wasi-root field to make it clear that the
field is only valid in wasm32-wasi target and needs to be moved to a
`[target.wasm32-wasi]` section to be valid.
Fixes#82317
- Use the same compiler for stage0 and stage1. This should be fixed at
some point (so bootstrap isn't constantly rebuilt).
- Make sure `x.py build` and `x.py check` work.
- Use `git merge-base` to determine the most recent commit to download.
- Copy stage0 to the various sysroots in `Sysroot`, and delegate to
Sysroot in Assemble. Leave all other code unchanged.
- Rename date -> key
This can also be a commit hash, so 'date' is no longer a good name.
- Add the commented-out option to config.toml.example
- Disable all steps by default when `download-rustc` is enabled
Most steps don't make sense when downloading a compiler, because they'll
be pre-built in the sysroot. Only enable the ones that might be useful,
in particular Rustdoc and all `check` steps.
At some point, this should probably enable other tools, but rustdoc is
enough to test out `download-rustc`.
- Don't print 'Skipping' twice in a row
Bootstrap forcibly enables a dry run if it isn't already set, so
previously it would print the message twice:
```
Skipping bootstrap::compile::Std because it is not enabled for `download-rustc`
Skipping bootstrap::compile::Std because it is not enabled for `download-rustc`
```
Now it correctly only prints once.
## Future work
- Add FIXME about supporting beta commits
- Debug logging will never work. This should be fixed.
`dsymutil` adds time to builds on Apple platforms for no clear benefit, and also
makes it more difficult for debuggers to find debug info. The compiler currently
defaults to running `dsymutil` to preserve its historical default, but when
compiling the compiler itself, we skip it by default since we know it's safe to
do so in that case.
This adds `rust.description` to the config as a descriptive string to be
appended to `rustc --version` output, which is also used in places like
debuginfo `DW_AT_producer`. This may be useful for supplementary build
information, like distro-specific package versions.
For example, in Fedora 33, `gcc --version` outputs:
gcc (GCC) 10.2.1 20201016 (Red Hat 10.2.1-6)
With this change, we can add similar vendor info to `rustc --version`.
Support enable/disable sanitizers/profiler per target
This PR add options under `[target.*]` of `config.toml` which can enable or disable sanitizers/profiler runtime for corresponding target.
If these options are empty, the global options under `[build]` will take effect.
Fix#78329
Update `changelog-seen` in config.toml.example
This got out of sync when the version was bumped last time in #77133
Long-term we may want to find an easier way to maintain this that
doesn't require bumping the version in three different places. Off the
top of my head I can't think of anything, though. It _is_ documented in src/bootstrap/README.md, although I don't know how many people read that.
r? @Mark-Simulacrum
cc @spastorino
This got out of sync when the version was bumped last time.
Long-term we may want to find an easier way to maintain this that
doesn't require bumping the version in three different places. Off the
top of my head I can't think of anything, though.
- Add a changelog and instructions for updating it
- Use `changelog-seen` in `config.toml` and `VERSION` in bootstrap to determine whether the changelog has been read
- Nag people if they haven't read the x.py changelog
+ Print message twice to make sure it's seen
- Give different error messages depending on whether the version needs to be updated or added
- Allow including defaults in `src/bootstrap/defaults` using `profile = "..."`
- Add default config files
- Combine config files using the merge dependency.
- Add comments to default config files
- Add a README asking to open an issue if the defaults are bad
- Give a loud error if trying to merge `.target`, since it's not
currently supported
- Use an exhaustive match
- Use `<none>` in config.toml.example to avoid confusion
- Fix bugs in `Merge` derives
Previously, it would completely ignore the profile defaults if there
were any settings in `config.toml`. I sent an email to the `merge` maintainer
asking them to make the behavior in this commit the default.
This introduces a new dependency on `merge` that hasn't yet been vetted.
I want to improve the output when `include = "x"` isn't found:
```
thread 'main' panicked at 'fs::read_to_string(&file) failed with No such file or directory (os error 2) ("configuration file did not exist")', src/bootstrap/config.rs:522:28
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
failed to run: /home/joshua/rustc/build/bootstrap/debug/bootstrap test tidy
Build completed unsuccessfully in 0:00:00
```
However that seems like it could be fixed in a follow-up.
Make the default stage for x.py configurable
This also allows configuring each sub-command individually.
Possibly #76617 should land before this? I don't feel strongly either way, I don't mind waiting.
Closes https://github.com/rust-lang/rust/issues/76165.
r? `@Mark-Simulacrum`
This allows configuring the default stage for each sub-command individually.
- Normalize the stage as early as possible, so there's no confusion
about which to use.
- Don't add an explicit `stage` option in config.toml
This offers no more flexibility than `*_stage` and makes it confusing
which takes precedence.
- Always give `--stage N` precedence over config.toml
- Fix bootstrap tests
This changes the tests to go through `Config::parse` so that they test
the actual defaults, not the dummy ones provided by `default_opts`. To
make this workable (and independent of the environment), it does not
read `config.toml` for tests.
Add a dedicated debug-logging option to config.toml
`@Mark-Simulacrum` and I were talking in zulip and we found that turning on debug/trace logging in rustc is fairly confusing, as it effectively depends on debug-assertions and is not documented as such. `@Mark-Simulacrum` mentioned that we should probably have a separate option for logging anyways.
this diff adds that, having the option follow debug-assertions (so everyone's existing config.toml should be fine) and if the option is false
to test I ran ./x.py test <something> twice, once with `debug-logging = false` and once with `debug-logging = true` and made sure i only saw trace's when it was true
Previously, the CLI --target/--host definitions and configured options differed
in their effect: when setting these on the CLI, only the passed triples would be
compiled for, while in config.toml we would also compile for the build triple
and any host triples. This is needlessly confusing; users expect --target and
--host to be identical to editing the configuration file.
The new behavior is to respect --host and --target when passed as the *only*
configured triples (no triples are implicitly added). The default for --host is
the build triple, and the default for --target is the host triple(s), either
configured or the default build triple.
rustbuild: Build tests with LLD if `use-lld = true` was passed
Addresses https://github.com/rust-lang/rust/pull/76127#discussion_r479932392.
Our test suite is generally ready to run with an explicitly specified linker (https://github.com/rust-lang/rust/pull/45191),
so LLD specified with `use-lld = true` works as well.
Only 4 tests fail (on `x86_64-pc-windows-msvc`):
```
ui/panic-runtime/lto-unwind.rs
run-make-fulldeps/debug-assertions
run-make-fulldeps/foreign-exceptions
run-make-fulldeps/test-harness
```
All of them are legitimate issues with LLD (or at least with combination Rust+LLD) and manifest in segfaults on access to TLS (https://github.com/rust-lang/rust/pull/76127#issuecomment-683473325). UPD: These issues are caused by https://github.com/rust-lang/rust/issues/72145 and appear because I had `-Ctarget-cpu=native` set.
UPD: Further commits build tests with LLD for non-MSVC targets and propagate LLD to more places when `use-lld` is enabled.
Move `rustllvm` into `compiler/rustc_llvm`
The `rustllvm` directory is not self-contained, it contains C++ code built by a build script of the `rustc_llvm` crate which is then linked into that crate.
So it makes sense to make `rustllvm` a part of `rustc_llvm` and move it into its directory.
I replaced `rustllvm` with more obvious `llvm-wrapper` as the subdirectory name, but something like `llvm-adapter` would work as well, other suggestions are welcome.
To make things more confusing, the Rust side of FFI functions defined in `rustllvm` can be found in `rustc_codegen_llvm` rather than in `rustc_llvm`. Perhaps they need to be moved as well, but this PR doesn't do that.
The presence of multiple LLVM-related directories in `src` (`llvm-project`, `rustllvm`, `librustc_llvm`, `librustc_codegen_llvm` and their predecessors) historically confused me and made me wonder about their purpose.
With this PR we will have LLVM itself (`llvm-project`), a FFI crate (`rustc_llvm`, kind of `llvm-sys`) and a codegen backend crate using LLVM through the FFI crate (`rustc_codegen_llvm`).
- Use prettier syntax for codegen-units defaults
- Remove comment about parallelism that only made sense for specific
values of codegen-units
- Be more specific about what `verbose-tests` does
Ninja substantially improves LLVM build time. On a 96-way system, using
Make took 248s, and using Ninja took 161s, a 35% improvement.
We already require a variety of tools to build Rust. If someone wants to
build without Ninja (for instance, to minimize the set of packages
required to bootstrap a new target), they can easily set `ninja=false`
in `config.toml`. Our defaults should help people build Rust (and LLVM)
faster, to speed up development.
Improve defaults in x.py
- Make the default stage dependent on the subcommand
- Don't build stage1 rustc artifacts with x.py build --stage 1. If this is what you want, use x.py build --stage 2 instead, which gives you a working libstd.
- Change default debuginfo when debug = true from 2 to 1
I tried to fix CI to use `--stage 2` everywhere it currently has no stage, but I might have missed a spot.
This does not update much of the documentation - most of it is in https://github.com/rust-lang/rustc-dev-guide/ or https://github.com/rust-lang/rust-forge and will need a separate PR.
See individual commits for a detailed rationale of each change.
See also the MCP: https://github.com/rust-lang/compiler-team/issues/326
r? @Mark-Simulacrum , but anyone is free to give an opinion.
From [a conversation in discord](https://discordapp.com/channels/442252698964721669/443151243398086667/719200989269327882):
> Linking seems to consume all available RAM, leading to the OS to swap memory to disk and slowing down everything in the process
Compiling itself doesn't seem to take up as much RAM, and I'm only looking to check whether a minimal testcase can be compiled by rustc, where the runtime performance isn't much of an issue
> do you have debug = true or debuginfo-level = 2 in config.toml?
> if so I think that results in over 2GB of debuginfo nowadays and is likely the culprit
> which might mean we're giving out bad advice :(
Anecdotally, this sped up my stage 1 build from 15 to 10 minutes.
This still adds line numbers, it only removes variable and type information.
- Improve wording for debuginfo description
Co-authored-by: Teymour Aldridge <42674621+teymour-aldridge@users.noreply.github.com>
bootstrap: Configurable musl libdir
Make it possible to customize the location of musl libdir using
musl-libdir in config.toml, e.g., to use lib64 instead of lib.
If config.toml `profiler = false`, the test/mir-opt/instrument_coverage
test is ignored. Otherwise, this patch ensures the profiler_runtime is
loaded when -Zinstrument-coverage is enabled. Confirmed that this works
for MacOS.
Provide separate option for std debug asserts
On local one-off benchmarking of libcore metadata-only, debug asserts in std are a significant hit (15s to 20s). Provide an option for compiler developers to disable them. A build with a nightly compiler is around 10s, for reference.
This allows configuring the directory for build artifacts, instead of having it always be ./build. This means you can set it to a constant location, letting you reuse the same cache while working in several different directories.
The configuration lives in config.toml under build.build-dir. By default, it keeps the existing default of ./build, but it can be configured to any relative or absolute path. Additionally, it allows making outputs relative to the root of the git repository using $ROOT.
Make LLVM version suffix independent of rustc version on dev channel
Remove rustc version from LLVM version suffix on dev channel,
avoiding the need for full rebuilds when switching between
branches with different LLVM submodule & rustc version.
Note: To avoid full rebuild, on subsequent LLVM submodule update, copy the
current value of `LLVM_VERSION_SUFFIX` from `build/*/llvm/build/CMakeCache.txt`,
to `version-suffix` in `config.toml`.
Remove rustc version from LLVM version suffix on dev channel, avoiding
the need for full rebuilds when moving between commits with different
LLVM submodule & rustc version.
Enable Control Flow Guard in rustbuild
Now that Rust supports Control Flow Guard (#68180), add a config.toml option to build the standard library with CFG enabled.
r? @nagisa
allow rustfmt key in [build] section
Permit using `rustfmt` in `config.toml`. It will allow to not download `rustfmt` binary, which is not possible for at least some tiers-3 platforms.
Fixes: #67624
r? @Mark-Simulacrum
This commit builds on #65501 continue to simplify the build system and
compiler now that we no longer have multiple LLVM backends to ship by
default. Here this switches the compiler back to what it once was long
long ago, which is linking LLVM directly to the compiler rather than
dynamically loading it at runtime. The `codegen-backends` directory of
the sysroot no longer exists and all relevant support in the build
system is removed. Note that `rustc` still supports a dynamically loaded
codegen backend as it did previously, it just no longer supports
dynamically loaded codegen backends in its own sysroot.
Additionally as part of this the `librustc_codegen_llvm` crate now once
again explicitly depends on all of its crates instead of implicitly
loading them through the sysroot. This involved filling out its
`Cargo.toml` and deleting all the now-unnecessary `extern crate`
annotations in the header of the crate. (this in turn required adding a
number of imports for names of macros too).
The end results of this change are:
* Rustbuild's build process for the compiler as all the "oh don't forget
the codegen backend" checks can be easily removed.
* Building `rustc_codegen_llvm` is much simpler since it's simply
another compiler crate.
* Managing the dependencies of `rustc_codegen_llvm` is much simpler since
it's "just another `Cargo.toml` to edit"
* The build process should be a smidge faster because there's more
parallelism in the main rustc build step rather than splitting
`librustc_codegen_llvm` out to its own step.
* The compiler is expected to be slightly faster by default because the
codegen backend does not need to be dynamically loaded.
* Disabling LLVM as part of rustbuild is still supported, supporting
multiple codegen backends is still supported, and dynamic loading of a
codegen backend is still supported.
With #65251 landed there's no need to build two LLVM backends and ship
them with rustc, every target we have now uses the same LLVM backend!
This removes the `src/llvm-emscripten` submodule and additionally
removes all support from rustbuild for building the emscripten LLVM
backend. Multiple codegen backend support is left in place for now, and
this is intended to be an easy 10-15 minute win on CI times by avoiding
having to build LLVM twice.
- Compatible with Emscripten 1.38.46-upstream or later upstream.
- Refactors the Emscripten target spec to share code with other wasm
targets.
- Replaces the old incorrect wasm32 C call ABI with the correct one,
preserving the old one as wasm32_bindgen_compat for wasm-bindgen
compatibility.
- Updates the varargs ABI used by Emscripten and deletes the old one.
- Removes the obsolete wasm32-experimental-emscripten target.
- Uses EMCC_CFLAGS on CI to avoid the timeout problems with #63649.
- Refactors the Emscripten target spec to share code with other wasm
targets.
- Replaces the incorrect wasm32 C call ABI with the old asmjs
version, which is correct for both wasm32 and JS.
- Updates the varargs ABI used by Emscripten and deletes the old one.
- Removes the obsolete wasm32-experimental-emscripten target.
- Temporarily makes Emscripten targets use panic=abort by default
because supporting unwinding will require an LLVM patch.
This commit removes the `wasm_syscall` feature from the
wasm32-unknown-unknown build of the standard library. This feature was
originally intended to allow an opt-in way to interact with the
operating system in a posix-like way but it was never stabilized.
Nowadays with the advent of the `wasm32-wasi` target that should
entirely replace the intentions of the `wasm_syscall` feature.
This commit moves RISC-V from the experimental LLVM targets to the
regular LLVM targets. RISC-V was made non-experimental in
https://reviews.llvm.org/rL366399
I have also sorted the list of LLVM targets, and changed the code
around setting llvm_exp_targets (and its default) to match the code
setting llvm_targets (and its default), ensuring future changes to
the defaults, as LLVM targets become stable, affect as few places as
possible.
