Support `x test --stage 1 ui-fulldeps`
`@Nilstrieb` had an excellent idea the other day: the same way that rustdoc is able to load `rustc_driver` from the sysroot, ui-fulldeps tests should also be able to load it from the sysroot. That allows us to run fulldeps tests with stage1, without having to fully rebuild the compiler twice. It does unfortunately have the downside that we're building the tests with the *bootstrap* compiler, not the in-tree sources, but since most of the fulldeps tests are for the *API* of the compiler, that seems ok.
I think it's possible to extend this to `run-make-fulldeps`, but I've run out of energy for tonight.
- Move `plugin` tests into a subdirectory.
Plugins are loaded at runtime with `dlopen` and so require the ABI of the running compile to match the ABI of the compiler linked with `rustc_driver`. As a result they can't be supported in stage 1 and have to use `// ignore-stage1`.
- Remove `ignore-stage1` from most non-plugin tests
- Ignore diagnostic tests in stage 1. Even though this requires a stage 2 build to load rustc_driver, it's primarily testing the error message that the *running* compiler emits when the diagnostic struct is malformed.
- Pass `-Zforce-unstable-if-unmarked` in stage1, not just stage2. That allows running `hash-stable-is-unstable` in stage1, since it now suggests adding `rustc_private` to enable loading the crates.
- Add libLLVM.so to the stage0 target sysroot, to allow fulldeps tests that act as custom drivers to load it at runtime.
- Pass `--sysroot stage0-sysroot` in compiletest so that we use the correct version of std.
- Move a few lint tests from ui-fulldeps to ui
These had an `aux-build:lint-group-plugin-test.rs` that they never actually loaded with `feature(plugin)` nor tested. I removed the unused aux-build and they pass fine with stage 1.
Fixes https://github.com/rust-lang/rust/issues/75905.
Avoid including dry run steps in the build metrics
Including steps executed during the dry run will result in a duplication of all the steps in the build metrics, which just adds noise.
Nils had an excellent idea the other day: the same way that rustdoc is
able to load `rustc_driver` from the sysroot, ui-fulldeps tests should
also be able to load it from the sysroot. That allows us to run fulldeps
tests with stage1, without having to fully rebuild the compiler twice.
It does unfortunately have the downside that we're running the tests on
the *bootstrap* compiler, not the in-tree sources, but since most of the
fulldeps tests are for the *API* of the compiler, that seems ok.
I think it's possible to extend this to `run-make-fulldeps`, but I've
run out of energy for tonight.
- Move `plugin` tests into a subdirectory.
Plugins are loaded at runtime with `dlopen` and so require the ABI of
the running compile to match the ABI of the compiler linked with
`rustc_driver`. As a result they can't be supported in stage 1 and have
to use `// ignore-stage1`.
- Remove `ignore-stage1` from most non-plugin tests
- Ignore diagnostic tests in stage 1. Even though this requires a stage
2 build to load rustc_driver, it's primarily testing the error message
that the *running* compiler emits when the diagnostic struct is malformed.
- Pass `-Zforce-unstable-if-unmarked` in stage1, not just stage2. That
allows running `hash-stable-is-unstable` in stage1, since it now
suggests adding `rustc_private` to enable loading the crates.
- Add libLLVM.so to the stage0 target sysroot, to allow fulldeps tests
that act as custom drivers to load it at runtime.
- Pass `--sysroot stage0-sysroot` in compiletest so that we use the
correct version of std.
This commit resolves an internal FIXME note within the bootstrap by implementing submodule detection.
This is accomplished through an iterative process over the `.gitmodules` file.
Signed-off-by: ozkanonur <work@onurozkan.dev>
Rename 'src/bootstrap/native.rs' to llvm.rs
Fixed#108240
Renamed 'native.rs' to 'llvm.rs', also moved `TestHelpers` to `test.rs`. Replaced all the `native.rs` occurrences at `src/bootstrap` files to `llvm.rs`
Permit the MIR inliner to inline diverging functions
This heuristic prevents inlining of `hint::unreachable_unchecked`, which in turn makes `Option/Result::unwrap_unchecked` a bad inlining candidate. I looked through the changes to `core`, `alloc`, `std`, and `hashbrown` by hand and they all seem reasonable. Let's see how this looks in perf...
---
Based on rustc-perf it looks like this regresses ctfe-stress, and the cachegrind diff indicates that this regression is in `InterpCx::statement`. I don't know how to do any deeper analysis because that function is _enormous_ in the try toolchain, which has no debuginfo in it. And a local build produces significantly different codegen for that function, even with LTO.
refactor `fn bootstrap::builder::Builder::compiler_for` logic
- check compiler stage before forcing for stage2.
- check if download_rustc is not set before forcing for stage1.
resolves#109286
- check compiler stage before forcing for stage2.
- check if download_rustc is not set before forcing for stage1.
Signed-off-by: ozkanonur <work@onurozkan.dev>
bootstrap: document tidy
Enable documentation of tidy, as suggested in #106803. Jyn mentioned they should probably be added to `doc.rust-lang.org`, how should that be done?
Set `LIBC_CHECK_CFG=1` when building Rust code in bootstrap
Downstream forks of the Rust compiler might want to use a custom `libc` to add support for targets that are not yet available upstream. Adding a patch to replace `libc` with a custom one would cause compilation errors though, because Cargo would interpret the custom `libc` as part of the workspace, and apply the check-cfg lints on it.
Since https://github.com/rust-lang/libc/pull/3037, the `libc` build script emits check-cfg flags only when the `LIBC_CHECK_CFG` environment variable is set, so this PR allows the use of custom `libc`s.
Import rust-installer & adjust compression settings
This brings in rust-lang/rust-installer#123, which enables a larger compression window (8 -> 64MB) amongst other changes to the xz compression settings. The net effect should be smaller compressed tarballs which will decrease bandwidth usage for
static.rust-lang.org, download times, and decompression time.
This comes at the cost of higher baseline requirements for running rustup to use these files, which we believe should be largely acceptable (running rustc is likely to use at least this much memory) but if we get specific reports we may explore options to decrease impact (e.g., using the gzip tarballs automatically in rustup).
To simplify iteration on compression settings this also imports the rust-lang/rust-installer submodule, it is now hosted fully inside rust-lang/rust. Once we land this I'll file a followup to add a note to that repo and we can subsequently archive it.
--
CI times for dist-x86_64-linux builds:
* threads=6, master - 2h 50m
* threads=1, new - 3h 40m
* threads=6, new - 2h 50m
Split `x setup` sub-actions to CLI arguments
Closes#107846
This adds a new `none` profile option which simply skips the `config.toml` step. It also adds `hook` and `vscode` subcommands, for installing the pre-push hook and getting `settings.json` respectively.
Most tests involving save-analysis were removed, but I kept a few where
the `-Zsave-analysis` was an add-on to the main thing being tested,
rather than the main thing being tested.
For `x.py install`, the `rust-analysis` target has been removed.
For `x.py dist`, the `rust-analysis` target has been kept in a
degenerate form: it just produces a single file `reduced.json`
indicating that save-analysis has been removed. This is necessary for
rustup to keep working.
Closes#43606.
Update mdbook
This updates mdbook from 0.4.21 to 0.4.25. The list of changes is [here](https://github.com/rust-lang/mdBook/blob/master/CHANGELOG.md#mdbook-0425). The only user-visible changes are some changes around the theme picker, and change to the copy-to-clipboard ignoring hidden lines.
Internally there were some dependency updates and small fixes.
This also updates `clap` from 4.0.15 to 4.0.32 whose changelog is [here](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md#4032---2022-12-22). This impacts tools like cargo. I don't see anything particularly noteworthy there, though there are some small user-visible changes.
Unfortunately this required adding a hack for building `rustix` with a bootstrap tool. The comment explains why. I am unable to think of some other workaround (or even a cleaner way to set the rustflag). Ideas are welcome if you can think of alternatives. I'm struggling to even think of a long-term solution, other than asking projects not to do auto-nightly feature detection.
One medium-term solution is to avoid the clap dependency for the mdbook library (which is how rustix gets pulled in). That is one of my goals for the 0.5 release of mdbook, but that probably won't happen until later this year. It would also require dropping clap from `rustbook` and using some other means to parse arguments (there's only two options, so it can probably be done manually).
Before:
```
Testing ["rustc_interface"] stage0 (aarch64-unknown-linux-gnu -> aarch64-unknown-linux-gnu)
```
After:
```
Testing {rustc_interface} stage0 (aarch64-unknown-linux-gnu -> aarch64-unknown-linux-gnu)
```
Note there is a slight consistency between `build` and `test`: The
former doesn't print "compiler artifacts". It would be annoying to fix
and doesn't hurt anything, so I left it be.
```
; x t rustc_interface --stage 0 --dry-run
Testing {rustc_interface} stage0 (aarch64-unknown-linux-gnu -> aarch64-unknown-linux-gnu)
; x b rustc_interface --stage 0 --dry-run
Building {rustc_interface} stage0 compiler artifacts (aarch64-unknown-linux-gnu -> aarch64-unknown-linux-gnu)
```
Previously, clean only supported `--stage 0` for specific crates.
The new `crate_description` function generates a string that looks
like
```
: {rustc_query_impl}
```
Rollup of 6 pull requests
Successful merges:
- #105978 (Mark `proc_macro_decls_static` as always used)
- #106051 (Allow building std with cranelift)
- #106056 (Make `sess.bug` much less noisy)
- #106057 (Give a more helpful error for "trimmed_def_paths constructed")
- #106058 (Fix the issue number in comment for as_local_call_operand)
- #106059 (Avoid running the `Profile` step twice on `x setup`)
Failed merges:
r? `@ghost`
`@rustbot` modify labels: rollup
Allow building std with cranelift
- Don't pass llvm-specific args when using cranelift
- Don't use `asm` in compiler_builtins when using cranelift
r? `@bjorn3` cc `@Mark-Simulacrum`
LLVM_CONFIG_PATH is no longer supported as of LLVM 16, switch to
using the cmake module instead.
We separately return the llvm-config and cmake directory paths,
because llvm-config always refers to the host binary, while
the cmake directory is for the target triple.
Fix `x setup -h -v` should work
r? `@jyn514`
I have to convert profile to path and back in order to remove special-casing in bootstrap. I also check for `dry_run` so that `config.toml` and/ or `.git/hooks/pre-push` will not be created if `--dry-run` is specified.
Please help me see if this is ok, thanks alot!
Add prototype to generate `COPYRIGHT` from REUSE metadata
This PR adds a prototype to generate the `COPYRIGHT` file from the metadata gathered with REUSE. There are two new tools:
* `src/tools/collect-license-metadata` invokes REUSE, parses its output and stores a concise JSON representation of the metadata in `src/etc/license-metadata.json`.
* `src/tools/generate-copyright` parses the metadata generated above, (in the future will) gather crate dependencies metadata, and renders the `COPYRIGHT.md` file.
Note that since the contents of those files are currently incorrect, rather than outputting in the paths above, the files will be stored in `build/` and not committed. This will be changed once we're confident about the metadata.
Eventually, `src/etc/license-metadata.json` will be committed into the repository and verified to be up to date by CI (similar to our GitHub Actions configuration), to avoid having people install REUSE on their local machine in most cases.
You can see the (incorrect) generated files in https://gist.github.com/pietroalbini/3f3f22b6f9cc8533abf7494b6a50cf97.
r? `@pnkfelix`
copy doc output files by format
This pr provides copying doc outputs by checking output format without removing output directory on each trigger.
Resolves#103785
bootstrap: add support for running Miri on a file
This enables:
```
./x.py run src/tools/miri --stage 0 --args src/tools/miri/tests/pass/hello.rs
```
That can be super helpful for debugging.
Also avoid sharing the Miri sysroot dir with a system-wide (rustup-managed) installation of Miri.
Fixes https://github.com/rust-lang/rust/issues/76666
This allows open() to be called from other places in bootstrap (I need
this for Ferrocene), and it simplifies the callers by moving the
"was_invoked_explicitly" check into the function.
Distribute bootstrap in CI
This pre-compiles bootstrap from source and adds it to the existing `rust-dev` component. There are two main goals here:
1. Make it faster to build rust from source, both the first time and incrementally
2. Make it easier to add non-python entrypoints, since they can call out to bootstrap directly rather than having to figure out the right flags to pre-compile it. This second part is still in a bit of flux, see the tracking issue below for more information.
There are also several changes to make bootstrap able to run on a machine other than the one it was built (particularly around `config.src` and `config.out` detection). I (`@jyn514)` am slightly concerned these will regress unless tested - maybe we should add an automated test that runs bootstrap in a chroot or something? Unclear whether the effort is worth the test coverage.
Helps with https://github.com/rust-lang/rust/issues/94829.
Distribute json doc
# Overview
We add a new component, `rust-json-docs`, to distribute the JSON version of rustdoc's output for public compiler crates (i.e. `std`, `alloc`, `proc_macro`, `core` and `test`).
As discussed in #101383, we do not bundle this up as part of the existing `rust-docs` component since `rustdoc`'s JSON format is still unstable.
# Open questions / Doubts
I tried my best, but I never touched this codebase and I couldn't find much documentation on how `dist` works - I pattern-matched existing code, which might have led to some non-sensical choices in the eyes of people more familiar with the codebase. In particular, I am not sure if my choice of adding a new config flag is appropriate or if the decision to build/not build the JSON docs is more appropriately gated by one of the existing flags.
Any suggestion is more than welcome.
Closes#101383
This CL modifies the handing of env vars during the bootstrap process in
two ways:
1. Replaces '-' characters with '_' characters in target names to
increase compatibility with different shells
2. Passes Stage0 snapshot compiler related env vars to early invocations
of Cargo
- Add a new `bootstrap` component
Originally, we planned to combine this with the `rust-dev` component.
However, I realized that would force LLVM to be redownloaded whenever bootstrap is modified.
LLVM is a much larger download, so split this to get better caching.
- Build bootstrap for all tier 1 and 2 targets
This PR will fix some typos detected by [typos].
I only picked the ones I was sure were spelling errors to fix, mostly in
the comments.
[typos]: https://github.com/crate-ci/typos
session: stabilize split debuginfo on linux
Stabilize the `-Csplit-debuginfo` flag...
- ...on Linux for all values of the flag. Split DWARF has been implemented for a few months, hasn't had any bug reports and has had some promising benchmarking for incremental debug build performance.
- ..on other platforms for the default value. It doesn't make any sense that `-Csplit-debuginfo=packed` is unstable on Windows MSVC when that's the default behaviour, but keep the other values unstable.
Stabilize the `-Csplit-debuginfo` flag...
- ...on Linux for all values of the flag. Split DWARF has been
implemented for a few months, hasn't had any bug reports and has had
some promising benchmarking for incremental debug build performance.
- ..on other platforms for the default value. It doesn't make any sense
that `-Csplit-debuginfo=packed` is unstable on Windows MSVC when
that's the default behaviour, but keep the other values unstable.
Signed-off-by: David Wood <david.wood@huawei.com>
This builds `src/tools/rust-analyzer/crates/proc-macro-srv-cli` and
ships it as part of Rustc's dist component. This allows rust-analyzer's
proc macro support to work on all rustc versions (stable, beta and
nightly) starting now.
Enable check-cfg in stage0
Now that the bootstrap cargo supports `rustc-check-cfg` we can now enable it with `-Zcheck-cfg=output` and use it in `rustc_llvm` to unblock `--check-cfg` support in stage0.
r? `@Mark-Simulacrum`
Build the Clippy book as part of x.py doc
r? ``@ehuss`` since you said you would be interested in helping moving this forward.
cc ``@jyn514`` as part of the bootstrap team.
Fix caching bug in `download-rustc = true`
When moving this to rustbuild, I introduced a bug: if you had the file already downloaded, but
deleted the sysroot for whatever reason, rustbuil would fail to unpack the cached tarball.
This only affects people if they have a cached tarball, which is probably why we haven't seen an issue yet -
wiping `build/cache` would work around the issue, or just not deleting `build/$TARGET/stage2`.
Fixes the following error:
```
thread 'main' panicked at 'fs::read_dir(&lib_dir) failed with No such file or directory (os error 2) ("/home/jnelson/rust-lang/rust2/build/x86_64-unknown-linux-gnu/ci-rustc/lib")', config.rs:1563:20
```
r? ``@Mark-Simulacrum``
implement detail_exit but I'm not sure it is right.
not create new file and write detail exit in lib.rs
replace std::process::exit to detail_exit
that is not related to code runnning.
remove pub
bootstrap: Allow building individual crates
This aims to be as unintrusive as possible, but did still require adding a new `tail_args` field to all `Rustc` and `Std` steps.
New library and compiler crates are added to the sysroot as they are built, since it's useful to have e.g. just alloc and not std.
Fixes https://github.com/rust-lang/rust/issues/44293.
When moving this to rustbuild, I introduced a bug: if you had the file already downloaded, but
deleted the sysroot for whatever reason, rustbuil would fail to unpack the cached tarball.
This only affects people if they have a cached tarball, which is probably why we haven't seen an issue yet -
wiping `build/cache` would work around the issue, or just not deleting `build/$TARGET/stage2`.
Pass all paths to `Step::run` at once when using `ShouldRun::krate`
Helps with https://github.com/rust-lang/rust/pull/95503. The goal is to run `cargo test -p rustc_data_structures -p rustc_lint_defs` instead of `cargo test -p rustc_data_structures; cargo test -p rustc_lint_defs`, which should both recompile less and avoid replaying cached warnings.
This was surprisingly complicated. The main changes are:
1. Invert the order of iteration in `StepDescription::run`.
Previously, it did something like:
```python
for path in paths:
for (step, should_run) in should_runs:
if let Some(set) = should_run.pathset_for_path(path):
step.run(builder, set)
```
That worked ok for individual paths, but didn't allow passing more than one path at a time to `Step::run`
(since `pathset_for_paths` only had one path available to it).
Change it to instead look at the intersection of `paths` and `should_run.paths`:
```python
for (step, should_run) in should_runs:
if let Some(set) = should_run.pathset_for_paths(paths):
step.run(builder, set)
```
2. Change `pathset_for_path` to take multiple pathsets.
The goal is to avoid `x test library/alloc` testing *all* library crates, instead of just alloc.
The changes here are similarly subtle, to use the intersection between the paths rather than all
paths in `should_run.paths`. I added a test for the behavior to try and make it more clear.
Note that we use pathsets instead of just paths to allow for sets with multiple aliases (*cough* `all_krates` *cough*).
See the documentation added in the next commit for more detail.
3. Change `StepDescription::run` to explicitly handle 0 paths.
Before this was implicitly handled by the `for` loop, which just didn't excute when there were no paths.
Now it needs a check, to avoid trying to run all steps (this is a problem for steps that use `default_condition`).
4. Change `RunDescription` to have a list of pathsets, rather than a single path.
5. Remove paths as they're matched
This allows checking at the end that no invalid paths are left over.
Note that if two steps matched the same path, this will no longer run both;
but that's a bug anyway.
6. Handle suite paths separately from regular sets.
Running multiple suite paths at once instead of in separate `make_run` invocations is both tricky and not particularly useful.
The respective test Steps already handle this by introspecting the original paths.
Avoid having to deal with it by moving suite handling into a seperate loop than `PathSet::Set` checks.
`@rustbot` label +A-rustbuild
This was surprisingly complicated. The main changes are:
1. Invert the order of iteration in `StepDescription::run`.
Previously, it did something like:
```python
for path in paths:
for (step, should_run) in should_runs:
if let Some(set) = should_run.pathset_for_path(path):
step.run(builder, set)
```
That worked ok for individual paths, but didn't allow passing more than one path at a time to `Step::run`
(since `pathset_for_paths` only had one path available to it).
Change it to instead look at the intersection of `paths` and `should_run.paths`:
```python
for (step, should_run) in should_runs:
if let Some(set) = should_run.pathset_for_paths(paths):
step.run(builder, set)
```
2. Change `pathset_for_path` to take multiple pathsets.
The goal is to avoid `x test library/alloc` testing *all* library crates, instead of just alloc.
The changes here are similarly subtle, to use the intersection between the paths rather than all
paths in `should_run.paths`. I added a test for the behavior to try and make it more clear.
Note that we use pathsets instead of just paths to allow for sets with multiple aliases (*cough* `all_krates` *cough*).
See the documentation added in the next commit for more detail.
3. Change `StepDescription::run` to explicitly handle 0 paths.
Before this was implicitly handled by the `for` loop, which just didn't excute when there were no paths.
Now it needs a check, to avoid trying to run all steps (this is a problem for steps that use `default_condition`).
4. Change `RunDescription` to have a list of pathsets, rather than a single path.
5. Remove paths as they're matched
This allows checking at the end that no invalid paths are left over.
Note that if two steps matched the same path, this will no longer run both;
but that's a bug anyway.
6. Handle suite paths separately from regular sets.
Running multiple suite paths at once instead of in separate `make_run` invocations is both tricky and not particularly useful.
The respective test Steps already handle this by introspecting the original paths.
Avoid having to deal with it by moving suite handling into a seperate loop than `PathSet::Set` checks.
