bootstrap: add support for running Miri on a file
This enables:
```
./x.py run src/tools/miri --stage 0 --args src/tools/miri/tests/pass/hello.rs
```
That can be super helpful for debugging.
Also avoid sharing the Miri sysroot dir with a system-wide (rustup-managed) installation of Miri.
Fixes https://github.com/rust-lang/rust/issues/76666
This allows open() to be called from other places in bootstrap (I need
this for Ferrocene), and it simplifies the callers by moving the
"was_invoked_explicitly" check into the function.
Distribute bootstrap in CI
This pre-compiles bootstrap from source and adds it to the existing `rust-dev` component. There are two main goals here:
1. Make it faster to build rust from source, both the first time and incrementally
2. Make it easier to add non-python entrypoints, since they can call out to bootstrap directly rather than having to figure out the right flags to pre-compile it. This second part is still in a bit of flux, see the tracking issue below for more information.
There are also several changes to make bootstrap able to run on a machine other than the one it was built (particularly around `config.src` and `config.out` detection). I (`@jyn514)` am slightly concerned these will regress unless tested - maybe we should add an automated test that runs bootstrap in a chroot or something? Unclear whether the effort is worth the test coverage.
Helps with https://github.com/rust-lang/rust/issues/94829.
Distribute json doc
# Overview
We add a new component, `rust-json-docs`, to distribute the JSON version of rustdoc's output for public compiler crates (i.e. `std`, `alloc`, `proc_macro`, `core` and `test`).
As discussed in #101383, we do not bundle this up as part of the existing `rust-docs` component since `rustdoc`'s JSON format is still unstable.
# Open questions / Doubts
I tried my best, but I never touched this codebase and I couldn't find much documentation on how `dist` works - I pattern-matched existing code, which might have led to some non-sensical choices in the eyes of people more familiar with the codebase. In particular, I am not sure if my choice of adding a new config flag is appropriate or if the decision to build/not build the JSON docs is more appropriately gated by one of the existing flags.
Any suggestion is more than welcome.
Closes#101383
This CL modifies the handing of env vars during the bootstrap process in
two ways:
1. Replaces '-' characters with '_' characters in target names to
increase compatibility with different shells
2. Passes Stage0 snapshot compiler related env vars to early invocations
of Cargo
- Add a new `bootstrap` component
Originally, we planned to combine this with the `rust-dev` component.
However, I realized that would force LLVM to be redownloaded whenever bootstrap is modified.
LLVM is a much larger download, so split this to get better caching.
- Build bootstrap for all tier 1 and 2 targets
This PR will fix some typos detected by [typos].
I only picked the ones I was sure were spelling errors to fix, mostly in
the comments.
[typos]: https://github.com/crate-ci/typos
session: stabilize split debuginfo on linux
Stabilize the `-Csplit-debuginfo` flag...
- ...on Linux for all values of the flag. Split DWARF has been implemented for a few months, hasn't had any bug reports and has had some promising benchmarking for incremental debug build performance.
- ..on other platforms for the default value. It doesn't make any sense that `-Csplit-debuginfo=packed` is unstable on Windows MSVC when that's the default behaviour, but keep the other values unstable.
Stabilize the `-Csplit-debuginfo` flag...
- ...on Linux for all values of the flag. Split DWARF has been
implemented for a few months, hasn't had any bug reports and has had
some promising benchmarking for incremental debug build performance.
- ..on other platforms for the default value. It doesn't make any sense
that `-Csplit-debuginfo=packed` is unstable on Windows MSVC when
that's the default behaviour, but keep the other values unstable.
Signed-off-by: David Wood <david.wood@huawei.com>
This builds `src/tools/rust-analyzer/crates/proc-macro-srv-cli` and
ships it as part of Rustc's dist component. This allows rust-analyzer's
proc macro support to work on all rustc versions (stable, beta and
nightly) starting now.
Enable check-cfg in stage0
Now that the bootstrap cargo supports `rustc-check-cfg` we can now enable it with `-Zcheck-cfg=output` and use it in `rustc_llvm` to unblock `--check-cfg` support in stage0.
r? `@Mark-Simulacrum`
Build the Clippy book as part of x.py doc
r? ``@ehuss`` since you said you would be interested in helping moving this forward.
cc ``@jyn514`` as part of the bootstrap team.
Fix caching bug in `download-rustc = true`
When moving this to rustbuild, I introduced a bug: if you had the file already downloaded, but
deleted the sysroot for whatever reason, rustbuil would fail to unpack the cached tarball.
This only affects people if they have a cached tarball, which is probably why we haven't seen an issue yet -
wiping `build/cache` would work around the issue, or just not deleting `build/$TARGET/stage2`.
Fixes the following error:
```
thread 'main' panicked at 'fs::read_dir(&lib_dir) failed with No such file or directory (os error 2) ("/home/jnelson/rust-lang/rust2/build/x86_64-unknown-linux-gnu/ci-rustc/lib")', config.rs:1563:20
```
r? ``@Mark-Simulacrum``
implement detail_exit but I'm not sure it is right.
not create new file and write detail exit in lib.rs
replace std::process::exit to detail_exit
that is not related to code runnning.
remove pub
bootstrap: Allow building individual crates
This aims to be as unintrusive as possible, but did still require adding a new `tail_args` field to all `Rustc` and `Std` steps.
New library and compiler crates are added to the sysroot as they are built, since it's useful to have e.g. just alloc and not std.
Fixes https://github.com/rust-lang/rust/issues/44293.
When moving this to rustbuild, I introduced a bug: if you had the file already downloaded, but
deleted the sysroot for whatever reason, rustbuil would fail to unpack the cached tarball.
This only affects people if they have a cached tarball, which is probably why we haven't seen an issue yet -
wiping `build/cache` would work around the issue, or just not deleting `build/$TARGET/stage2`.
Pass all paths to `Step::run` at once when using `ShouldRun::krate`
Helps with https://github.com/rust-lang/rust/pull/95503. The goal is to run `cargo test -p rustc_data_structures -p rustc_lint_defs` instead of `cargo test -p rustc_data_structures; cargo test -p rustc_lint_defs`, which should both recompile less and avoid replaying cached warnings.
This was surprisingly complicated. The main changes are:
1. Invert the order of iteration in `StepDescription::run`.
Previously, it did something like:
```python
for path in paths:
for (step, should_run) in should_runs:
if let Some(set) = should_run.pathset_for_path(path):
step.run(builder, set)
```
That worked ok for individual paths, but didn't allow passing more than one path at a time to `Step::run`
(since `pathset_for_paths` only had one path available to it).
Change it to instead look at the intersection of `paths` and `should_run.paths`:
```python
for (step, should_run) in should_runs:
if let Some(set) = should_run.pathset_for_paths(paths):
step.run(builder, set)
```
2. Change `pathset_for_path` to take multiple pathsets.
The goal is to avoid `x test library/alloc` testing *all* library crates, instead of just alloc.
The changes here are similarly subtle, to use the intersection between the paths rather than all
paths in `should_run.paths`. I added a test for the behavior to try and make it more clear.
Note that we use pathsets instead of just paths to allow for sets with multiple aliases (*cough* `all_krates` *cough*).
See the documentation added in the next commit for more detail.
3. Change `StepDescription::run` to explicitly handle 0 paths.
Before this was implicitly handled by the `for` loop, which just didn't excute when there were no paths.
Now it needs a check, to avoid trying to run all steps (this is a problem for steps that use `default_condition`).
4. Change `RunDescription` to have a list of pathsets, rather than a single path.
5. Remove paths as they're matched
This allows checking at the end that no invalid paths are left over.
Note that if two steps matched the same path, this will no longer run both;
but that's a bug anyway.
6. Handle suite paths separately from regular sets.
Running multiple suite paths at once instead of in separate `make_run` invocations is both tricky and not particularly useful.
The respective test Steps already handle this by introspecting the original paths.
Avoid having to deal with it by moving suite handling into a seperate loop than `PathSet::Set` checks.
`@rustbot` label +A-rustbuild
This was surprisingly complicated. The main changes are:
1. Invert the order of iteration in `StepDescription::run`.
Previously, it did something like:
```python
for path in paths:
for (step, should_run) in should_runs:
if let Some(set) = should_run.pathset_for_path(path):
step.run(builder, set)
```
That worked ok for individual paths, but didn't allow passing more than one path at a time to `Step::run`
(since `pathset_for_paths` only had one path available to it).
Change it to instead look at the intersection of `paths` and `should_run.paths`:
```python
for (step, should_run) in should_runs:
if let Some(set) = should_run.pathset_for_paths(paths):
step.run(builder, set)
```
2. Change `pathset_for_path` to take multiple pathsets.
The goal is to avoid `x test library/alloc` testing *all* library crates, instead of just alloc.
The changes here are similarly subtle, to use the intersection between the paths rather than all
paths in `should_run.paths`. I added a test for the behavior to try and make it more clear.
Note that we use pathsets instead of just paths to allow for sets with multiple aliases (*cough* `all_krates` *cough*).
See the documentation added in the next commit for more detail.
3. Change `StepDescription::run` to explicitly handle 0 paths.
Before this was implicitly handled by the `for` loop, which just didn't excute when there were no paths.
Now it needs a check, to avoid trying to run all steps (this is a problem for steps that use `default_condition`).
4. Change `RunDescription` to have a list of pathsets, rather than a single path.
5. Remove paths as they're matched
This allows checking at the end that no invalid paths are left over.
Note that if two steps matched the same path, this will no longer run both;
but that's a bug anyway.
6. Handle suite paths separately from regular sets.
Running multiple suite paths at once instead of in separate `make_run` invocations is both tricky and not particularly useful.
The respective test Steps already handle this by introspecting the original paths.
Avoid having to deal with it by moving suite handling into a seperate loop than `PathSet::Set` checks.
Add build metrics to rustbuild
This PR adds a new module of rustbuild, `ci_profiler`, whose job is to gather as much information as possible about the CI build as possible and store it in a JSON file uploaded to `ci-artifacts`. Right now for each step it collects:
* Type name and debug representation of the `Step` object.
* Duration of the step (excluding child steps).
* Systemwide CPU stats for the duration of the step (both single core and all cores).
* Which child steps were executed.
This is capable of replacing both the scripts to collect CPU stats and the `[TIMING]` lines in build logs (not yet removed, until we port our tooling to use the CI profiler). The format is also extensible to be able in the future to collect more information.
r? `@Mark-Simulacrum`
Re-add help_on_error for download-ci-llvm
Closes#97503
- Re-added `help_on_error` for `download_component()` and the downstream functions
- Removed dead code in `bootstrap.py`
Thanks `@jyn514` for the helpful tips!
(first contribution here, please let me know if I missed anything out!)
Finish bumping stage0
It looks like the last time had left some remaining cfg's -- which made me think
that the stage0 bump was actually successful. This brings us to a released 1.62
beta though.
This now brings us to cfg-clean, with the exception of check-cfg-features in bootstrap;
I'd prefer to leave that for a separate PR at this time since it's likely to be more tricky.
cc https://github.com/rust-lang/rust/pull/97147#issuecomment-1132845061
r? `@pietroalbini`
Move download-rustc from python to rustbuild
- Remove download-rustc handling from bootstrap.py
- Allow a custom `pattern` in `builder.unpack()`
- Only download rustc once another part of bootstrap depends on it.
This is somewhat necessary since the download functions rely on having a full
`Builder`, which isn't available until after config parsing finishes.
Helps with https://github.com/rust-lang/rust/issues/94829.
Print stderr consistently
Solves https://github.com/rust-lang/rust/issues/96712
I tried to follow what I perceived as the general consensus for error messages in boostrap i.e messages that were ..
* resulting from an Err(...) =>
* literally called as "Error: ...."
* by the end of the block scope forced to run a panic! or process::exit with a guaranteed non-zero error code.
It looks like the last time had left some remaining cfg's -- which made me think
that the stage0 bump was actually successful. This brings us to a released 1.62
beta though.
- Remove download-rustc handling from bootstrap.py
- Allow a custom `pattern` in `builder.unpack()`
- Only download rustc once another part of bootstrap depends on it.
This is somewhat necessary since the download functions rely on having a full
`Builder`, which isn't available until after config parsing finishes.
Avoid accidentally enabling unstable features in compilers (take 2)
This allows rustbuild to control whether crates can use nightly features or not.
It also prevents rustbuild from using nightly features itself.
This is #92261, but I fixed the CI error.
Enable cfg checking of cargo features for everything but std
This PR enable `cfg` checking of cargo features for everything but std, it also adds a `FIXME` to myself.
> Note: `std`, `alloc` and `core` imports some dependencies by #[path] (like
> backtrace, core_simd, std_float, ...), those dependencies have their own features
> but cargo isn't involved in the #[path] and so cannot pass the complete list of
> features, so for that reason we don't enable checking of features for std.
r? `@Mark-Simulacrum`
bootstrap: bsd platform flags for split debuginfo
Addresses https://github.com/rust-lang/rust/pull/96597#issuecomment-1118905025.
Bootstrap currently provides `-Zunstable-options` for OpenBSD when using split debuginfo - this commit provides it for all BSD targets.
We should probably work out a better way of handling the stability of the split debuginfo flag - all options for the flag are unstable but one of them is the default for each platform already.
cc `@m-ou-se`
r? `@Mark-Simulacrum`
[bootstrap] Give a better error when trying to run a path with no registered step
Before:
```
thread 'main' panicked at 'error: no rules matched invalid', src/bootstrap/builder.rs:287:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
After:
```
error: no `check` rules matched 'invalid'
help: run `x.py check --help --verbose` to show a list of available paths
note: if you are adding a new Step to bootstrap itself, make sure you register it with `describe!`
```
Bootstrap currently provides `-Zunstable-options` for OpenBSD when using
split debuginfo - this commit provides it for all BSD targets.
Signed-off-by: David Wood <david.wood@huawei.com>
openbsd: unbreak build on native platform
after #95612, only linux and windows target are build with `-Zunstable-options`, but others platforms might use `-Csplit-debuginfo`
currently, without this PR, the build of rustc on OpenBSD fails with:
```
Building stage0 tool unstable-book-gen (x86_64-unknown-openbsd)
running: "/data/semarie/build-rust/install_dir/beta/bin/cargo" "build" "--target" "x86_64-unknown-openbsd" "-Zbinary-dep-depinfo" "-j" "4" "-v" "--release" "--frozen" "--manifest-path"
"/data/semarie/build-rust/build_dir/rustc-nightly-src/src/tools/unstable-book-gen/Cargo.toml" "--message-format" "json-render-diagnostics"
error: failed to run `rustc` to learn about target-specific information
Caused by:
process didn't exit successfully: `/data/semarie/build-rust/build_dir/build/bootstrap/debug/rustc - --crate-name ___ --print=file-names --cfg=bootstrap -Csymbol-mangling-version=v0 -Zmacro-backtrace -Clink-args=-Wl,-z,origin
'-Clink-args=-Wl,-rpath,$ORIGIN/../lib' -Csplit-debuginfo=off -Ztls-model=initial-exec --target x86_64-unknown-openbsd --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro
--print=sysroot --print=cfg` (exit status: 1)
--- stdout
Did not run successfully: exit status: 1
"/data/semarie/build-rust/install_dir/beta/bin/rustc" "-" "--crate-name" "___" "--print=file-names" "--cfg=bootstrap" "-Csymbol-mangling-version=v0" "-Zmacro-backtrace" "-Clink-args=-Wl,-z,origin" "-Clink-args=-Wl,-rpath,$ORIGIN/../lib"
"-Csplit-debuginfo=off" "-Ztls-model=initial-exec" "--target" "x86_64-unknown-openbsd" "--crate-type" "bin" "--crate-type" "rlib" "--crate-type" "dylib" "--crate-type" "cdylib" "--crate-type" "staticlib" "--crate-type" "proc-macro"
"--print=sysroot" "--print=cfg" "-Wrust_2018_idioms" "-Wunused_lifetimes" "-Wsemicolon_in_expressions_from_macros" "-Dwarnings" "--sysroot" "/data/semarie/build-rust/install_dir/beta"
-------------
--- stderr
error: `-Csplit-debuginfo` is unstable on this platform
command did not execute successfully: "/data/semarie/build-rust/install_dir/beta/bin/cargo.bin" "build" "--target" "x86_64-unknown-openbsd" "-Zbinary-dep-depinfo" "-j" "4" "-v" "--release" "--frozen" "--manifest-path"
```
I am suspecting that all unix might be affected, but I am unsure about the right conditional to use. so I only added "openbsd" target inside it as I am able to test it.
rustc nightly built correctly with this PR on openbsd.
Before:
```
thread 'main' panicked at 'error: no rules matched invalid', src/bootstrap/builder.rs:287:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
After:
```
error: no `check` rules matched 'invalid'
help: run `x.py check --help --verbose` to show a list of available paths
note: if you are adding a new Step to bootstrap itself, make sure you register it with `describe!`
```
This tool will generate a JSON file with statistics about each
individual step to disk. It will be used in rust-lang/rust's CI to
replace the mix of scripts and log scraping we currently have to gather
this data.
Remove DebbugerScripts from bootstrap CLI
This PR implements #95992 and removes the debugger scripts from the bootstrap CLI.
I could not find a lot of documentation on the bootstrap binary so perhaps there's still some documentation to be updated.
after #95612, only linux and windows target are build with -Zunstable-options,
but others platforms might use -Csplit-debuginfo
add "openbsd" target in the list of platforms using it.
Move `download-ci-llvm` out of bootstrap.py
This is ready for review. It has been tested on Windows, Linux, and NixOS.
The second commit ports the changes from https://github.com/rust-lang/rust/pull/95234 to Rust; I can remove it if desired.
Helps with https://github.com/rust-lang/rust/issues/94829.
As a follow-up, this makes it possible to avoid downloading llvm until it's needed for building `rustc_llvm`; it would be nice to do that, but it shouldn't go in the first draft. It might also be possible to avoid requiring python until tests run (currently there's a check in `sanity.rs`), but I haven't looked too much into that.
`@rustbot` label +A-rustbuild
This attempts to keep the logic as close to the original python as possible.
`probably_large` has been removed, since it was always `True`, and UTF-8 paths are no longer supported when patching files for NixOS.
I can readd UTF-8 support if desired.
Note that this required making `llvm_link_shared` computed on-demand,
since we don't know whether it will be static or dynamic until we download LLVM from CI.
bootstrap: consolidate subcommand parsing and matching
There's several places where the x.py command names are matched as
strings, leading to some inconsistencies and opportunities for cleanup.
* Add Format, Clean, and Setup variants to builder::Kind.
* Use Kind to parse the x.py subcommand name (including aliases)
* Match on the subcommand Kind rather than strings when handling
options and help text.
* Several subcommands don't display any paths when run with `-h -v` even
though the help text indicates that they should. Fix this and refactor
so that manually keeping matches in sync isn't necessary.
Fixes#95937
bootstrap: add split-debuginfo config
Replace `run-dysutil` option with more general `split-debuginfo` option that works on all platforms.
r? `@Mark-Simulacrum`
Temporarily, only enable split debuginfo on Windows if not building with
the boostrap compiler as there is a bug that isn't fixed in the
bootstrap compiler which would result in `thorin` being run on Windows.
Signed-off-by: David Wood <david.wood@huawei.com>
This breaks on submodules (see #96188). Disable the assertion for now until I can think of a proper
fix.
This doesn't revert any of the changes in `Step`s themselves, only what
`ShouldRun::paths` does.
Require all paths passed to `ShouldRun::paths` to exist on disk
This has two benefits:
1. There is a clearer mental model of how bootstrap works. Steps correspond to paths on disk unless it's strictly impossible for them to do so (e.g. dist components).
2. Bootstrap has better checks for internal consistency. This caught several issues:
- `src/sanitizers` doesn't exist; I changed it to just be a `sanitizers` alias.
- `src/tools/lld` doesn't exist; I removed it, since `lld` alone already works.
- `src/llvm` doesn't exist; removed it since `llvm` and `src/llvm-project` both work.
- `src/lldb_batchmode.py` doesn't exist, it was moved to `src/etc`.
- `install` was still using `src/librustc` instead of `compiler/rustc`.
- None of the tools in `dist` / `install` allowed using `src/tools/X` to build them. This might be intentional - I can change them to aliases if you like.
Builds on https://github.com/rust-lang/rust/pull/95901 and should not be merged before.
There's several places where the x.py command names are matched as
strings, leading to some inconsistencies and opportunities for cleanup.
* Add Format, Clean, and Setup variants to builder::Kind.
* Use Kind to parse the x.py subcommand name (including aliases)
* Match on the subcommand Kind rather than strings when handling
options and help text.
* Several subcommands don't display any paths when run with `-h -v` even
though the help text indicates that they should. Fix this and refactor
so that manually keeping matches in sync isn't necessary.
Fixes#95937
This has two benefits:
1. There is a clearer mental model of how bootstrap works. Steps correspond to paths on disk unless it's strictly impossible for them to do so (e.g. dist components).
2. Bootstrap has better checks for internal consistency. This caught several issues:
- `src/sanitizers` doesn't exist; I changed it to just be a `sanitizers` alias.
- `src/tools/lld` doesn't exist; I removed it, since `lld` alone already works.
- `src/llvm` doesn't exist; removed it since `llvm` and `src/llvm-project` both work.
- `src/lldb_batchmode.py` doesn't exist, it was moved to `src/etc`.
- `install` was still using `src/librustc` instead of `compiler/rustc`.
- None of the tools in `dist` / `install` allowed using `src/tools/X` to build them. This might be intentional - I can change them to aliases if you like.
Running `./x.py build -h -v` shows a list of available build targets,
but the short alias `./x.py b -h -v` does not. Fix so that the aliases
behave the same as their spelled out counterparts.
Fix `x test src/tools/error_index_generator --stage {0,1}`
There were two fixes needed:
1. Use `top_stage` instead of `top_stage - 1`. There was a long and torturous comment about trying to match rustdoc's version, but it works better without the hard-coding than with (before it gave errors that `libtest.so` couldn't be found).
2. Make sure that `ci-llvm/lib` is added to LD_LIBRARY_PATH. Previously the error index would be unable to load LLVM for stage0 builds.
At some point we should probably have a discussion about how rustdoc stages should be numbered;
confusion between 0/1/2 has come up several times in bootstrap now. cc https://github.com/rust-lang/rust/issues/92538
Note that this is still broken when using `download-rustc = true` and `--stage 1`,
but that's *really* a corner case and should affect almost no one. `--stage {0,2}`
work fine with download-rustc.
Fixes https://github.com/rust-lang/rust/issues/80096.
There were two fixes needed:
1. Use `top_stage` instead of `top_stage - 1`. There was a long and torturous comment about trying to match rustdoc's version, but it works better without the hard-coding than with.
2. Make sure that `ci-llvm/lib` is added to LD_LIBRARY_PATH. Previously the error index would be unable to load LLVM for stage0 builds.
At some point we should probably have a discussion about how rustdoc stages should be numbered;
confusion between 0/1/2 has come up several times in bootstrap now.
Note that this is still broken when using `download-rustc = true` and `--stage 1`,
but that's *really* a corner case and should affect almost no one. `--stage {0,2}`
work fine with download-rustc.
This was so verbose before that it made it hard to see what effect the flag actually had.
Before:
```
Set({test::src/tools/tidy}) not skipped for "bootstrap::test::Tidy" -- not in [src/test/ui, src/test/mir-opt/, src/test/debuginfo, src/test/ui-fulldeps]
Skipping Suite(test::src/test/ui) because it is excluded
Suite(test::src/test/run-pass-valgrind) not skipped for "bootstrap::test::RunPassValgrind" -- not in [src/test/ui, src/test/mir-opt/, src/test/debuginfo, src/test/ui-fulldeps]
Skipping Suite(test::src/test/mir-opt) because it is excluded
Suite(test::src/test/codegen) not skipped for "bootstrap::test::Codegen" -- not in [src/test/ui, src/test/mir-opt/, src/test/debuginfo, src/test/ui-fulldeps]
Suite(test::src/test/codegen-units) not skipped for "bootstrap::test::CodegenUnits" -- not in [src/test/ui, src/test/mir-opt/, src/test/debuginfo, src/test/ui-fulldeps]
Suite(test::src/test/assembly) not skipped for "bootstrap::test::Assembly" -- not in [src/test/ui, src/test/mir-opt/, src/test/debuginfo, src/test/ui-fulldeps]
Suite(test::src/test/incremental) not skipped for "bootstrap::test::Incremental" -- not in [src/test/ui, src/test/mir-opt/, src/test/debuginfo, src/test/ui-fulldeps]
Skipping Suite(test::src/test/debuginfo) because it is excluded
Skipping Suite(test::src/test/ui-fulldeps) because it is excluded
... about 100 more lines ...
```
After:
```
Skipping Suite(test::src/test/ui) because it is excluded
Skipping Suite(test::src/test/mir-opt) because it is excluded
Skipping Suite(test::src/test/debuginfo) because it is excluded
Skipping Suite(test::src/test/ui-fulldeps) because it is excluded
```
Enable conditional checking of values in the Rust codebase
This pull-request enable conditional checking of (well known) values in the Rust codebase.
Well known values were added in https://github.com/rust-lang/rust/pull/94362. All the `target_*` values are taken from all the built-in targets which is why some extra values were needed do be added as they are not (yet ?) defined in any built-in targets.
r? `@Mark-Simulacrum`
this also fixes a bug where bootstrap would try to use the fake `rustc` binary built by bootstrap -
cargo puts it in a different directory when using `cargo run` instead of x.py
The majority of the code is only used by either rustbuild or
rustc_llvm's build script. Rust_build is compiled once for rustbuild and
once for every stage. This means that the majority of the code in this
crate is needlessly compiled multiple times. By moving only the code
actually used by the respective crates to rustbuild and rustc_llvm's
build script, this needless duplicate compilation is avoided.
First, this reverts the `CFLAGS`/`CXXFLAGS` of #93918. Those flags are
already read by `cc` and populated into `Build` earlier on in the
process. We shouldn't be overriding that based on `CFLAGS`, since `cc`
also respects overrides like `CFLAGS_{TARGET}` and `HOST_CFLAGS`, which
we want to take into account.
Second, this adds the same capability to specify target-specific
versions of `LDFLAGS` as we have through `cc` for the `C*` flags:
https://github.com/alexcrichton/cc-rs#external-configuration-via-environment-variables
Note that this also necessitated an update to compiletest to treat
CXXFLAGS separately from CFLAGS.
build: dist: defer PlainSourceTarball
Apparently it changes some tool sources and invalidates their fingerprints, forcing us to build them several times (before and after vendoring sources).
I have not dug into why vendoring actually invalidates the figreprints, but moving the vendoring lower in the pipeline seems to avoid the issue.
I could imagine that we somehow write a .cargo/config somewhere which somehow makes subsequent builds use the vendored deps but I was not able to find anything.
I checked the sizes of generated archives pre and post patch and their are the same, so I hope there is no functional change.
Fixes#93033
x.py has support for excluding some steps from the invocation, but
unfortunately that's not granular enough: some steps have the same name
in different modules, and that prevents excluding only *some* of them.
As a practical example, let's say you need to run everything in `./x.py
test` except for the standard library tests, as those tests require IPv6
and need to be executed on a separate machine. Before this commit, if
you were to just run this:
./x.py test --exclude library/std
...the execution would fail, as that would not only exclude running the
tests for the standard library, it would also exclude generating its
documentation (breaking linkchecker).
This commit adds support for an optional module annotation in --exclude
paths, allowing the user to choose which module to exclude from:
./x.py test --exclude test::library/std
This maintains backward compatibility, but also allows for more ganular
exclusion. More examples on how this works:
| `--exclude` | Docs | Tests |
| ------------------- | ------- | ------- |
| `library/std` | Skipped | Skipped |
| `doc::library/std` | Skipped | Run |
| `test::library/std` | Run | Skipped |
Note that the new behavior only works in the `--exclude` flag, and not
in other x.py arguments or flags yet.
Apparently it changes some tool sources and invalidates their fingerprints, forcing us to build them several times (before and after vendoring sources).
I have not dug into why vendoring actually invalidates the figreprints, but the moving the vendoring lower in the pipeline seems to avoid the issue.
I could imagine that we somehow write a .cargo/config somewhere which somehow makes subsequent builds use the vendored deps but I was not able to find anything.
I checked the sizes of generated archives pre and post patch and their are the same, so I hope there is not functional change.
Fixes#93033
DF_ORIGIN flag signifies that the object being loaded may make reference to the $ORIGIN substitution string.
Some implementations are just ignoring DF_ORIGIN and do substitution for $ORIGIN if present (whatever DF_ORIGIN pr
Set the flag inconditionally if rpath is wanted.
Currently the verbosity settings are:
- 2: RUSTC-SHIM envvars get spammed on every invocation, O(30) lines
cargo is passed -v which outputs CLI invocations, O(5) lines
- 3: cargo is passed -vv which outputs build script output, O(0-10) lines
This commit changes it to:
- 1: cargo is passed -v, O(5) lines
- 2: cargo is passed -vv, O(10) lines
- 3: RUSTC-SHIM envvars get spammed, O(30) lines
Assemble the compiler when running `x.py build`
Previously, there was no way to actually get binaries in
`build/$TARGET/stage1/bin` without building the standard library. This
makes it possible to build just the compiler. This can be useful when
the standard library isn't actually necessary for trying out your tests
(e.g. a bug that can be reproduced with only a `no_core` crate).
Closes https://github.com/rust-lang/rust/issues/73519.
Previously, there was no way to actually get binaries in
`build/$TARGET/stage1/bin` without building the standard library. This
makes it possible to build just the compiler. This can be useful when
the standard library isn't actually necessary for trying out your tests
(e.g. a bug that can be reproduced with only a `no_core` crate).
Remove hack ignoring unused attributes for stage 0 std
This seems to no longer be giving spurious errors when incremental is
enabled.
Closes https://github.com/rust-lang/rust/issues/58633.
Fix compiling other codegen backends when llvm is enabled
Extracted from #81746
Without this change rustbuild will not pass the required linker argument to find libllvm. While other backends likely don't use libllvm, it is necessary to be able to link against rustc_driver as the llvm backend is linked into it.
Now that Cargo deduplicates diagnostics from different targets, this doesn't flood the console with
duplicate errors.
Note that this doesn't add `--all-targets` in `Builder::cargo` directly because `impl Step for Std`
actually wants to omit `--all-targets` the first time while it's still building libtest.
When passed `--all-targets`, this warns that the option isn't needed, but still continues to compile.
Rollup of 7 pull requests
Successful merges:
- #85807 (bootstrap: Disable initial-exec TLS model on powerpc)
- #87761 (Fix overflow in rustc happening if the `err_count()` is reduced in a stage.)
- #87775 (Add hint for unresolved associated trait items if the trait has a single item)
- #87779 (Remove special case for statement `NodeId` assignment)
- #87787 (Use `C-unwind` ABI for `__rust_start_panic` in `panic_abort`)
- #87809 (Fix typo in the ptr documentation)
- #87816 (Sync rustc_codegen_cranelift)
Failed merges:
r? `@ghost`
`@rustbot` modify labels: rollup
The recursion_limit attribute avoids the following error:
```
error[E0275]: overflow evaluating the requirement `std::ptr::Unique<rustc_ast::Pat>: std::marker::Send`
|
= help: consider adding a `#![recursion_limit="256"]` attribute to your crate (`rustfmt_nightly`)
```
The `RustdocGUI::should_run` condition spawns `npm list` several times
which adds up to seconds of wall-time.
Evaluate the condition lazily to to keep `./x.py test tidy` and similar
short-running tasks fast.
## User-facing changes
- Intra-doc links to primitives that currently go to rust-lang.org/nightly/std/primitive.x.html will start going to channel that rustdoc was built with. Nightly will continue going to /nightly; Beta will link to /beta; stable compilers will link to /1.52.1 (or whatever version they were built as).
- Cross-crate links from std to core currently go to /nightly unconditionally. They will start going to /1.52.0 on stable channels (but remain the same on nightly channels).
- Intra-crate links from std to std (or core to core) currently go to the same URL they are hosted at; they will continue to do so. Notably, this is different from everything else because it can preserve the distinction between /stable and /1.52.0 by using relative links.
Note that "links" includes both intra-doc links and rustdoc's own
automatically generated hyperlinks.
## Implementation changes
- Update the testsuite to allow linking to /beta and /1.52.1 in docs
- Use an html_root_url for the standard library that's dependent on the channel
This avoids linking to nightly docs on stable.
- Update rustdoc to use channel-dependent links for primitives from an
unknown crate
- Set DOC_RUST_LANG_ORG_CHANNEL from bootstrap to ensure it's in sync
- Include doc.rust-lang.org in the channel
- Add rustfmt to `x.py check`
- Update Cargo.lock
- Remove rustfmt from the toolstate list
- Make rustfmt an in-tree tool
- Give an error on `x.py test rustfmt` if rustfmt fails to build or if tests fail
- Don't call `save_toolstate` when testing rustfmt
- Add clippy_dev to the rust workspace
Before, it would give an error that it wasn't either included or
excluded from the workspace:
```
error: current package believes it's in a workspace when it's not:
current: /home/joshua/rustc/src/tools/clippy/clippy_dev/Cargo.toml
workspace: /home/joshua/rustc/Cargo.toml
this may be fixable by adding `src/tools/clippy/clippy_dev` to the `workspace.members` array of the manifest located at: /home/joshua/rustc/Cargo.toml
Alternatively, to keep it out of the workspace, add the package to the `workspace.exclude` array, or add an empty `[workspace]` table to the package's manifest.
```
- Change clippy's copy of compiletest not to special-case
rust-lang/rust. Using OUT_DIR confused `clippy_dev` and it couldn't find
the test outputs. This is one of the reasons why `cargo dev bless` used
to silently do nothing (the others were that `CARGO_TARGET_DIR` and
`PROFILE` weren't set appropriately).
- Run clippy_dev on test failure
I tested this by removing a couple lines from a stderr file, and they
were correctly replaced.
- Fix clippy_dev warnings
Adds bootstrap rules to support installing rust-demangler.
When compiling with `-Z instrument-coverage`, the coverage reports are
generated by `llvm-cov`. `llvm-cov` includes a built-in demangler for
C++, and an option to supply an alternate demangler. For Rust, we have
`rust-demangler`, currently used in `rustc` coverage tests.
Fuchsia's toolchain for Rust is built via `./x.py install`. Fuchsia is
adding support for Rust coverage, and we need to include the
`rust-demangler` in the installed `bin` directory.
Configured rust-demangler as an in-tree extended tool.
Added tests to support `./x.py test rust-demangler`.
Install with extended tools by default only if `profiler = true`.
