Enable cfg checking of cargo features for everything but std
This PR enable `cfg` checking of cargo features for everything but std, it also adds a `FIXME` to myself.
> Note: `std`, `alloc` and `core` imports some dependencies by #[path] (like
> backtrace, core_simd, std_float, ...), those dependencies have their own features
> but cargo isn't involved in the #[path] and so cannot pass the complete list of
> features, so for that reason we don't enable checking of features for std.
r? `@Mark-Simulacrum`
bootstrap: bsd platform flags for split debuginfo
Addresses https://github.com/rust-lang/rust/pull/96597#issuecomment-1118905025.
Bootstrap currently provides `-Zunstable-options` for OpenBSD when using split debuginfo - this commit provides it for all BSD targets.
We should probably work out a better way of handling the stability of the split debuginfo flag - all options for the flag are unstable but one of them is the default for each platform already.
cc `@m-ou-se`
r? `@Mark-Simulacrum`
[bootstrap] Give a better error when trying to run a path with no registered step
Before:
```
thread 'main' panicked at 'error: no rules matched invalid', src/bootstrap/builder.rs:287:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
After:
```
error: no `check` rules matched 'invalid'
help: run `x.py check --help --verbose` to show a list of available paths
note: if you are adding a new Step to bootstrap itself, make sure you register it with `describe!`
```
Bootstrap currently provides `-Zunstable-options` for OpenBSD when using
split debuginfo - this commit provides it for all BSD targets.
Signed-off-by: David Wood <david.wood@huawei.com>
openbsd: unbreak build on native platform
after #95612, only linux and windows target are build with `-Zunstable-options`, but others platforms might use `-Csplit-debuginfo`
currently, without this PR, the build of rustc on OpenBSD fails with:
```
Building stage0 tool unstable-book-gen (x86_64-unknown-openbsd)
running: "/data/semarie/build-rust/install_dir/beta/bin/cargo" "build" "--target" "x86_64-unknown-openbsd" "-Zbinary-dep-depinfo" "-j" "4" "-v" "--release" "--frozen" "--manifest-path"
"/data/semarie/build-rust/build_dir/rustc-nightly-src/src/tools/unstable-book-gen/Cargo.toml" "--message-format" "json-render-diagnostics"
error: failed to run `rustc` to learn about target-specific information
Caused by:
process didn't exit successfully: `/data/semarie/build-rust/build_dir/build/bootstrap/debug/rustc - --crate-name ___ --print=file-names --cfg=bootstrap -Csymbol-mangling-version=v0 -Zmacro-backtrace -Clink-args=-Wl,-z,origin
'-Clink-args=-Wl,-rpath,$ORIGIN/../lib' -Csplit-debuginfo=off -Ztls-model=initial-exec --target x86_64-unknown-openbsd --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro
--print=sysroot --print=cfg` (exit status: 1)
--- stdout
Did not run successfully: exit status: 1
"/data/semarie/build-rust/install_dir/beta/bin/rustc" "-" "--crate-name" "___" "--print=file-names" "--cfg=bootstrap" "-Csymbol-mangling-version=v0" "-Zmacro-backtrace" "-Clink-args=-Wl,-z,origin" "-Clink-args=-Wl,-rpath,$ORIGIN/../lib"
"-Csplit-debuginfo=off" "-Ztls-model=initial-exec" "--target" "x86_64-unknown-openbsd" "--crate-type" "bin" "--crate-type" "rlib" "--crate-type" "dylib" "--crate-type" "cdylib" "--crate-type" "staticlib" "--crate-type" "proc-macro"
"--print=sysroot" "--print=cfg" "-Wrust_2018_idioms" "-Wunused_lifetimes" "-Wsemicolon_in_expressions_from_macros" "-Dwarnings" "--sysroot" "/data/semarie/build-rust/install_dir/beta"
-------------
--- stderr
error: `-Csplit-debuginfo` is unstable on this platform
command did not execute successfully: "/data/semarie/build-rust/install_dir/beta/bin/cargo.bin" "build" "--target" "x86_64-unknown-openbsd" "-Zbinary-dep-depinfo" "-j" "4" "-v" "--release" "--frozen" "--manifest-path"
```
I am suspecting that all unix might be affected, but I am unsure about the right conditional to use. so I only added "openbsd" target inside it as I am able to test it.
rustc nightly built correctly with this PR on openbsd.
Before:
```
thread 'main' panicked at 'error: no rules matched invalid', src/bootstrap/builder.rs:287:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
After:
```
error: no `check` rules matched 'invalid'
help: run `x.py check --help --verbose` to show a list of available paths
note: if you are adding a new Step to bootstrap itself, make sure you register it with `describe!`
```
Remove DebbugerScripts from bootstrap CLI
This PR implements #95992 and removes the debugger scripts from the bootstrap CLI.
I could not find a lot of documentation on the bootstrap binary so perhaps there's still some documentation to be updated.
after #95612, only linux and windows target are build with -Zunstable-options,
but others platforms might use -Csplit-debuginfo
add "openbsd" target in the list of platforms using it.
Move `download-ci-llvm` out of bootstrap.py
This is ready for review. It has been tested on Windows, Linux, and NixOS.
The second commit ports the changes from https://github.com/rust-lang/rust/pull/95234 to Rust; I can remove it if desired.
Helps with https://github.com/rust-lang/rust/issues/94829.
As a follow-up, this makes it possible to avoid downloading llvm until it's needed for building `rustc_llvm`; it would be nice to do that, but it shouldn't go in the first draft. It might also be possible to avoid requiring python until tests run (currently there's a check in `sanity.rs`), but I haven't looked too much into that.
`@rustbot` label +A-rustbuild
This attempts to keep the logic as close to the original python as possible.
`probably_large` has been removed, since it was always `True`, and UTF-8 paths are no longer supported when patching files for NixOS.
I can readd UTF-8 support if desired.
Note that this required making `llvm_link_shared` computed on-demand,
since we don't know whether it will be static or dynamic until we download LLVM from CI.
bootstrap: consolidate subcommand parsing and matching
There's several places where the x.py command names are matched as
strings, leading to some inconsistencies and opportunities for cleanup.
* Add Format, Clean, and Setup variants to builder::Kind.
* Use Kind to parse the x.py subcommand name (including aliases)
* Match on the subcommand Kind rather than strings when handling
options and help text.
* Several subcommands don't display any paths when run with `-h -v` even
though the help text indicates that they should. Fix this and refactor
so that manually keeping matches in sync isn't necessary.
Fixes#95937
bootstrap: add split-debuginfo config
Replace `run-dysutil` option with more general `split-debuginfo` option that works on all platforms.
r? `@Mark-Simulacrum`
Temporarily, only enable split debuginfo on Windows if not building with
the boostrap compiler as there is a bug that isn't fixed in the
bootstrap compiler which would result in `thorin` being run on Windows.
Signed-off-by: David Wood <david.wood@huawei.com>
This breaks on submodules (see #96188). Disable the assertion for now until I can think of a proper
fix.
This doesn't revert any of the changes in `Step`s themselves, only what
`ShouldRun::paths` does.
Require all paths passed to `ShouldRun::paths` to exist on disk
This has two benefits:
1. There is a clearer mental model of how bootstrap works. Steps correspond to paths on disk unless it's strictly impossible for them to do so (e.g. dist components).
2. Bootstrap has better checks for internal consistency. This caught several issues:
- `src/sanitizers` doesn't exist; I changed it to just be a `sanitizers` alias.
- `src/tools/lld` doesn't exist; I removed it, since `lld` alone already works.
- `src/llvm` doesn't exist; removed it since `llvm` and `src/llvm-project` both work.
- `src/lldb_batchmode.py` doesn't exist, it was moved to `src/etc`.
- `install` was still using `src/librustc` instead of `compiler/rustc`.
- None of the tools in `dist` / `install` allowed using `src/tools/X` to build them. This might be intentional - I can change them to aliases if you like.
Builds on https://github.com/rust-lang/rust/pull/95901 and should not be merged before.
There's several places where the x.py command names are matched as
strings, leading to some inconsistencies and opportunities for cleanup.
* Add Format, Clean, and Setup variants to builder::Kind.
* Use Kind to parse the x.py subcommand name (including aliases)
* Match on the subcommand Kind rather than strings when handling
options and help text.
* Several subcommands don't display any paths when run with `-h -v` even
though the help text indicates that they should. Fix this and refactor
so that manually keeping matches in sync isn't necessary.
Fixes#95937
This has two benefits:
1. There is a clearer mental model of how bootstrap works. Steps correspond to paths on disk unless it's strictly impossible for them to do so (e.g. dist components).
2. Bootstrap has better checks for internal consistency. This caught several issues:
- `src/sanitizers` doesn't exist; I changed it to just be a `sanitizers` alias.
- `src/tools/lld` doesn't exist; I removed it, since `lld` alone already works.
- `src/llvm` doesn't exist; removed it since `llvm` and `src/llvm-project` both work.
- `src/lldb_batchmode.py` doesn't exist, it was moved to `src/etc`.
- `install` was still using `src/librustc` instead of `compiler/rustc`.
- None of the tools in `dist` / `install` allowed using `src/tools/X` to build them. This might be intentional - I can change them to aliases if you like.
Running `./x.py build -h -v` shows a list of available build targets,
but the short alias `./x.py b -h -v` does not. Fix so that the aliases
behave the same as their spelled out counterparts.
Fix `x test src/tools/error_index_generator --stage {0,1}`
There were two fixes needed:
1. Use `top_stage` instead of `top_stage - 1`. There was a long and torturous comment about trying to match rustdoc's version, but it works better without the hard-coding than with (before it gave errors that `libtest.so` couldn't be found).
2. Make sure that `ci-llvm/lib` is added to LD_LIBRARY_PATH. Previously the error index would be unable to load LLVM for stage0 builds.
At some point we should probably have a discussion about how rustdoc stages should be numbered;
confusion between 0/1/2 has come up several times in bootstrap now. cc https://github.com/rust-lang/rust/issues/92538
Note that this is still broken when using `download-rustc = true` and `--stage 1`,
but that's *really* a corner case and should affect almost no one. `--stage {0,2}`
work fine with download-rustc.
Fixes https://github.com/rust-lang/rust/issues/80096.
There were two fixes needed:
1. Use `top_stage` instead of `top_stage - 1`. There was a long and torturous comment about trying to match rustdoc's version, but it works better without the hard-coding than with.
2. Make sure that `ci-llvm/lib` is added to LD_LIBRARY_PATH. Previously the error index would be unable to load LLVM for stage0 builds.
At some point we should probably have a discussion about how rustdoc stages should be numbered;
confusion between 0/1/2 has come up several times in bootstrap now.
Note that this is still broken when using `download-rustc = true` and `--stage 1`,
but that's *really* a corner case and should affect almost no one. `--stage {0,2}`
work fine with download-rustc.
This was so verbose before that it made it hard to see what effect the flag actually had.
Before:
```
Set({test::src/tools/tidy}) not skipped for "bootstrap::test::Tidy" -- not in [src/test/ui, src/test/mir-opt/, src/test/debuginfo, src/test/ui-fulldeps]
Skipping Suite(test::src/test/ui) because it is excluded
Suite(test::src/test/run-pass-valgrind) not skipped for "bootstrap::test::RunPassValgrind" -- not in [src/test/ui, src/test/mir-opt/, src/test/debuginfo, src/test/ui-fulldeps]
Skipping Suite(test::src/test/mir-opt) because it is excluded
Suite(test::src/test/codegen) not skipped for "bootstrap::test::Codegen" -- not in [src/test/ui, src/test/mir-opt/, src/test/debuginfo, src/test/ui-fulldeps]
Suite(test::src/test/codegen-units) not skipped for "bootstrap::test::CodegenUnits" -- not in [src/test/ui, src/test/mir-opt/, src/test/debuginfo, src/test/ui-fulldeps]
Suite(test::src/test/assembly) not skipped for "bootstrap::test::Assembly" -- not in [src/test/ui, src/test/mir-opt/, src/test/debuginfo, src/test/ui-fulldeps]
Suite(test::src/test/incremental) not skipped for "bootstrap::test::Incremental" -- not in [src/test/ui, src/test/mir-opt/, src/test/debuginfo, src/test/ui-fulldeps]
Skipping Suite(test::src/test/debuginfo) because it is excluded
Skipping Suite(test::src/test/ui-fulldeps) because it is excluded
... about 100 more lines ...
```
After:
```
Skipping Suite(test::src/test/ui) because it is excluded
Skipping Suite(test::src/test/mir-opt) because it is excluded
Skipping Suite(test::src/test/debuginfo) because it is excluded
Skipping Suite(test::src/test/ui-fulldeps) because it is excluded
```
Enable conditional checking of values in the Rust codebase
This pull-request enable conditional checking of (well known) values in the Rust codebase.
Well known values were added in https://github.com/rust-lang/rust/pull/94362. All the `target_*` values are taken from all the built-in targets which is why some extra values were needed do be added as they are not (yet ?) defined in any built-in targets.
r? `@Mark-Simulacrum`
this also fixes a bug where bootstrap would try to use the fake `rustc` binary built by bootstrap -
cargo puts it in a different directory when using `cargo run` instead of x.py
The majority of the code is only used by either rustbuild or
rustc_llvm's build script. Rust_build is compiled once for rustbuild and
once for every stage. This means that the majority of the code in this
crate is needlessly compiled multiple times. By moving only the code
actually used by the respective crates to rustbuild and rustc_llvm's
build script, this needless duplicate compilation is avoided.
First, this reverts the `CFLAGS`/`CXXFLAGS` of #93918. Those flags are
already read by `cc` and populated into `Build` earlier on in the
process. We shouldn't be overriding that based on `CFLAGS`, since `cc`
also respects overrides like `CFLAGS_{TARGET}` and `HOST_CFLAGS`, which
we want to take into account.
Second, this adds the same capability to specify target-specific
versions of `LDFLAGS` as we have through `cc` for the `C*` flags:
https://github.com/alexcrichton/cc-rs#external-configuration-via-environment-variables
Note that this also necessitated an update to compiletest to treat
CXXFLAGS separately from CFLAGS.
build: dist: defer PlainSourceTarball
Apparently it changes some tool sources and invalidates their fingerprints, forcing us to build them several times (before and after vendoring sources).
I have not dug into why vendoring actually invalidates the figreprints, but moving the vendoring lower in the pipeline seems to avoid the issue.
I could imagine that we somehow write a .cargo/config somewhere which somehow makes subsequent builds use the vendored deps but I was not able to find anything.
I checked the sizes of generated archives pre and post patch and their are the same, so I hope there is no functional change.
Fixes#93033
x.py has support for excluding some steps from the invocation, but
unfortunately that's not granular enough: some steps have the same name
in different modules, and that prevents excluding only *some* of them.
As a practical example, let's say you need to run everything in `./x.py
test` except for the standard library tests, as those tests require IPv6
and need to be executed on a separate machine. Before this commit, if
you were to just run this:
./x.py test --exclude library/std
...the execution would fail, as that would not only exclude running the
tests for the standard library, it would also exclude generating its
documentation (breaking linkchecker).
This commit adds support for an optional module annotation in --exclude
paths, allowing the user to choose which module to exclude from:
./x.py test --exclude test::library/std
This maintains backward compatibility, but also allows for more ganular
exclusion. More examples on how this works:
| `--exclude` | Docs | Tests |
| ------------------- | ------- | ------- |
| `library/std` | Skipped | Skipped |
| `doc::library/std` | Skipped | Run |
| `test::library/std` | Run | Skipped |
Note that the new behavior only works in the `--exclude` flag, and not
in other x.py arguments or flags yet.
Apparently it changes some tool sources and invalidates their fingerprints, forcing us to build them several times (before and after vendoring sources).
I have not dug into why vendoring actually invalidates the figreprints, but the moving the vendoring lower in the pipeline seems to avoid the issue.
I could imagine that we somehow write a .cargo/config somewhere which somehow makes subsequent builds use the vendored deps but I was not able to find anything.
I checked the sizes of generated archives pre and post patch and their are the same, so I hope there is not functional change.
Fixes#93033
DF_ORIGIN flag signifies that the object being loaded may make reference to the $ORIGIN substitution string.
Some implementations are just ignoring DF_ORIGIN and do substitution for $ORIGIN if present (whatever DF_ORIGIN pr
Set the flag inconditionally if rpath is wanted.
Currently the verbosity settings are:
- 2: RUSTC-SHIM envvars get spammed on every invocation, O(30) lines
cargo is passed -v which outputs CLI invocations, O(5) lines
- 3: cargo is passed -vv which outputs build script output, O(0-10) lines
This commit changes it to:
- 1: cargo is passed -v, O(5) lines
- 2: cargo is passed -vv, O(10) lines
- 3: RUSTC-SHIM envvars get spammed, O(30) lines
Assemble the compiler when running `x.py build`
Previously, there was no way to actually get binaries in
`build/$TARGET/stage1/bin` without building the standard library. This
makes it possible to build just the compiler. This can be useful when
the standard library isn't actually necessary for trying out your tests
(e.g. a bug that can be reproduced with only a `no_core` crate).
Closes https://github.com/rust-lang/rust/issues/73519.
