Add cg_clif as optional codegen backend
Rustc_codegen_cranelift is an alternative codegen backend for rustc based on Cranelift. It has the potential to improve compilation times in debug mode. In my experience the compile time improvements over debug mode LLVM for a clean build are about 20-30% in most cases.
This PR adds cg_clif as optional codegen backend. By default it is only enabled for `./x.py check`. It can be enabled for `./x.py build` too by adding `cranelift` to the `rust.codegen-backends` array in `config.toml`.
MCP: https://github.com/rust-lang/compiler-team/issues/270
r? `@Mark-Simulacrum`
Sync LLVM submodule if it has been initialized
Since having enabled the download-ci-llvm option,
and having rebased on top of #76864,
I've noticed that I had to update the llvm-project
submodule manually if it was checked out.
Orignally, the submodule update logic was
introduced to reduce the friction for contributors
to manage the submodules, or in other words, to prevent
getting PRs that have unwanted submodule rollbacks
because the contributors didn't run git submodule update.
This commit adds logic to ensure there is no inadvertent
LLVM submodule rollback in a PR if download-ci-llvm
(or llvm-config) is enabled. It will detect whether the
llvm-project submodule is initialized, and if so, update
it in any case. If it is not initialized, behaviour is
kept to not do any update/initialization.
An alternative to the chosen implementation would
be to not pass the --init command line arg to
`git submodule update` for the src/llvm-project
submodule. This would show a confusing error message
however on all builds with an uninitialized repo.
We could pass the --silent param, but we still want
it to print something if it is initialized and has
to update something.
So we just do a manual check for whether the
submodule is initialized.
Since having enabled the download-ci-llvm option,
and having rebased on top of f05b47ccdf,
I've noticed that I had to update the llvm-project
submodule manually if it was checked out.
Orignally, the submodule update logic was
introduced to reduce the friction for contributors
to manage the submodules, or in other words, to prevent
getting PRs that have unwanted submodule rollbacks
because the contributors didn't run git submodule update.
This commit adds logic to ensure there is no inadvertent
LLVM submodule rollback in a PR if download-ci-llvm
(or llvm-config) is enabled. It will detect whether the
llvm-project submodule is initialized, and if so, update
it in any case. If it is not initialized, behaviour is
kept to not do any update/initialization.
An alternative to the chosen implementation would
be to not pass the --init command line arg to
`git submodule update` for the src/llvm-project
submodule. This would show a confusing error message
however on all builds with an uninitialized repo.
We could pass the --silent param, but we still want
it to print something if it is initialized and has
to update something.
So we just do a manual check for whether the
submodule is initialized.
Don't download/sync llvm-project submodule if download-ci-llvm is set
llvm-project takes > 1GB storage space and a long time to download.
It's better to not download it unless needed.
This requires that bootstrap is run from the same worktree as the sources it'll
build, but this is basically required for the build to work anyway. You can
still run it from a different directory, just that the files it builds must be
beside it.
This moves build triple discovery for rustbuild from bootstrap.py into a build
script, meaning it will "just work" if building rustbuild via Cargo rather than
Python.
rustbuild: Do not use `rust-mingw` component when bootstrapping windows-gnu targets
Addresses https://github.com/rust-lang/rust/pull/76326#issuecomment-687273473 (ancient `x86_64-w64-mingw32-gcc` is selected as a linker wrapper, which is not usable in `use_lld=true` mode).
Perhaps the comment about incompatible mingw was true in the past, but many things changed since then.
With this change I was able to build everything successfully locally using a newer mingw toolchain, if it passes through the older toolchain on CI, then it should be good, I think.
This is generally a good idea, and will help with being able to build bootstrap
without Python over time as it means we can "just" build with cargo +beta build
rather than needing the user to set environment variables. This is a minor step,
but a necessary one on that road.
The current plan is that submodule tracks the `release` branch of
rust-analyzer, which is updated once a week.
rust-analyzer is a workspace (with a virtual manifest), the actual
binary is provide by `crates/rust-analyzer` package.
Note that we intentionally don't add rust-analyzer to `Kind::Test`,
for two reasons.
*First*, at the moment rust-analyzer's test suite does a couple of
things which might not work in the context of rust repository. For
example, it shells out directly to `rustup` and `rustfmt`. So, making
this work requires non-trivial efforts.
*Second*, it seems unlikely that running tests in rust-lang/rust repo
would provide any additional guarantees. rust-analyzer builds with
stable and does not depend on the specifics of the compiler, so
changes to compiler can't break ra, unless they break stability
guarantee. Additionally, rust-analyzer itself is gated on bors, so we
are pretty confident that test suite passes.
This commit fixes a regression introduced in #73317 where an oversight
meant that `config.toml` was assumed to exist.
Signed-off-by: David Wood <david@davidtw.co>
This commit modifies bootstrap so that `config.toml` is read first from
`RUST_BOOTSTRAP_CONFIG`, then `--config` and finally `config.toml` in the
current directory.
This is a subjective change, intended to improve the ergnomics when
using "development shells" for rustc development (for example, using tools
such as Nix) which set environment variables to ensure a reproducible
environment (these development shells can then be version controlled). By
optionally reading `config.toml` from an environment variable, a `config.toml`
can be defined in the development shell and a path to it exposed in the
`RUST_BOOTSTRAP_CONFIG` environment variable - avoiding the need to manually
symlink the contents of this file to `config.toml` in the working
directory.
Signed-off-by: David Wood <david@davidtw.co>
This allows configuring the directory for build artifacts, instead of having it always be ./build. This means you can set it to a constant location, letting you reuse the same cache while working in several different directories.
The configuration lives in config.toml under build.build-dir. By default, it keeps the existing default of ./build, but it can be configured to any relative or absolute path. Additionally, it allows making outputs relative to the root of the git repository using $ROOT.
This also abstracts checking for a command into `require`.
Before:
```
Updating only changed submodules
Submodules updated in 0.01 seconds
Traceback (most recent call last):
File "./x.py", line 11, in <module>
bootstrap.main()
...
File "/home/joshua/src/rust/src/bootstrap/bootstrap.py", line 137, in run
ret = subprocess.Popen(args, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
```
After:
```
error: unable to run `curl --version`: [Errno 2] No such file or directory
Please make sure it's installed and in the path.
```
With #65251 landed there's no need to build two LLVM backends and ship
them with rustc, every target we have now uses the same LLVM backend!
This removes the `src/llvm-emscripten` submodule and additionally
removes all support from rustbuild for building the emscripten LLVM
backend. Multiple codegen backend support is left in place for now, and
this is intended to be an easy 10-15 minute win on CI times by avoiding
having to build LLVM twice.
Ensure edition lints and internal lints are enabled with deny-warnings=false
Previously we only passed the deny command line flags if deny-warnings was enabled, but now we either pass -W... or -D... for each of the flags as appropriate.
This is also a breaking change to x.py as it changes `--warnings=allow` to `--warnings=warn` which is what that flag actually did; we don't have an allow warnings mode.
When deny-warnings is not specified or set to true, the behaviour is the same as before.
When deny-warnings is set to false, warnings are now allowed
Fixes#63911
Signed-off-by: Marc-Antoine Perennou <Marc-Antoine@Perennou.com>
The new git submodule src/llvm-project is a monorepo replacing src/llvm
and src/tools/{clang,lld,lldb}. This also serves as a rebase for these
projects to the new 8.x branch from trunk.
The src/llvm-emscripten fork is unchanged for now.
./x.py used to automatically check out the right commit when a submodule was outdated and ./x.py build was run
and submodules handling was enabled in config.toml (submodules = true).
But it threw an error:
[...]
failed to run: git submodule -q sync --progress src/tools/clippy
The commit removes the --progress from git submodule call.
Fixes#57080
This commit removes all jemalloc related submodules, configuration, etc,
from the bootstrap, from the standard library, and from the compiler.
This will be followed up with a change to use jemalloc specifically as
part of rustc on blessed platforms.
This optionally adds lldb (and clang, which it needs) to the build.
Because rust uses LLVM 7, and because clang 7 is not yet released, a
recent git master version of clang is used.
The lldb that is used includes the Rust plugin.
lldb is only built when asked for, or when doing a nightly build on
macOS. Only macOS is done for now due to difficulties with the Python
dependency.
If you are using a hard-linked file as your config.toml, this change will affect the way other instances of the file is modified.
The original version would modify all other instances whereas the new version will leave others unchanged, reducing the ref count by one.
Signed-off-by: NODA, Kai <nodakai@gmail.com>
This ensures that the working directory of rustbuild has no effect on
it's run; since tests will run with a different cwd this is required for
consistent behavior.
This option was introduced in 72cb109bec, but it uses two different
spellings (fast-submodule vs fast-submodules) and isn't handled by
Rust bootstrap which means that any attempt to set this flag fails.
Faster submodule updating
For the common case when there are no submodules which need updating, this takes 0.48 seconds instead of 47 seconds.
r? @alexcrichton
- The bootstrap crate currently passes -v to Cargo if itself invoked
with -vv. But Cargo supports -vv (to show build script output), so make
bootstrap pass that if itself invoked with -vvv. (More specifically,
pass N '-v's to Cargo if invoked with N+1 of them.)
- bootstrap.py currently tries to pass on up to two '-v's to cargo when
building bootstrap, but incorrectly ('-v' is marked as 'store_true', so
argparse stores either False or True, ignoring multiple '-v's). Fix
this, allow passing any number of '-v's, and make it consistent with
bootstrap's invocation of Cargo (i.e. subtract one from the number of
'-v's).
- Also improve bootstrap.py's config.toml 'parsing' to support arbitrary
verbosity levels, + allow command line to override it.
This commit imports the LLD project from LLVM to serve as the default linker for
the `wasm32-unknown-unknown` target. The `binaryen` submoule is consequently
removed along with "binaryen linker" support in rustc.
Moving to LLD brings with it a number of benefits for wasm code:
* LLD is itself an actual linker, so there's no need to compile all wasm code
with LTO any more. As a result builds should be *much* speedier as LTO is no
longer forcibly enabled for all builds of the wasm target.
* LLD is quickly becoming an "official solution" for linking wasm code together.
This, I believe at least, is intended to be the main supported linker for
native code and wasm moving forward. Picking up support early on should help
ensure that we can help LLD identify bugs and otherwise prove that it works
great for all our use cases!
* Improvements to the wasm toolchain are currently primarily focused around LLVM
and LLD (from what I can tell at least), so it's in general much better to be
on this bandwagon for bugfixes and new features.
* Historical "hacks" like `wasm-gc` will soon no longer be necessary, LLD
will [natively implement][gc] `--gc-sections` (better than `wasm-gc`!) which
means a postprocessor is no longer needed to show off Rust's "small wasm
binary size".
LLD is added in a pretty standard way to rustc right now. A new rustbuild target
was defined for building LLD, and this is executed when a compiler's sysroot is
being assembled. LLD is compiled against the LLVM that we've got in tree, which
means we're currently on the `release_60` branch, but this may get upgraded in
the near future!
LLD is placed into rustc's sysroot in a `bin` directory. This is similar to
where `gcc.exe` can be found on Windows. This directory is automatically added
to `PATH` whenever rustc executes the linker, allowing us to define a `WasmLd`
linker which implements the interface that `wasm-ld`, LLD's frontend, expects.
Like Emscripten the LLD target is currently only enabled for Tier 1 platforms,
notably OSX/Windows/Linux, and will need to be installed manually for compiling
to wasm on other platforms. LLD is by default turned off in rustbuild, and
requires a `config.toml` option to be enabled to turn it on.
Finally the unstable `#![wasm_import_memory]` attribute was also removed as LLD
has a native option for controlling this.
[gc]: https://reviews.llvm.org/D42511
Restore the download of rust-mingw
The build might otherwise break due to mixing MinGW object files from rust-std and the local MinGW which might be newer/older than the version used to build rust-std.
Fixes#48272
r? @alexcrichton
Improve --help performance for x.py
Since compiling the bootstrap command doesn't require any submodules,
we can skip updating submodules when a --help command is passed in.
On my machine, this saves 1 minute if the submodules are already
downloaded, and 10 minutes if run on a clean repo.
This commit also adds a message before compiling/downloading anything
when a --help command is passed in, to tell the user WHY --help
takes so long to complete. It also points the user to the bootstrap
README.md for faster help.
Finally, this fixes one warning message that still referenced using
make instead of x.py, even though x.py is now the standard way of
building rust.
Closes#37305
It was an existing solution to tell the user why a --help command
takes a long time to process. However, it would only print if the
stage0 rust compiler needed to be downloaded, it came after
update_submodules (which took a long time), and it was immediately
followed by download messages and loading bars, meaning users could
easily gloss over the message.
This commit also moves the help message out of main(), and instead
puts it at the top of bootstrap(). main() is intended to be minimal,
only handling error messages.
The build might otherwise break due to mixing MinGW object files from
rust-std and the local MinGW which might be newer/older than the version
used to build rust-std.
Fixes#48272
Since compiling the bootstrap command doesn't require any submodules,
we can skip updating submodules when a --help command is passed in.
On my machine, this saves 1 minute if the submodules are already
downloaded, and 10 minutes if run on a clean repo.
This commit also adds a message before compiling/downloading anything
when a --help command is passed in, to tell the user WHY --help
takes so long to complete. It also points the user to the bootstrap
README.md for faster help.
Finally, this fixes one warning message that still referenced using
make instead of x.py, even though x.py is now the standard way of
building rust.
This commit introduces a separately compiled backend for Emscripten, avoiding
compiling the `JSBackend` target in the main LLVM codegen backend. This builds
on the foundation provided by #47671 to create a new codegen backend dedicated
solely to Emscripten, removing the `JSBackend` of the main codegen backend in
the process.
A new field was added to each target for this commit which specifies the backend
to use for translation, the default being `llvm` which is the main backend that
we use. The Emscripten targets specify an `emscripten` backend instead of the
main `llvm` one.
There's a whole bunch of consequences of this change, but I'll try to enumerate
them here:
* A *second* LLVM submodule was added in this commit. The main LLVM submodule
will soon start to drift from the Emscripten submodule, but currently they're
both at the same revision.
* Logic was added to rustbuild to *not* build the Emscripten backend by default.
This is gated behind a `--enable-emscripten` flag to the configure script. By
default users should neither check out the emscripten submodule nor compile
it.
* The `init_repo.sh` script was updated to fetch the Emscripten submodule from
GitHub the same way we do the main LLVM submodule (a tarball fetch).
* The Emscripten backend, turned off by default, is still turned on for a number
of targets on CI. We'll only be shipping an Emscripten backend with Tier 1
platforms, though. All cross-compiled platforms will not be receiving an
Emscripten backend yet.
This commit means that when you download the `rustc` package in Rustup for Tier
1 platforms you'll be receiving two trans backends, one for Emscripten and one
that's the general LLVM backend. If you never compile for Emscripten you'll
never use the Emscripten backend, so we may update this one day to only download
the Emscripten backend when you add the Emscripten target. For now though it's
just an extra 10MB gzip'd.
Closes#46819
This currently only supports a limited subset of the full compilation,
but is likely 90% of what people will want and is possible without
building a full compiler (i.e., running LLVM). In theory, this means
that contributors who don't want to build LLVM now have an easy way to
compile locally, though running tests won't work.
If config.toml doesn't exist, then an IOError will be raised
on the `with open(...)` line. Prior to e788fa7, this was
caught because the `except` clause didn't specify what
exceptions it caught, so both IOError and OSError were
caught
This commit rewrites our ancient `./configure` script from shell into Python.
The impetus for this change is to remove `config.mk` which is just a vestige of
the old makefile build system at this point. Instead all configuration is now
solely done through `config.toml`.
The python script allows us to more flexibly program (aka we can use loops
easily) and create a `config.toml` which is based off `config.toml.example`.
This way we can preserve comments and munge various values as we see fit.
It is intended that the configure script here is a drop-in replacement for the
previous configure script, no functional change is intended. Also note that the
rationale for this is also because our build system requires Python, so having a
python script a bit earlier shouldn't cause too many problems.
Closes#40730
