This commit adds support to rustbuild for compiling Cargo as part of the release
process. Previously rustbuild would simply download a Cargo snapshot and
repackage it. With this change we should be able to turn off artifacts from the
rust-lang/cargo repository and purely rely on the artifacts Cargo produces here.
The infrastructure added here is intended to be extensible to other components,
such as the RLS. It won't exactly be a one-line addition, but the addition of
Cargo didn't require too much hooplah anyway.
The process for release Cargo will now look like:
* The rust-lang/rust repository has a Cargo submodule which is used to build a
Cargo to pair with the rust-lang/rust release
* Periodically we'll update the cargo submodule as necessary on rust-lang/rust's
master branch
* When branching beta we'll create a new branch of Cargo (as we do today), and
the first commit to the beta branch will be to update the Cargo submodule to
this exact revision.
* When branching stable, we'll ensure that the Cargo submodule is updated and
then make a stable release.
Backports to Cargo will look like:
* Send a PR to cargo's master branch
* Send a PR to cargo's release branch (e.g. rust-1.16.0)
* Send a PR to rust-lang/rust's beta branch updating the submodule
* Eventually send a PR to rust-lang/rust's master branch updating the submodule
For reference, the process to add a new component to the rust-lang/rust release
would look like:
* Add `$foo` as a submodule in `src/tools`
* Add a `tool-$foo` step which compiles `$foo` with the specified compiler,
likely mirroring what Cargo does.
* Add a `dist-$foo` step which uses `src/tools/$foo` and the `tool-$foo` output
to create a rust-installer package for `$foo` likely mirroring what Cargo
does.
* Update the `dist-extended` step with a new dependency on `dist-$foo`
* Update `src/tools/build-manifest` for the new component.
Currently we create a source tarball on almost all of the `DEPLOY=1` builders
but this has the adverse side effect of all source tarballs overriding
themselves in the S3 bucket. Normally this is ok but unfortunately a source
tarball created on Windows is not buildable on Unix.
On Windows the vendored sources contain paths with `\` characters in them which
when interpreted on Unix end up in "file not found" errors.
Instead of this overwriting behavior, whitelist just one linux builder for
producing tarballs and avoid producing tarballs on all other hosts.
Automate vendoring by invoking cargo-vendor when building src dist tarballs.
This avoids #39633 bringing the `src/vendor` checked into git by #37524, past 200,000 lines of code.
I believe the strategy of having rustbuild run `cargo vendor` during the `dist src` step is sound.
However, the only way to be sure `cargo-vendor` exists is to run `cargo install --force cargo-vendor`, which will recompile it every time (not passing `--force` means you can't tell between "already exists" and "build error"). ~~This is quite suboptimal and I'd like to somehow do it in each `Dockerfile` that would need it.~~
* [ ] Cache `CARGO_HOME` (i.e. `~/.cargo`) between CI runs
* `bin/cargo-vendor` and the actual caches are the relevant bits
* [x] Do not build `cargo-vendor` all the time
* ~~Maybe detect `~/.cargo/bin/cargo-vendor` already exists?~~
* ~~Could also try to build it in a `Dockerfile` but do we have `cargo`/`rustc` there?~~
* Final solution: check `cargo install --list` for a line starting with `cargo-vendor `
cc @rust-lang/tools
Delete the makefile build system
This PR deletes the makefile build system in favor of the rustbuild build system. The beta has now been branched so 1.16 will continue to be buildable from the makefiles, but going forward 1.17 will only be buildable with rustbuild.
Rustbuild has been the default build system [since 1.15.0](https://github.com/rust-lang/rust/pull/37817) and the makefiles were [proposed for deletion](https://internals.rust-lang.org/t/proposal-for-promoting-rustbuild-to-official-status/4368) at this time back in November of last year.
And now with the deletion of these makefiles we can start getting those sweet sweet improvements of using crates.io crates in the compiler!
Add support for test suites emulated in QEMU
This commit adds support to the build system to execute test suites that cannot
run natively but can instead run inside of a QEMU emulator. A proof-of-concept
builder was added for the `arm-unknown-linux-gnueabihf` target to show off how
this might work.
In general the architecture is to have a server running inside of the emulator
which a local client connects to. The protocol between the server/client
supports compiling tests on the host and running them on the target inside the
emulator.
Closes#33114
This commit adds support to the build system to execute test suites that cannot
run natively but can instead run inside of a QEMU emulator. A proof-of-concept
builder was added for the `arm-unknown-linux-gnueabihf` target to show off how
this might work.
In general the architecture is to have a server running inside of the emulator
which a local client connects to. The protocol between the server/client
supports compiling tests on the host and running them on the target inside the
emulator.
Closes#33114
This commit adds a new flag to the configure script,
`--enable-extended`, which is intended for specifying a desire to
compile the full suite of Rust tools such as Cargo, the RLS, etc. This
is also an indication that the build system should create combined
installers such as the pkg/exe/msi artifacts.
Currently the `--enable-extended` flag just indicates that combined
installers should be built, and Cargo is itself not compiled just yet
but rather only downloaded from its location. The intention here is to
quickly get to feature parity with the current release process and then
we can start improving it afterwards.
All new files in this PR inside `src/etc/installer` are copied from the
rust-packaging repository.
This commit passes the `--release-channel=nightly` flag to all images which have
the `DEPLOY` flag set. This means that we'll name artifacts and the compiler
appropriately.
This reworks a bit how arguments are passed, but for now doesn't change what's
already being passed. Eventually we'll want to avoid enabling debug assertions
and llvm assertions for *all* releases, but I figure we can tackle that a little
bit more down the road.
In #37280 we enabled line number debugging information in release artifacts,
primarily to close out #36452 where debugging information was critical for MSVC
builds of Rust to be useful in production. This commit, however, apparently had
some unfortunate side effects.
Namely it was noticed in #37477 that if `RUST_BACKTRACE=1` was set then any
compiler error would take a very long time for the compiler to exit. The cause
of the problem here was somewhat deep:
* For all compiler errors, the compiler will `panic!` with a known value. This
tears down the main compiler thread and allows cleaning up all the various
resources. By default, however, this panic output is suppressed for "normal"
compiler errors.
* When `RUST_BACKTRACE=1` was set this caused every compiler error to generate a
backtrace.
* The libbacktrace library hits a pathological case where it spends a very long
time in its custom allocation function, `backtrace_alloc`, because the
compiler has so much debugging information. More information about this can be
found in #29293 with a summary at the end of #37477.
To solve this problem this commit simply removes debuginfo from the compiler but
not from the standard library. This should allow us to keep #36452 closed while
also closing #37477. I've measured the difference to be orders of magnitude
faster than it was before, so we should see a much quicker time-to-exit after a
compile error when `RUST_BACKTRACE=1` is set.
Closes#37477Closes#37571
This commit switches the rustbuild build system to compiling the
compiler twice for a normal bootstrap rather than the historical three
times.
Rust is a bootstrapped language which means that a previous version of
the compiler is used to build the next version of the compiler. Over
time, however, we change many parts of compiler artifacts such as the
metadata format, symbol names, etc. These changes make artifacts from
one compiler incompatible from another compiler. Consequently if a
compiler wants to be able to use some artifacts then it itself must have
compiled the artifacts.
Historically the rustc build system has achieved this by compiling the
compiler three times:
* An older compiler (stage0) is downloaded to kick off the chain.
* This compiler now compiles a new compiler (stage1)
* The stage1 compiler then compiles another compiler (stage2)
* Finally, the stage2 compiler needs libraries to link against, so it
compiles all the libraries again.
This entire process amounts in compiling the compiler three times.
Additionally, this process always guarantees that the Rust source tree
can compile itself because the stage2 compiler (created by a freshly
created compiler) would successfully compile itself again. This
property, ensuring Rust can compile itself, is quite important!
In general, though, this third compilation is not required for general
purpose development on the compiler. The third compiler (stage2) can
reuse the libraries that were created during the second compile. In
other words, the second compilation can produce both a compiler and the
libraries that compiler will use. These artifacts *must* be compatible
due to the way plugins work today anyway, and they were created by the
same source code so they *should* be compatible as well.
So given all that, this commit switches the default build process to
only compile the compiler three times, avoiding this third compilation
by copying artifacts from the previous one. Along the way a new entry in
the Travis matrix was also added to ensure that our full bootstrap can
succeed. This entry does not run tests, though, as it should not be
necessary.
To restore the old behavior of a full bootstrap (three compiles) you can
either pass:
./configure --enable-full-bootstrap
or if you're using config.toml:
[build]
full-bootstrap = true
Overall this will hopefully be an easy 33% win in build times of the
compiler. If we do 33% less work we should be 33% faster! This in turn
should affect cycle times and such on Travis and AppVeyor positively as
well as making it easier to work on the compiler itself.
This commit adds support for sccache, a ccache-like compiler which works on MSVC
and stores results into an S3 bucket. This also switches over all Travis and
AppVeyor automation to using sccache to ensure a shared and unified cache over
time which can be shared across builders.
The support for sccache manifests as a new `--enable-sccache` option which
instructs us to configure LLVM differently to use a 'sccache' binary instead of
a 'ccache' binary. All docker images for Travis builds are updated to download
Mozilla's tooltool builds of sccache onto various containers and systems.
Additionally a new `rust-lang-ci-sccache` bucket is configured to hold all of
our ccache goodies.
This commit switches the default build system for Rust from the makefiles to
rustbuild. The rustbuild build system has been in development for almost a year
now and has become quite mature over time. This commit is an implementation of
the proposal on [internals] which slates deletion of the makefiles on
2016-01-02.
[internals]: https://internals.rust-lang.org/t/proposal-for-promoting-rustbuild-to-official-status/4368
This commit also updates various documentation in `README.md`,
`CONTRIBUTING.md`, `src/bootstrap/README.md`, and throughout the source code of
rustbuild itself.
Closes#37858
rustbuild: allow dynamically linking LLVM
The makefiles and `mklldeps.py` called `llvm-config --shared-mode` to
find out if LLVM defaulted to shared or static libraries, and just went
with that. But under rustbuild, `librustc_llvm/build.rs` was assuming
that LLVM should be static, and even forcing `--link-static` for 3.9+.
Now that build script also uses `--shared-mode` to learn the default,
which should work better for pre-3.9 configured for dynamic linking, as
it wasn't possible back then to choose differently via `llvm-config`.
Further, the configure script now has a new `--enable-llvm-link-shared`
option, which allows one to manually override `--link-shared` on 3.9+
instead of forcing static.
Update: There are now four static/shared scenarios that can happen
for the supported LLVM versions:
- 3.9+: By default use `llvm-config --link-static`
- 3.9+ and `--enable-llvm-link-shared`: Use `--link-shared` instead.
- 3.8: Use `llvm-config --shared-mode` and go with its answer.
- 3.7: Just assume static, maintaining the status quo.
The makefiles and `mklldeps.py` called `llvm-config --shared-mode` to
find out if LLVM defaulted to shared or static libraries, and just went
with that. But under rustbuild, `librustc_llvm/build.rs` was assuming
that LLVM should be static, and even forcing `--link-static` for 3.9+.
Now that build script also uses `--shared-mode` to learn the default,
which should work better for pre-3.9 configured for dynamic linking, as
it wasn't possible back then to choose differently via `llvm-config`.
Further, the configure script now has a new `--enable-llvm-link-shared`
option, which allows one to manually override `--link-shared` on 3.9+
instead of forcing static.
Fix grammar verification
* Use make check-lexer to verify the grammar.
* Extend grammar/README
* Add make clean-grammar rule
* Add target check-build-lexer-verifier to make tidy, so it will build the verifier with every build and catch future errors
This is the continuation of #34994
r? @steveklabnik @jonathandturner @alexcrichton
* Use `make check-lexer` to verify the grammar.
* Extend grammar/README
* Add make clean-grammar rule
* Add target `check-build-lexer-verifier` to `make tidy`, so it will build the verifier with every build and catch future errors
* Search for antlr4 with configure and find
Add llvm debuginfo configure option
CC @nnethercote @Mark-Simulacrum
We add a new configure option, `--enable-llvm-debuginfo`, to do exactly what you'd think.
Re: #31033Fixes#37738
A few changes are included here:
* The `winapi` and `url` dependencies were dropped. The source code for these
projects is pretty weighty, and we're about to vendor them, so let's not
commit to that intake just yet. If necessary we can vendor them later but for
now it shouldn't be necessary.
* The `--frozen` flag is now always passed to Cargo, obviating the need for
tidy's `cargo_lock` check.
* Tidy was updated to not check the vendor directory
Closes#34687
armv7l is armv7 architecture and CFG_CPUTYPE should be armv7 in order to end up
with armv7-unknown-linux-gnueabihf.mk rather than
arm-unknown-linux-gnueabihf.mk
This commit enables by default passing the `-C debuginfo=1` argument to the
compiler for the stable, beta, and nightly release channels. A new configure
option was also added, `--enable-debuginfo-lines`, to enable this behavior in
developer builds as well.
Closes#36452
configure: Add options for separate musl roots
This allows using the `./configure` script to enable rustbuild to compile
multiple musl targets at once. We'll hopefully use this soon on our bots to
produce a bunch of targets.
