This was recommended by GitHub Support to try reducing the things that
could've caused #78743. I checked the changelog and there should be no
practical impact for us (we already set an explicit fetch-depth).
While technically the i686-unknown-freebsd target has been a tier 2
development platform for a long time, with full toolchain tarballs
available on static.rust-lang.org, due to a bug in the manifest
generation the target was never available for download through rustup.
The infrastructure team privately inquired the FreeBSD package
maintainers, and they weren't relying on those tarballs either, so it's
a fair assumption to say practically nobody is using those tarballs.
This PR then removes the CI builder that produces full tarballs for the
target, and moves the compilation of rust-std for the target in
dist-various-2.
The x86_64-unknown-freebsd target is *not* affected.
The former `ubuntu:19.10` reached EOL in July, 2020, whereas
`ubuntu:20.04` is an LTS release supported until 2025.
These are non-dist CI images, so the impact should be low.
Dist build manifest
This PR makes two changes that should remove a significant chunk of the time spent in our release process: cloning the `rust-lang/rust` monorepo, all its submodules, and building `bootstrap` to then invoke `build-manifest`:
* `build-manifest` doesn't rely on a clone of the monorepo being present anymore. The only remaining bit of information it fetched from it (the Rust version) is instead bundled in the binary.
* A new "component" is added, `build-manifest`. That component includes a prebuilt version of the tool, and it's *not* included in the Rustup manifest. This will allow `promote-release` to directly invoke the tool without interacting with our build system.
* The Linux x86_64 CI is changed to also build the component mentioned above. It's the only CI builder tasked to do so, and to cleanly support this a new `--include-default-paths` flag was added to `./x.py`.
* The `BUILD_MANIFEST_NUM_THREADS` environment variable is added to configure the number of threads at runtime.
This PR is best reviewed commit-by-commit.
r? `@Mark-Simulacrum`
ci: Fix riscv64gc linux test QEMU fault, plus doc link fix
Newer versions of the `qemu` package (used for riscv64gc-unknown-linux-gnu testing) don't work with the version of the RISC-V bootloader we were using. a4a0342cf5 bumps to a revision which should fix the problem.
e0b033e965 fixes a documentation failure I encountered while running the tests.
Promote aarch64-pc-windows-msvc to Tier 2 Development Platform
Adds a GitHub Actions CI build for `aarch64-pc-windows-msvc` via cross-compilation on an x86_64 host.
This promotes `aarch64-pc-windows-msvc` from a Tier 2 Compilation Target (std) to a Tier 2 Development Platform (std+rustc+cargo+tools).
Fixes#72881
r? `@pietroalbini`
This bumps the version of the bbl bootloader not to perform 64-bit
accesses to the PLIC. Doing so resulted in the QEMU test machine to fail
to boot:
bbl loader
../machine/mtrap.c:21: machine mode: unhandlable trap 7 @ 0x0000000080001f6e
Power off
Signed-off-by: Tom Eccles <tom.eccles@codethink.co.uk>
Stop running macOS builds on Azure Pipelines
The Infrastructure Team agreed to migrate macOS builds to GitHub Actions, so this commit stops running those builders on Azure Pipelines. The GitHub Actions runners are already configured to upload to the production bucket.
We can't still fully remove the Azure Pipelines configuration, as we still need to have that available until no stable releases run any of their builds on Azure Pipelines anymore. I'll open an issue to track fully removing our Azure Pipelines setup once the PR is merged.
r? @Mark-Simulacrum
Ensure that all LLVM components requested by tests are available on CI
Addresses https://github.com/rust-lang/rust/pull/75064#issuecomment-667722652
I used an environment variable because passing a command line option all the way from CI to compiletest would be just too much hassle for this task.
I added a new variable, but any of the already existing ones defined by CI could be used instead.
r? @Mark-Simulacrum
Previously we would have some platforms where LLVM was linked to rustc
statically, but to the LLVM tools dynamically. That meant we were distributing
two copies of LLVM: one as a separate dylib and one statically linked in to
librustc_driver.
Add aarch64-unknown-linux-musl host builds
This adds aarch64-unknown-linux-musl to the hosts list and adds the build to the dist-arm-linux builder as `@Mark-Simulacrum` suggested to me in Zulip. `@jyn514` requested to be mentioned 😄
I had to update the config for crosstool-ng as it had a prompt about the glibc version.
I ran `src/ci/docker/run.sh dist-arm-linux` to test it.
```
Build completed successfully in 1:31:50
Compile requests 8180
Compile requests executed 8135
Cache hits 287
Cache misses 7848
Cache timeouts 0
Cache read errors 0
Forced recaches 0
Cache write errors 0
Compilation failures 0
Cache errors 0
Non-cacheable compilations 0
Non-cacheable calls 36
Non-compilation calls 9
Unsupported compiler calls 0
Average cache write 0.000 s
Average cache read miss 6.389 s
Average cache read hit 0.000 s
Cache location Local disk: "/sccache"
Cache size 202 MiB
Max cache size 10 GiB
== clock drift check ==
local time: Sun Sep 6 19:30:17 UTC 2020
network time: Sun, 06 Sep 2020 19:30:17 GMT
== end clock drift check ==
```
Only errors were in miri due to struct fields being private (already been reported [here](https://github.com/rust-lang/rust/issues/76337))
Edit: Maybe it is helpful if I add that it is a working compiler
```sh
/rust-nightly-aarch64-unknown-linux-musl # ash install.sh
install: creating uninstall script at /usr/local/lib/rustlib/uninstall.sh
install: installing component 'rustc'
install: installing component 'cargo'
install: installing component 'rls-preview'
install: installing component 'rust-analyzer-preview'
install: installing component 'clippy-preview'
install: installing component 'rustfmt-preview'
install: installing component 'llvm-tools-preview'
install: installing component 'rust-analysis-aarch64-unknown-linux-musl'
install: installing component 'rust-std-aarch64-unknown-linux-musl'
install: WARNING: failed to run ldconfig. this may happen when not installing as root. run with --verbose to see the error
Rust is ready to roll.
/ # cat test.rs
fn main() { println!("hello world"); }
/ # rustc test.rs
/ # ./test
hello world
# file test
test: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, not stripped
```
Windows doesn't quite support dynamic linking to LLVM yet, but on other
platforms we do. In #76708, it was discovered that we dynamically link to LLVM
from the LLVM tools (e.g., rust-lld), so we need the shared LLVM library to link
against. That means that if we do not have a shared link to LLVM, and want LLVM
tools to work, we'd be shipping two copies of LLVM on all of these platforms:
one in librustc_driver and one in libLLVM.
Also introduce an error into rustbuild if we do end up configured for shared
linking on Windows.
While waiting for a PR job to start testing my code, I noticed the
symlink-build-dir step took 10 minutes to complete, so I investigated
what caused that.
It seems like something changed in the build environment between version
20200901.1 (where the step took 45 seconds) and version 20200908.1
(where the step took 10 minutes). At the time of writing this commit,
the rust-lang organization is on vertsion 20200908.1, while the
rust-lang-ci organization is at version 20200901.1 (and is not affected
by this yet).
There is no need for this step anymore on GHA, as our XL builders got an
increase in the root paritition size, so this commit removes the code
that moved stuff around on GHA (while keeping it on Azure).
For the record, at the time of writing this, the disk situation is:
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 667G 60G 607G 9% /
/dev/sdb1 110G 4.1G 101G 4% /mnt
Set ninja=true by default
Ninja substantially improves LLVM build time. On a 96-way system, using
Make took 248s, and using Ninja took 161s, a 35% improvement.
We already require a variety of tools to build Rust. If someone wants to
build without Ninja (for instance, to minimize the set of packages
required to bootstrap a new target), they can easily set `ninja=false`
in `config.toml`. Our defaults should help people build Rust (and LLVM)
faster, to speed up development.
Build dist-x86_64-musl with --enable-profiler.
Trying to build a Rust project with `-Zprofile` for target
x86_64-unknown-linux-musl using rustc 1.46.0-nightly (346aec9b0
2020-07-11), installed with rustup, results in the following error.
```
export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -Cpanic=abort"export CARGO_INCREMENTAL=0$ cargo build --target=x86_64-unknown-linux-muslCompiling hello_world v0.1.0 (…)error[E0463]: can't find crate for `profiler_builtins`
|
= note: the compiler may have been built without the profiler runtime
error: aborting due to previous error
For more information about this error, try `rustc --explain E0463`.error: could not compile `hello_world`.
To learn more, run the command again with --verbose.
```
`-Zprofile` is required here to enable grcov profiling.
This is similar in nature to issue
https://github.com/rust-lang/rust/issues/57257, which has been fixed in
asimilar way at https://github.com/rust-lang/rust/pull/60476 .
A fix for Android has also landed not long ago:
https://github.com/rust-lang/rust/pull/70054 .
Signed-off-by: Tiago Lam <tiagol@hadean.com>
Trying to build a Rust project with `-Zprofile` for target
x86_64-unknown-linux-musl using rustc 1.46.0-nightly (346aec9b0
2020-07-11), installed with rustup, results in the following error.
```
export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -Cpanic=abort"export CARGO_INCREMENTAL=0$ cargo build --target=x86_64-unknown-linux-muslCompiling hello_world v0.1.0 (…)error[E0463]: can't find crate for `profiler_builtins`
|
= note: the compiler may have been built without the profiler runtime
error: aborting due to previous error
For more information about this error, try `rustc --explain E0463`.error: could not compile `hello_world`.
To learn more, run the command again with --verbose.
```
`-Zprofile` is required here to enable grcov profiling.
This is similar in nature to issue
https://github.com/rust-lang/rust/issues/57257, which has been fixed in
asimilar way at https://github.com/rust-lang/rust/pull/60476 .
A fix for Android has also landed not long ago:
https://github.com/rust-lang/rust/pull/70054 .
Signed-off-by: Tiago Lam <tiagol@hadean.com>
This Emscripten version was the first to be cut after the LLVM 11
release branch was created, so it should be the most compatible with
LLVM 11. The old version we were using was incompatible with LLVM 11
because its wasm-ld did not understand all the relocations that LLVM
11 emits.
Move CloudABI to tier 3.
The CloudABI target hasn't had much work done on it in a while, and it doesn't appear to be in active use. It has a fairly substantial amount of code, particularly in the [sys module](5addb135ed/library/std/src/sys/cloudabi) that requires actively supporting. I contacted @EdSchouten who indicated that many of the CloudABI concepts are now in WASI, and that they are OK with the target being moved to tier 3.
Release Notes:
```
v1.38.47: 10/02/2019
--------------------
- Add support for FETCH API in WASM backend. This doesn't support FETCH in the
main thread (`USE_FETCH_WORKER=0` is enforced). #9490
- Redefine errno values to be consistent with wasi. This will let us avoid
needing to convert the values back and forth as we use more wasi APIs.
This is an ABI change, which should not be noticeable from user code
unless you use errno defines (like EAGAIN) *and* keep around binaries
compiled with an older version that you link against. In that case, you
should rebuild them. See #9545.
- Removed build option `-s ONLY_MY_CODE` as we now have much better solutions
for that, like building to a wasm object file or using `STANDALONE_WASM`
etc. (see
https://github.com/emscripten-core/emscripten/wiki/WebAssembly-Standalone).
- Emscripten now supports the config file (.emscripten) being placed in the
emscripten directory rather that the current user's home directory.
See #9543
```
Add sanitizer support on FreeBSD
Restarting #47337. Everything is better now, no more weird llvm problems, well not everything:
Unfortunately, the sanitizers don't have proper support for versioned symbols (https://github.com/google/sanitizers/issues/628), so `libc`'s usage of `stat@FBSD_1.0` and so on explodes, e.g. in calling `std::fs::metadata`.
Building std (now easy thanks to cargo `-Zbuild-std`) and libc with `freebsd12/13` config via the `LIBC_CI=1` env variable is a good workaround…
```
LIBC_CI=1 RUSTFLAGS="-Z sanitizer=address" cargo +san-test -Zbuild-std run --target x86_64-unknown-freebsd --verbose
```
…*except* std won't build because there's no `st_lspare` in the ino64 version of the struct, so an std patch is required:
```diff
--- i/src/libstd/os/freebsd/fs.rs
+++ w/src/libstd/os/freebsd/fs.rs
@@ -66,8 +66,6 @@ pub trait MetadataExt {
fn st_flags(&self) -> u32;
#[stable(feature = "metadata_ext2", since = "1.8.0")]
fn st_gen(&self) -> u32;
- #[stable(feature = "metadata_ext2", since = "1.8.0")]
- fn st_lspare(&self) -> u32;
}
#[stable(feature = "metadata_ext", since = "1.1.0")]
@@ -136,7 +134,4 @@ impl MetadataExt for Metadata {
fn st_flags(&self) -> u32 {
self.as_inner().as_inner().st_flags as u32
}
- fn st_lspare(&self) -> u32 {
- self.as_inner().as_inner().st_lspare as u32
- }
}
```
I guess std could like.. detect that `libc` isn't built for the old ABI, and replace the implementation of `st_lspare` with a panic?
Run standard library unit tests without optimizations in `nopt` CI jobs
This was discussed in #73288 as a way to catch similar issues in the future. This builds an unoptimized standard library with the bootstrap compiler and runs the unit tests. This takes about 2 minutes on my laptop.
I confirmed that this method works locally, although there may be a better way of implementing it. It would be better to use the stage 2 compiler instead of the bootstrap one.
Notably, there are currently four `libstd` unit tests that fail in debug mode on `i686-unkown-linux-gnu` (a tier one target):
```
failures:
f32::tests::test_float_bits_conv
f32::tests::test_total_cmp
f64::tests::test_float_bits_conv
f64::tests::test_total_cmp
```
These are the tests that prompted #73288 as well as the ones added in #72568, which is currently broken due to #73328.
We already have builders which built standard library *test*s without
optimizations, but we previously did not have builders which built the standard
library itself without optimizations and then tested that.
This adds those builds for i686 and x86_64 linux.
FreeBSD 10 reached its end-of-life in October 2018, and that toolchain
caused issues in the LLVM 11 upgrade (#73526) that are resolved with the
toolchain from FreeBSD 11.
The purpose of the auto-fallible job is to run builders that are likely
to fail on CI without gating on them. Having fail-fast enabled there
kinda defeats the purpose, as if one of them fails we can't monitor the
outcome of the other ones.
This was prompted by the aarch64-gnu builder consistently failing due to
a broken test, preventing us from seeing if the macOS spurious failure
is fixed.
Add fallible AArch64 CI builder
This adds the `aarch64-gnu` CI builder to the `auto-fallible` job, as a first step in the process of actually gating on it.
r? @Mark-Simulacrum
Update the WASI libc build to LLVM 10.
Among other things, this brings in [the `__main_argc_argv`] patch,
which simplifies the interaction between the compiler and WASI libc's
startup code, which will help work on reactor support.
[the `__main_argc_argv` patch]: 00072c08c7
r? @alexcrichton
Among other things, this brings in [the `__main_argc_argv`] patch,
which simplifies the interaction between the compiler and WASI libc's
startup code, which will help work on reactor support.
[the `__main_argc_argv` patch]: 00072c08c7
Enable docs on dist-x86_64-musl
Add the `rust-docs` component to toolchain `x86_64-unknown-linux-musl`, which allows people using rustup on their musl-based linux distribution to download the rust-docs.
`--disable-docs` is based on the assumption that `x86_64-unknown-linux-musl` is only a cross-compile target.
I have tested that the docs are built. I assume the build-system will automatically detect the docs and create a `rust-docs` component. I will [monitor](https://rust-lang.github.io/rustup-components-history/x86_64-unknown-linux-musl.html) the components and create a follow-up PR, if the docs aren't published.
See also #70619, where we enabled `rust-lld` to enable the wasm-workflow on musl-based linux distributions.
Add the rust-docs component to toolchain x86_64-unknown-linux-musl, which allows
people using rustup on their musl-based linux distribution to download the
rust-docs.
Downgrade glibc to 2.11.1 for ppc, ppc64 and s390x
As discussed in #73782
I've tested these changes on rust 1.43.0 for all the specified archs and used the resulting binaries to bootstrap building rust 1.43.1.
I've also shortly tested these changes on master on ppc64.
Enable perf try builder
This adds a dedicated branch for perf to use for CI, intended to allow perf to
enqueue builds without needing to use bors. bors is great, but bors requires an
open PR to work, and we want to invoke perf on closed PRs sometimes (in
particular, rollups).
This adds a dedicated branch for perf to use for CI, intended to allow perf to
enqueue builds without needing to use bors. bors is great, but bors requires an
open PR to work, and we want to invoke perf on closed PRs sometimes (in
particular, rollups).
ci: Update dist-{i686,x86_64}-linux to Debian 6
This increases the minimum `{i686,x86_64}-unknown-linux-gnu` platform
from RHEL/CentOS 5 (glibc 2.5 and kernel 2.6.18) to a slightly newer
Debian 6 `squeeze` (glibc 2.11 and kernel 2.6.32). While that release is
already EOL, it happens to match the minimum common versions of two
enterprise distros that do still need Rust support -- RHEL 6 (glibc 2.12
and kernel 2.6.32) and SLES 11 SP4 (glibc 2.11 and kernel 3.0).
Closes#62516.
ci: Set `shell: bash` as a default, remove duplicates
A follow-up to #74406, this commit merely removes the `shell: bash` lines where they were added in favor of setting defaults for *all* "run" steps in the jobs that run the tests.
The changes in #74406 were needed because of an upstream change to the `windows-2019` GitHub Actions image. Previously, the configuration worked fine without specifying `shell: bash`, but for some reason this broke with a new change that was deployed today. The preceding PR was a hotfix to get CI passing, but there was a slightly less duplicative way to specify the default shell for the jobs, which was to set the `defaults.run` option.
This change applies to the `pr`, `try`, `auto`, and `auto-fallible` jobs, which are derived from the YAML-anchor `base-ci-job`. I did not apply these changes to the `master`, `try-success`, `try-failure`, `auto-success`, or `auto-failure` jobs because they have only a few steps.
cc/r? @Mark-Simulacrum
Also, promote defaults.run.shell from inside only the primary jobs to
the top level.
The src/ci/exec-with-shell.py wrapper script was formerly used to change
out the shell mid-job by intercepting a CI_OVERRIDE_SHELL environment
variable. Now, instead, we just set `bash` as the global default across
all jobs, and we also delete the exec-with-shell.py script.
Signed-off-by: Kristofer Rye <kristofer.rye@gmail.com>
This will render the src/ci/exec-with-shell.py script more or less
useless, but we're going to replace that by just using the system bash
instead.
Signed-off-by: Kristofer Rye <kristofer.rye@gmail.com>
Bump version to 1.47
This also bumps to a more recent rustfmt version, just to keep us relatively up to date (though almost nothing has changed in rustfmt we use beyond bumps to the parser infra). No formatting changes as a result of this.
r? @pietroalbini
A follow-up to #74406, this commit merely removes the `shell: bash`
lines where they are explicitly added in favor of setting defaults for
*all* "run" steps.
Signed-off-by: Kristofer Rye <kristofer.rye@gmail.com>
Update cross-compilation README
README seemed rather out of date. I hope the information in my PR is now correct (it was more or less assembled by asking in zulip and learning-by-doing).
RISC-V GNU/Linux as host platform
This PR add a new builder named `dist-riscv64-linux` that builds the compiler toolchain for RISC-V 64-bit GNU/Linux.
r? @alexcrichton
This increases the minimum `{i686,x86_64}-unknown-linux-gnu` platform
from RHEL/CentOS 5 (glibc 2.5 and kernel 2.6.18) to a slightly newer
Debian 6 `squeeze` (glibc 2.11 and kernel 2.6.32). While that release is
already EOL, it happens to match the minimum common versions of two
enterprise distros that do still need Rust support -- RHEL 6 (glibc 2.12
and kernel 2.6.32) and SLES 11 SP4 (glibc 2.11 and kernel 3.0).
build dist for x86_64-unknown-illumos
This change creates a new Docker image, "dist-x86_64-illumos", and sets
things up to build the full set of "dist" packages for illumos hosts, so
that illumos users can use "rustup" to install packages. It also
adjusts the manifest builder to expect complete toolchains for this
platform.
Fix disabled dockerfiles
When the dockerfiles were moved into the host-x86_64 directory, paths
for COPY commands were updated with the new host-x86_64/ prefix. This
suggested that the intended context was src/ci/docker. However, the context
for disabled docker images was src/ci/docker/host-x86_64. This broke the new
paths and prevented src/ci/docker/scripts from being included in the
context at all.
This commit corrects this context allowing docker to find the files it
needs for COPY commands.
Also includes a quick fix to riscv recommended by @bjorn3
In our GitHub Actions setup macOS is too unreliable to gate on it, but
the other builders work fine. This commit splits the macOS builders into
a separate job (called auto-fallible), allowing us to gate on the auto
job without failing due to macOS spurious failures.
This change creates a new Docker image, "dist-x86_64-illumos", and sets
things up to build the full set of "dist" packages for illumos hosts, so
that illumos users can use "rustup" to install packages. It also
adjusts the manifest builder to expect complete toolchains for this
platform.
When the dockerfiles were moved into the host-x86_64 directory, paths
for COPY commands were updated with the new host-x86_64/ prefix. This
suggested that the intended context was src/ci/docker. However, the context
for disabled docker images was src/ci/docker/host-x86_64. This broke the new
paths and prevented src/ci/docker/scripts from being included in the
context at all.
This commit corrects this context allowing docker to find the files it
needs for COPY commands.
Suggested by @bjorn3
Every RUN command creates a new overlay on top of the image as of before
the RUN command. Using fewer RUN commands prevents intermediate overlays
(which in this case would have contained the entire Linux source tree).
We're starting to include native aarch64 machines in our CI, but before
this commit the architecture wasn't included in the cache key for our
Docker images. This means there could be conflicts between images
produced on different architectures, hurting our CI times.
This commit fixes the problem by including the output of `uname -m` in
the cache key.
We need to add runners designed for an aarch64 host system, and it'd be
nice to return an error message if someone tries to run an image
designed for an host architecture in another one.
To start the work on this, this commit moves all the existing builders
in the host-x86_64 directory, and changes the run.sh script to look up
the image in the correct directory based on the host architecture.
Add option for local docker testing.
This adds the option `--dev` to `src/ci/docker/run.sh` so that it will enter an interactive environment for local testing. I have often needed this for testing things, but I always needed to edit this script. I wanted the ability to interact in the environment, run different commands, inspect errors, etc.
Use WASM's saturating casts if they are available
WebAssembly supports saturating floating point to integer casts behind a target feature. The feature is already available on many browsers. Beginning with 1.45 Rust will start defining the behavior of floating point to integer casts to be saturating as well. For this Rust constructs additional checks on top of the `fptoui` / `fptosi` instructions it emits. Here we introduce the possibility for the codegen backend to construct saturating casts itself and only fall back to constructing the checks ourselves if that is not possible.
Resolves part of #73591
Rollup of 10 pull requests
Successful merges:
- #73414 (Implement `slice_strip` feature)
- #73564 (linker: Create GNU_EH_FRAME header by default when producing ELFs)
- #73622 (Deny unsafe ops in unsafe fns in libcore)
- #73684 (add spans to injected coverage counters, extract with CoverageData query)
- #73812 (ast_pretty: Pass some token streams and trees by reference)
- #73853 (Add newline to rustc MultiSpan docs)
- #73883 (Compile rustdoc less often.)
- #73885 (Fix wasm32 being broken due to a NodeJS version bump)
- #73903 (Changes required for rustc/cargo to build for iOS targets)
- #73938 (Optimise fast path of checked_ops with `unlikely`)
Failed merges:
r? @ghost
Fix wasm32 being broken due to a NodeJS version bump
Emscripten's SDK [recently bumped the version of NodeJS they shipped](https://github.com/emscripten-core/emsdk/pull/529), but our Dockerfile for the wasm32 builder hardcoded the version number. This will cause consistent CI failures once the currently cached image is rebuilt (either due to a change or due to the cache expiring).
This PR fixes the problem by finding the latest version of NodeJS in the Emscripten SDK and symlinking it to a "latest" directory, which is then added to the `PATH`.
Remove legacy InnoSetup GUI installer
On Windows the InnoSetup `.exe` installer was superseded by the MSI installer long ago. It's no longer needed.
The `.exe` installer hasn't been linked from the [other installation methods](https://forge.rust-lang.org/infra/other-installation-methods.html#standalone) page in many years. As far as I can tell the intent was always to remove this installer once the MSI proved itself. Though admittedly both installers feel very "legacy" at this point.
Removing this would mean we only maintain one Windows GUI installer and would speed up the distribution phase.
As a result of removing InnoSetup, this closes#24397
Test UI tests for pass=check
I'm going to just compare the builder times since I wasn't able to get this working nicely locally (hit some obscure linker error).
Fixes part of #69823
We do not test cross-compilation here as the PR builder lacks a sufficiently
recent LLVM to cross-compile to 32-bit linux. Once we bump the minimum LLVM
version to LLVM 9, this can use normal 32-bit linux.
Revert "ci: allow gating gha on everything but macOS"
The macOS issue on GHA's side seems to be fixed, so this is not needed anymore.
r? @Mark-Simulacrum
Emscripten's SDK recently bumped the version of NodeJS they shipped, but
our Dockerfile for the wasm32 builder hardcoded the version number. This
will cause consistent CI failures once the currently cached image is
rebuilt (either due to a change or due to the cache expiring).
This commit fixes the problem by finding the latest version of NodeJS in
the Emscripten SDK and symlinking it to a "latest" directory, which is
then added to the PATH.
Update mdbook
This updates mdbook to 0.4. The list of changes can be found at https://github.com/rust-lang/mdBook/blob/master/CHANGELOG.md#mdbook-040. I think the most important one is the change to include fonts with the book instead of using the Google Fonts CDN. This adds a few megabytes of fonts to the docs component. It may be possible to share the fonts across the books, but would take a fair bit of work to make that happen, so I'm not sure if it is necessary.
This also removes mdbook-linkcheck. It is currently not being used, and I don't foresee it going back into use anytime soon. I would prefer not to maintain something that isn't being used, and it removes a very large number of dependencies.
ci: allow gating GHA on everything but macOS
In our GitHub Actions setup macOS is too unreliable to gate on it, but the other builders work fine. This commit splits the macOS builders into a separate job (called `auto-fallible`), allowing us to gate on the auto job without failing due to macOS spurious failures.
cc https://github.com/rust-lang/rust-central-station/issues/848
r? @Mark-Simulacrum
ci: disable alt build during try builds
The alt build is not actually needed often, and it can be added back on a case-by-case basis if a specific PR needs access to it.
This will free up a builder.
r? @Mark-Simulacrum
RISC-V Emulated Testing
Adds a disabled docker image on which to run RISC-V tests. Based on the armhf image.
Test using
```
./src/ci/docker/run.sh riscv64gc-linux
```
cc: @msizanoen1
When we're running with dry_run enabled (i.e. all builds do this initially), we're
guaranteed to save of a toolstate of TestFail for tools that aren't tested. In practice,
we do test tools as well, so for those tools we would initially record them as being
TestPass, and then later on re-record the correct state after actually testing them.
However, this would not work well if the build failed for whatever reason (e.g. panicking
in bootstrap, or as was the case in 73097, clippy failing to test successfully), we would
just go on believing that things passed when they in practice did not.
This commit also adjusts saving toolstate to never record clippy explicitly (otherwise, it
would be recorded when building it); eventually that'll likely move to other tools as well
but not yet. This is deemed simpler than checking everywhere we generically save
toolstate.
We also move clippy out of the "toolstate" no-fail-fast build into a separate x.py
invocation; this should no longer be technically required but provides the nice state of
letting us check toolstate for all tools and only then check clippy (giving full results
on every build).
In our GitHub Actions setup macOS is too unreliable to gate on it, but
the other builders work fine. This commit splits the macOS builders into
a separate job (called auto-fallible), allowing us to gate on the auto
job without failing due to macOS spurious failures.
Remove vestigial CI job msvc-aux.
This CI job isn't really doing anything, so it seems prudent to remove it.
For some history:
* This was introduced in #48809 when the msvc job was split in two to keep it under 2 hours (oh the good old days). At the time, this check-aux job did a bunch of things:
* tidy
* src/test/pretty
* src/test/run-pass/pretty
* src/test/run-fail/pretty
* src/test/run-pass-valgrind/pretty
* src/test/run-pass-fulldeps/pretty
* src/test/run-fail-fulldeps/pretty
* Tidy was removed in #60777.
* run-pass and run-pass-fulldeps moved to UI in #63029
* src/test/pretty removed in #58140
* src/test/run-fail moved to UI in #71185
* run-fail-fulldeps removed in #51285
Over time through attrition, the job was left with one lonely thing: `src/test/run-pass-valgrind/pretty`. And of course, this wasn't actually running the "pretty" tests. The normal `run-pass-valgrind` tests ran, and then when it tried to run in "pretty" mode, all the tests were ignored because compiletest thought nothing had changed (apparently compiletest isn't fingerprinting the mode? Needs more investigation…). `run-pass-valgrind` is already being run as part of `x86_64-msvc-1`, so there's no need to run it here.
I've taken the liberty of removing `src/test/run-pass-valgrind/pretty` as a distinct test. I'm guessing from the other PR's that the pretty tests should now live in `src/test/pretty`, and that the team has moved away from doing pretty tests on other parts of the `src/test` tree.
Update musl to 1.1.24
Release notes since previous version 1.1.22:
## 1.1.23 release notes
### new features:
- riscv64 port
- configure now allows customizing AR and RANLIB vars
- header-level support for new linux features in 5.1
### major internal changes:
- removed extern __syscall; syscall header code is now fully self-contained
### performance:
- new math library implementation for log/exp/pow
- aarch64 dynamic tlsdesc function is streamlined
### compatibility & conformance:
- O_TTY_INIT is now defined
- sys/types.h no longer pollutes namespace with sys/sysmacros.h in any profile
- powerpc asm is now compatible with clang internal assembler
### changes for new POSIX interpretations:
- fgetwc now sets stream error indicator on encoding errors
- fmemopen no longer rejects 0 size
### bugs fixed:
- static TLS for shared libraries was allocated wrong on "Variant I" archs
- crash in dladdr reading through uninitialized pointer on non-match
- sigaltstack wrongly errored out on invalid ss_size when doing SS_DISABLE
- getdents function misbehaved with buffer length larger than INT_MAX
- set*id could deadlock after fork from multithreaded process
### arch-specfic bugs fixed:
- s390x SO_PEERSEC definition was wrong
- passing of 64-bit syscall arguments was broken on microblaze
- posix_fadvise was broken on mips due to missing 7-arg syscall support
- vrregset_t layout and member naming was wrong on powerpc64
## 1.1.24 release notes
### new features:
- GLOB_TILDE extension to glob
- non-stub catgets localization API, using netbsd binary catalog format
- posix_spawn file actions for [f]chdir (extension, pending future standard)
- secure_getenv function (extension)
- copy_file_range syscall wrapper (Linux extension)
- header-level support for new linux features in 5.2
### performance:
- new fast path for lrint (generic C version) on 32-bit archs
### major internal changes:
- functions involving time are overhauled to be time64-ready in 32-bit archs
- x32 uses the new time64 code paths to replace nasty hacks in syscall glue
### compatibility & conformance:
- support for powerpc[64] unaligned relocation types
- powerpc[64] and sh sys/user.h no longer clash with kernel asm/ptrace.h
- select no longer modifies timeout on failure (or at all)
- mips64 stat results are no longer limited to 32-bit time range
- optreset (BSD extension) now has a public declaration
- support for clang inconsistencies in wchar_t type vs some 32-bit archs
- mips r6 syscall asm no longer has invalid lo/hi register clobbers
- vestigial asm declarations of __tls_get_new are removed (broke some tooling)
- riscv64 mcontext_t mismatch glibc's member naming is corrected
### bugs fixed:
- glob failed to match broken symlinks consistently
- invalid use of interposed calloc to allocate initial TLS
- various dlsym symbol resolution logic errors
- semctl with SEM_STAT_ANY didn't work
- pthread_create with explicit scheduling was subject to priority inversion
- pthread_create failure path had data race for thread count
- timer_create with SIGEV_THREAD notification had data race getting timer id
- wide printf family failed to support l modifier for float formats
### arch-specific bugs fixed:
- x87 floating point stack imbalance in math asm (i386-only CVE-2019-14697)
- x32 clock_adjtime, getrusage, wait3, wait4 produced junk (struct mismatches)
- lseek broken on x32 and mipsn32 with large file offsets
- riscv64 atomics weren't compiler barriers
- riscv64 atomics had broken asm constraints (missing earlyclobber flag)
- arm clone() was broken when compiled as thumb if start function returned
- mipsr6 setjmp/longjmp did not preserve fpu register state correctly
Fixes#71099.
This will reduce costs, as well as lays the groundwork for developers to be able
to locally pull the published docker images without needing AWS credentials.
[CI] Use the latest Python available on Windows
This PR changes our Windows CI to always use the latest Python interpreter available in the GHA tool cache instead of hardcoding Python 3.7.6. This is needed because occasionally GitHub bumps the installed version, deleting the previous one.
This fixes the current GHA outage we're having. I fully expect the outage to propagate to Azure Pipelines in the coming days if we don't merge this, as both GHA and Azure use the same underlying image. Once the PR is merged we can re-enabled the double-gating.
r? @Mark-Simulacrum
This commit changes our Windows CI to always use the latest Python
interpreter available in the GHA tool cache instead of hardcoding Python
3.7.6. This is needed because occasionally GitHub bumps the installed
version, deleting the previous one.
Ensure that `./x.py doc --stage 0 src/libstd` works via CI
This was split off from #71645, which recommends that users first try building `libstd` docs with the bootstrap `rustdoc`. This should work in most cases, but will fail if we start using a very recent `rustdoc` feature outside a `#[cfg(not(bootstrap))]`.
It would be very nice to guarantee that `./x.py doc --stage 0 src/libstd` works, since it allows documentation changes to be rendered locally without needing to build the compiler. However, it may put too big a burden on `rustdoc` developers who presumably want to dogfood new features.
Enable "full tools" option on ARM dist builders
This commit switches the `--enable-extended` option on the arm-related
dist builders to `--enable-full-tools`. This alias in `config.py`
corresponds to enabling a few more options:
* `rust.lld = true` - this is the main purpose of this PR, to enable LLD
on ARM-related platforms. This means it will effectively unlock
compilation of wasm programs from an arm host.
* `rust.llvm-tools = true` - it turns out that this option is largely
ignored in rustbuild today. This is only read in one location to set
some flags for the `llvm-tools` package, but the `llvm-tools` package
is already produced on all of these builders. It's predicted that this
will have no effect on build times.
* `rust.codegen-backends = ['llvm']` - historically this also enabled
the emscripten backend, but that has long since been removed.
This brings the ARM dist builders in line with the x86_64 dist builders
using this flag. The hope is that the extra time spent on CI building
LLD will acceptable because it's cached by `sccache`, LLD is a
relatively small C++ project, and the dist builders are all clocking
well under 3 hours (the slowest of all builders) around 2 hours.
There's likely some possible cleanup that can happen with these
configure options since it doesn't look like they've aged too too well,
but I'm hopeful that possible refactorings, if necessary, could be
deferred to future PRs.
This commit switches the `--enable-extended` option on the arm-related
dist builders to `--enable-full-tools`. This alias in `config.py`
corresponds to enabling a few more options:
* `rust.lld = true` - this is the main purpose of this PR, to enable LLD
on ARM-related platforms. This means it will effectively unlock
compilation of wasm programs from an arm host.
* `rust.llvm-tools = true` - it turns out that this option is largely
ignored in rustbuild today. This is only read in one location to set
some flags for the `llvm-tools` package, but the `llvm-tools` package
is already produced on all of these builders. It's predicted that this
will have no effect on build times.
* `rust.codegen-backends = ['llvm']` - historically this also enabled
the emscripten backend, but that has long since been removed.
This brings the ARM dist builders in line with the x86_64 dist builders
using this flag. The hope is that the extra time spent on CI building
LLD will acceptable because it's cached by `sccache`, LLD is a
relatively small C++ project, and the dist builders are all clocking
well under 3 hours (the slowest of all builders) around 2 hours.
There's likely some possible cleanup that can happen with these
configure options since it doesn't look like they've aged too too well,
but I'm hopeful that possible refactorings, if necessary, could be
deferred to future PRs.
ci: run mir-opt tests on PR CI also as 32-bit (for `EMIT_MIR_FOR_EACH_BIT_WIDTH`).
Background: #69916 and [`src/test/mir-opt/README.md`](https://github.com/rust-lang/rust/blob/master/src/test/mir-opt/README.md):
> By default 32 bit and 64 bit targets use the same dump files, which can be problematic in the
presence of pointers in constants or other bit width dependent things. In that case you can add
>
> ```
> // EMIT_MIR_FOR_EACH_BIT_WIDTH
> ```
>
> to your test, causing separate files to be generated for 32bit and 64bit systems.
However, if you change the output of such a test (intentionally or not), or if you add a test and it varies between 32-bit and 64-bit platforms, you have to run this command (for a x64 linux host):
`./x.py test --stage 1 --target x86_64-unknown-linux-gnu --target i686-unknown-linux-gnu --bless src/test/mir-opt`
Otherwise, bors trying to merge the PR will fail, since we test 32-bit targets there.
But we don't on PR CI, which means there's no way the PR author would know (unless they were burnt by this already and know what to look for).
This PR resolves that by running `mir-opt` tests for ~~`i686-unknown-linux-gnu`~~, on PR CI.
**EDIT**: switched to `armv5te-unknown-linux-gnueabi` to work around LLVM 7 crashes (see https://github.com/rust-lang/compiler-builtins/pull/311#issuecomment-612270089), found during testing.
cc @rust-lang/wg-mir-opt @rust-lang/infra
Stop explicitly depending on python 2
This PR revises our previous policy of officially only supporting and testing with python 2 in the CI environment to instead test with python 3. It also changes the defaults to python 3 in our various scripts (usually, by way of `python` rather than `python3` to preserve compatibility with systems that do not have a python 3 available).
The effect of this is that we expect all new patches to support python 3 (and will test as such). We explicitly also expect that patches support python 2.7 as well -- and test as such, though only on one builder. This is intended as a temporary, though likely long-lived, measure to preserve compatibility while looking towards the future which is likely to be a python 3 only world. We do not at this point set a timeline for when we'll drop support for python 2.7; it's plausible that this is months or years into the future, depending on how quickly the ecosystem drops support and how painful it is for us to maintain that support over time.
Closes#65063 (as far as I can tell; please file explicit and separate issues or PRs if not).
Enable rust-lld on dist-x86_64-musl
Add rust-lld to rustup llvm-tools-preview on nightly for musl
I am using a musl distro on my workstation, with `RUSTFLAGS="-C target-feature=-crt-static"` this works fine. I know that `x86_64-unknown-linux-musl` was originally only meant as a target and not as a host. But most problems have been fixed, and I have fewer problems with `unknown` (rustup) than when I am using `x86_64-alpine-linux-musl` (rust installed by the distro). The only thing I am missing is rust-lld in llvm-tools-preview on nightly.
I needed rust-lld for a wasm tutorial. I built rust-lld and tested it with that tutorial, and it worked well. I asked [here](https://users.rust-lang.org/t/enable-rust-lld-on-x86-64-unknown-linux-musl/39851) where to request to enable lld and ended up doing this PR.
I compared llvm-tools-preview `nightly-x86_64-unknown-linux-musl` and `nightly-x86_64-unknown-linux-gnu`: only rust-lld is missing in musl.
I tested the change using:
```bash
./src/ci/docker/run.sh dist-x86_64-musl
```
And I checked that the resulting rust-lld binary runs.
There are some builders that are running out of disk space while
building the Docker images, such as arm-android. This moves and symlinks
/var/lib/docker to the /mnt partition on Linux GHA.
GHA: enable running multiple try builds at the same time
While for auto, try and PR builds we only want the latest commit to be tested, that's not true for try builds: each commit pushed to the branch is a different PR being tested, and we want multiple PRs to be tested in parallel if there is enough demand.
Fixes#70569
While for auto, try and PR builds we only want the latest commit to be
tested, that's not true for try builds: each commit pushed to the branch
is a different PR being tested, and we want multiple PRs to be tested in
parallel if there is enough demand.
Fixes#70569
The /mnt mount point has 53GB of free disk space at the time of writing
this commit, so this moves the build there to avoid running out of disk
space during builds.
On Azure Pipeliones, the C: filesystem is huge with a lot of free space,
while D: is small. By default builds happened in D:, so we added a
script to symlink the big directories to C:, granting us more space.
Filesystem Size Used Avail Use%
C: 256G 143G 114G 56%
D: 14G 2.0G 13G 15%
On GitHub Actions instead C: is almost full, and we have a lot of free
space on D:, where the build happens.
Filesystem Size Used Avail Use%
C: 128G 114G 15G 89%
D: 56G 4.8G 52G 9%
This commit stops creating the symlink on GitHub Actions, fixing the out
of disk space errors we were seeing on some Windows builders.
Apparently the old path we were using for Python 2 on Windows was not
documented, and eventually got removed. This switches our CI to use the
correct path.
