Certain features of Linux (getauxval() and epoll_create1()) are only
available in android SDK/NDK levels 18 and 21 respectively. The 32bit
platform is currently on level 14 for compatibility with Android 4.0.
This patch adds SDK/NDK level 21 to the docker for 32 bit platforms,
while leaving the default setup at level 14.
With this done, projects such as `rustup` which rely on these dockers
can build with modern ecosystem crates such as tokio 1.0, by using
the level 21 toolchain, but those which do not need to switch will
be unaffected, since the level 14 toolchain remains available.
Signed-off-by: Daniel Silverstone <dsilvers@digital-scurf.org>
Pin es-check version to prevent unrelated CI failures
es-check v5.2.1 causes a lot of unrelated CI failures on mingw-check, e.g. https://github.com/rust-lang/rust/pull/80723#issuecomment-790294196.
es-check v5.2.2 fixes it but let's pin its version to prevent further failures.
Upgrade to LLVM 12
This implements the necessary adjustments to make rustc work with LLVM 12. I didn't encounter any major issues so far.
r? `@cuviper`
musl-toolchain.sh is called with REPLACE_CC=1, so it will replace
the host compiler and the subsequent cmake build will fail because
it cannot find the openssl headers.
Move the cmake build earlier, so it happens before the compiler
is replaced.
Python 2 is needed for Clang 10, Python 3 for LLVM 12.
The Python 2 dependency could be removed by upgrading to Clang 11,
but that causes linker errors of unclear origin.
Rustdoc gui tests
This is a reopening of #70533.
For this first version, there will be no screenshot comparison. Also, a big change compared to the previous version: the tests are now hosted in the rust repository directly. Since there is no image, it's pretty lightweight to say the least.
So now, only remains the nodejs script to run the tests and the tests themselves. Just one thing is missing: where should I put the documentation for these tests? I'm not sure where would be the best place for that. The doc will contain important information like the documentation of the framework used and how to install it (`npm install browser-ui-test`, but still needs to be put somewhere so no one is lost).
We'd also need to install the package when running the CI too. For now, it runs as long as we have nodejs installed, but I think we don't it to run in all nodejs targets?
cc `@jyn514`
r? `@Mark-Simulacrum`
Upgrade wasm32 image to Ubuntu 20.04
This switches the wasm32 image, which is used to test
wasm32-unknown-emscripten, to Ubuntu 20.04. While at it, enable
most of the excluded tests, as they seem to work fine with some
minor fixes.
This switches the wasm32 image, which is used to test
wasm32-unknown-emscripten to Ubuntu 20.04. While at it, enable
most of the excluded tests, as they seem to work fine with some
minor fixes.
CI: only copy python.exe to python3.exe if the latter does not exist
We're copying the binary to make sure we can call `python3.exe` around, but it seems like the base image of GitHub Actions changed, copying the file before we do so. This PR changes the CI script to only copy the file if it doesn't already exist.
r? `@m-ou-se`
cc `@Mark-Simulacrum`
We're copying the binary to make sure we can call python3.exe around,
but it seems like the base image of GitHub Actions changed, copying the
file before we do so.
This commit changes the CI script to only copy the file if it doesn't
already exist.
Update test-various to Ubuntu 20.04
The test command-setgroups.rs is adjusted to skip on musl, where `sysconf(_SC_NGROUPS_MAX)` always returns a dummy value of 32, even though the actual value is 65536. I'm not sure why this only became a problem now, as the information I found indicates that this value changed in kernel version 2.6.4, which is ages ago.
I'm a bit unsure whether this one will go through, because I locally also saw a failure in std-backtrace.rs which went away on subsequent runs, and also had port assignment failures, but I think those might be on my side. I'm kind of curious how the code in b122908617/library/std/src/net/test.rs (L43-L56) is supposed to work, as the directory names it checks don't seem to appear anywhere else.
r? `@Mark-Simulacrum`
Update armhf-gnu to Ubuntu 20.04
This requires updating the used Linux kernel to avoid an assembler
error, the used busybox version to avoid a linker error, the used
rootfs to match the host version and the qemu flags to work with
the newer version.
r? `@Mark-Simulacrum`
Update dist-various to Ubuntu 20.04
This updates the dist-various-1 and dist-various-2 images to Ubuntu
20.04. This requires some adjustments:
* `DEBIAN_FRONTEND=noninteractive` required for apt install.
* `team-gcc-argm-embedded` PPA does not support focal. However,
we can simply use the distro-provided `gcc-arm-none-eabi`. Per
the comment, the PPA was only used to get a newer version.
* rumprun has to be updated to avoid a linker error.
* We need to build rumrun with `NOGCCERROR`, which disables use
of `-Werror` and allows building with a newer compiler.
* We need to install `libtinfo5`, which appears to be a dependency
of the clang used during the fuchsia build.
* We need to switch to `g++-8` rather than `g++-7`, as at least
`g++-7-arm-linux-gnueabi` is not available on focal.
* We need to upgrade to GCC 6.5 for the Solaris build, as GCC 6.4
does not support the newer libisl version.
r? `@Mark-Simulacrum`
This requires updating the used Linux kernel to avoid an assembler
error, the used busybox version to avoid a linker error, the used
rootfs to match the host version and the qemu flags to work with
the newer version.
The test command-setgroups.rs is adjusted to skip on musl, where
`sysconf(_SC_NGROUPS_MAX)` always returns a dummy value of 32,
even though the actual value is 65536. I'm not sure why this becomes
relevant only now though, as this was apparently the case since
kernel 2.6.4.
This updates the dist-various-1 and dist-various-2 images to Ubuntu
20.04. This requires some adjustments:
* `DEBIAN_FRONTEND=noninteractive` required for apt install.
* `team-gcc-argm-embedded` PPA does not support focal. However,
we can simply use the distro-provided `gcc-arm-none-eabi`. Per
the comment, the PPA was only used to get a newer version.
* rumprun has to be updated to avoid a linker error.
* We need to build rumrun with `NOGCCERROR`, which disables use
of `-Werror` and allows building with a newer compiler.
* We need to install `libtinfo5`, which appears to be a dependency
of the clang used during the fuchsia build.
* We need to switch to `g++-8` rather than `g++-7`, as at least
`g++-7-arm-linux-gnueabi` is not available on focal.
* We need to upgrade to GCC 6.5 for the Solaris build, as GCC 6.4
does not support the newer libisl version.
LLVM 12 no longer builds with Python 2, so install Python 3 in
preparatin.
However, Clang 10 does not build with Python 3, so we need update
to Clang 11 as well, which supports both.
Unfortunately, doing so results in errors while linking the
libLLVM.so into other binaries:
> __morestack: invalid needed version 2
This is fixed by using LLD instead. Possibly this is due to a binutils
linker bug, but updating to the latest binutils version does not fix
it.
This brings in an implementation of `current_dir` and `set_current_dir`
(emulation in `wasi-libc`) as well as an updated version of finding
relative paths. This also additionally updates clang to the latest
release to build wasi-libc with.
Only produce .xz tarballs on CI
This PR adds a `./configure` option to choose which tarball compression formats to produce, and changes our CI configuration to only produce `.xz` tarballs. The release process will then recompress everything into `.gz` when producing a release.
This will drastically reduce our storage costs for CI artifacts, as we'd stop storing the same data twice. **Stable, beta and nightly releases will not be affected by this at all.**
Before landing this we'll need to increase the VM size of our release process, to recompress everything in a reasonable amount of time.
r? `@Mark-Simulacrum`
This implements support for applying PGO to the rustc compilation step (not
standard library or any tooling, including rustdoc). Expanding PGO to more tools
is not terribly difficult but will involve more work and greater CI time
commitment.
For the same reason of avoiding greater time commitment, this currently avoids
implementing for platforms outside of x86_64-unknown-linux-gnu, though in
practice it should be quite simple to extend over time to more platforms. The
initial implementation is intentionally minimal here to avoid too much work
investment before we start seeing wins for a subset of Rust users.
The choice of workloads to profile here is somewhat arbitrary, but the general
rationale was to aim for a small set that largely avoided time regressions on
perf.rust-lang.org's full suite of crates. The set chosen is libcore, cargo (and
its dependencies), and a few ad-hoc stress tests from perf.rlo. The stress tests
are arguably the most controversial, but they benefit those cases (avoiding
regressions) and do not really remove wins from other benchmarks.
The primary next step after this PR lands is to implement support for PGO in
LLVM. It is unclear whether we can afford a full LLVM rebuild in CI, though, so
the approach taken there may need to be more staggered. rustc-only PGO seems
well affordable on linux at least, giving us up to 20% wall time wins on some
crates for 15 minutes of extra CI time (1 hour up from 45 minutes).
The PGO data is uploaded to allow others to reuse it if attempting to reproduce
the CI build or potentially, in the future, on other platforms where an
off-by-one strategy is used for dist builds at minimal performance cost.
This commit switches the x86_64-gnu-nopt builder to use Ubuntu 20.04,
which contains a more recent gdb version than Ubuntu 16.04 (newer gdb
versions fix a bug that Split DWARF can trigger, see
rust-lang/rust#77177 for motivation). x86_64-gnu-nopt is chosen because
it runs compare modes, which is how Split DWARF testing is implemented
in rust-lang/rust#77177.
Signed-off-by: David Wood <david@davidtw.co>
It seems that by default the 32-bit headers are not actually installed
when installing development tooling. As we're using gcc headers,
we need to install them as an extra package.
See for reference:
- https://stackoverflow.com/a/54082790
- https://askubuntu.com/a/106092
Also removed the now unused arm tooling
Now with LLVM 9 being the minimum supported version, we can
finally remove the hacks in the dockerfile.
This wasn't in the main PR bumping the version as I didn't quite
understand what's going on and needed here.
apparently llvm-8-tools already had llvm-8-dev as a dependency
which was removed in llvm-9-tools, so we need to explicitly pull
llvm-9-dev to make a build
Demote i686-unknown-freebsd to tier 2 compiler target
While technically the `i686-unknown-freebsd` target has been a tier 2 development platform for a long time, with full toolchain tarballs available on static.rust-lang.org, due to a bug in the manifest generation the target was never available for download through rustup.
The infrastructure team privately inquired the FreeBSD package maintainers, and they weren't relying on those tarballs either, so it's a fair assumption to say practically nobody is using those tarballs.
This PR then removes the CI builder that produces full tarballs for the target, and moves the compilation of `rust-std` for the target in `dist-various-2`. The `x86_64-unknown-freebsd` target is *not* affected.
cc `@rust-lang/infra` `@rust-lang/compiler` `@rust-lang/release`
r? `@Mark-Simulacrum`
Test clippy on PR CI on changes
This runs the tools builder (which builds and tests tools, including clippy) when the clippy submodule changes. This essentially returns us to the prior state when clippy was a submodule; it makes sense for us to test it on CI when it changes. It might make sense for it to be tested regardless of changing but it is somewhat rare for it to fail and we don't want to add to CI time for the majority of PRs which don't affect it.
Fixes#76999.
Promote aarch64-unknown-linux-gnu to Tier 1
This PR promotes the `aarch64-unknown-linux-gnu` target to Tier 1, as proposed by [RFC 2959]:
* The `aarch64-gnu` CI job is moved from `auto-fallible` to `auto`.
* The platform support documentation is updated, uplifting the target to Tiert 1 with a note about missing stack probes support.
* Building the documentation is enabled for the target, as we produce the `rust-docs` component for all Tier 1 platforms.
[RFC 2959]: https://github.com/rust-lang/rfcs/pull/2959
Historically we've disabled these assertions on a number of platforms with the
goal of speeding up CI. Now, though, having migrated to GitHub actions, CI is
already pretty fast, and these debug assertions do bring us some value.
This does leave in some debug assertions that are performance-related: macOS
currently hovers at just under 2 hours.
There are also some other builders which have debug and LLVM assertions
disabled:
llvm-8, PR builder:
In one view, this builder tests our support for older LLVMs. But in reality, a
lot of our tests already disable themselves on older LLVMs, and I think our
general stance is that we really only support the in-tree LLVM. Plus, we really
want CI times on this builder to be really low, as it's run on *every* PR --
that's a lot of CI time.
test-various:
This disables debug asserts still -- as noted in the Dockerfile, we test code
size, and we need debug asserts off for that to work well.
This was recommended by GitHub Support to try reducing the things that
could've caused #78743. I checked the changelog and there should be no
practical impact for us (we already set an explicit fetch-depth).
While technically the i686-unknown-freebsd target has been a tier 2
development platform for a long time, with full toolchain tarballs
available on static.rust-lang.org, due to a bug in the manifest
generation the target was never available for download through rustup.
The infrastructure team privately inquired the FreeBSD package
maintainers, and they weren't relying on those tarballs either, so it's
a fair assumption to say practically nobody is using those tarballs.
This PR then removes the CI builder that produces full tarballs for the
target, and moves the compilation of rust-std for the target in
dist-various-2.
The x86_64-unknown-freebsd target is *not* affected.
The former `ubuntu:19.10` reached EOL in July, 2020, whereas
`ubuntu:20.04` is an LTS release supported until 2025.
These are non-dist CI images, so the impact should be low.
Dist build manifest
This PR makes two changes that should remove a significant chunk of the time spent in our release process: cloning the `rust-lang/rust` monorepo, all its submodules, and building `bootstrap` to then invoke `build-manifest`:
* `build-manifest` doesn't rely on a clone of the monorepo being present anymore. The only remaining bit of information it fetched from it (the Rust version) is instead bundled in the binary.
* A new "component" is added, `build-manifest`. That component includes a prebuilt version of the tool, and it's *not* included in the Rustup manifest. This will allow `promote-release` to directly invoke the tool without interacting with our build system.
* The Linux x86_64 CI is changed to also build the component mentioned above. It's the only CI builder tasked to do so, and to cleanly support this a new `--include-default-paths` flag was added to `./x.py`.
* The `BUILD_MANIFEST_NUM_THREADS` environment variable is added to configure the number of threads at runtime.
This PR is best reviewed commit-by-commit.
r? `@Mark-Simulacrum`
ci: Fix riscv64gc linux test QEMU fault, plus doc link fix
Newer versions of the `qemu` package (used for riscv64gc-unknown-linux-gnu testing) don't work with the version of the RISC-V bootloader we were using. a4a0342cf5 bumps to a revision which should fix the problem.
e0b033e965 fixes a documentation failure I encountered while running the tests.
Promote aarch64-pc-windows-msvc to Tier 2 Development Platform
Adds a GitHub Actions CI build for `aarch64-pc-windows-msvc` via cross-compilation on an x86_64 host.
This promotes `aarch64-pc-windows-msvc` from a Tier 2 Compilation Target (std) to a Tier 2 Development Platform (std+rustc+cargo+tools).
Fixes#72881
r? `@pietroalbini`
This bumps the version of the bbl bootloader not to perform 64-bit
accesses to the PLIC. Doing so resulted in the QEMU test machine to fail
to boot:
bbl loader
../machine/mtrap.c:21: machine mode: unhandlable trap 7 @ 0x0000000080001f6e
Power off
Signed-off-by: Tom Eccles <tom.eccles@codethink.co.uk>
Stop running macOS builds on Azure Pipelines
The Infrastructure Team agreed to migrate macOS builds to GitHub Actions, so this commit stops running those builders on Azure Pipelines. The GitHub Actions runners are already configured to upload to the production bucket.
We can't still fully remove the Azure Pipelines configuration, as we still need to have that available until no stable releases run any of their builds on Azure Pipelines anymore. I'll open an issue to track fully removing our Azure Pipelines setup once the PR is merged.
r? @Mark-Simulacrum
Ensure that all LLVM components requested by tests are available on CI
Addresses https://github.com/rust-lang/rust/pull/75064#issuecomment-667722652
I used an environment variable because passing a command line option all the way from CI to compiletest would be just too much hassle for this task.
I added a new variable, but any of the already existing ones defined by CI could be used instead.
r? @Mark-Simulacrum
Previously we would have some platforms where LLVM was linked to rustc
statically, but to the LLVM tools dynamically. That meant we were distributing
two copies of LLVM: one as a separate dylib and one statically linked in to
librustc_driver.
Add aarch64-unknown-linux-musl host builds
This adds aarch64-unknown-linux-musl to the hosts list and adds the build to the dist-arm-linux builder as `@Mark-Simulacrum` suggested to me in Zulip. `@jyn514` requested to be mentioned 😄
I had to update the config for crosstool-ng as it had a prompt about the glibc version.
I ran `src/ci/docker/run.sh dist-arm-linux` to test it.
```
Build completed successfully in 1:31:50
Compile requests 8180
Compile requests executed 8135
Cache hits 287
Cache misses 7848
Cache timeouts 0
Cache read errors 0
Forced recaches 0
Cache write errors 0
Compilation failures 0
Cache errors 0
Non-cacheable compilations 0
Non-cacheable calls 36
Non-compilation calls 9
Unsupported compiler calls 0
Average cache write 0.000 s
Average cache read miss 6.389 s
Average cache read hit 0.000 s
Cache location Local disk: "/sccache"
Cache size 202 MiB
Max cache size 10 GiB
== clock drift check ==
local time: Sun Sep 6 19:30:17 UTC 2020
network time: Sun, 06 Sep 2020 19:30:17 GMT
== end clock drift check ==
```
Only errors were in miri due to struct fields being private (already been reported [here](https://github.com/rust-lang/rust/issues/76337))
Edit: Maybe it is helpful if I add that it is a working compiler
```sh
/rust-nightly-aarch64-unknown-linux-musl # ash install.sh
install: creating uninstall script at /usr/local/lib/rustlib/uninstall.sh
install: installing component 'rustc'
install: installing component 'cargo'
install: installing component 'rls-preview'
install: installing component 'rust-analyzer-preview'
install: installing component 'clippy-preview'
install: installing component 'rustfmt-preview'
install: installing component 'llvm-tools-preview'
install: installing component 'rust-analysis-aarch64-unknown-linux-musl'
install: installing component 'rust-std-aarch64-unknown-linux-musl'
install: WARNING: failed to run ldconfig. this may happen when not installing as root. run with --verbose to see the error
Rust is ready to roll.
/ # cat test.rs
fn main() { println!("hello world"); }
/ # rustc test.rs
/ # ./test
hello world
# file test
test: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, not stripped
```
Windows doesn't quite support dynamic linking to LLVM yet, but on other
platforms we do. In #76708, it was discovered that we dynamically link to LLVM
from the LLVM tools (e.g., rust-lld), so we need the shared LLVM library to link
against. That means that if we do not have a shared link to LLVM, and want LLVM
tools to work, we'd be shipping two copies of LLVM on all of these platforms:
one in librustc_driver and one in libLLVM.
Also introduce an error into rustbuild if we do end up configured for shared
linking on Windows.
