Before this commit toolstates.json was stored in /tmp and it wasn't
mounted outside the build container. That caused uploading the file in
the upload-artifacts task to fail, as the file was missing on the host.
Mounting /tmp/toolstates.json alone is not the best approach: if the
file is missing when the container is started the Docker engine will
create a *directory* named /tmp/toolstates.json.
The Docker issue could be solved by pre-creating an empty file named
/tmp/toolstates.json, but doing that could cause problems if bootstrap
fails to generate the file and the toolstate scripts receive an empty
JSON.
The approach I took in this commit is to instead mount a /tmp/toolstate
directory inside Docker, and create the toolstates.json file in it. That
also required a small bootstrap change to ensure the directory is
created if it's missing.
The hack was added because upstream msys2 broke the ca-certificates
package, but since then it has been fixed. This reverts CI to use the
upstream package.
Uploading the toolstate data for each commit will help our release
tooling understand which components are failing, to possibly skip
shipping broken tools to users.
We have a job in our CI (PR's x86_64-gnu-tools) that's supposed to run
only when a submodule is changed in the PR, and it works by having a
task at the start of the build that skips all the following tasks if the
condition isn't met.
Before this commit that task was gated with template parameters, which
is a unique feature of Azure Pipelines. To make our CI more generic this
commit switches the gate to use a simple environment variable plus a
condition, which should be supported on more CI providers.
With #65251 landed there's no need to build two LLVM backends and ship
them with rustc, every target we have now uses the same LLVM backend!
This removes the `src/llvm-emscripten` submodule and additionally
removes all support from rustbuild for building the emscripten LLVM
backend. Multiple codegen backend support is left in place for now, and
this is intended to be an easy 10-15 minute win on CI times by avoiding
having to build LLVM twice.
Upgrade GCC to 8.3.0, glibc to 1.17.0 and crosstool-ng to 1.24.0 for dist-armv7-linux
#62896 was caused by the usage of the GCC 5.2.0 toolchain, which was released back in 2015 and may have bugs affecting LLVM 9.
This PR upgrade GCC to 8.3.0 from 5.2.0, glibc from 1.16.0 to 1.17.0 and crosstool-ng to 1.24.0 only for dist-armv7-linux.
Fixes#62896
r? @alexcrichton
- Compatible with Emscripten 1.38.46-upstream or later upstream.
- Refactors the Emscripten target spec to share code with other wasm
targets.
- Replaces the old incorrect wasm32 C call ABI with the correct one,
preserving the old one as wasm32_bindgen_compat for wasm-bindgen
compatibility.
- Updates the varargs ABI used by Emscripten and deletes the old one.
- Removes the obsolete wasm32-experimental-emscripten target.
- Uses EMCC_CFLAGS on CI to avoid the timeout problems with #63649.
- Refactors the Emscripten target spec to share code with other wasm
targets.
- Replaces the incorrect wasm32 C call ABI with the old asmjs
version, which is correct for both wasm32 and JS.
- Updates the varargs ABI used by Emscripten and deletes the old one.
- Removes the obsolete wasm32-experimental-emscripten target.
- Temporarily makes Emscripten targets use panic=abort by default
because supporting unwinding will require an LLVM patch.
Make all alt builders produce parallel-enabled compilers
We're not quite ready to ship parallel compilers by default, but the alt
builders are not used too much (in theory), so we believe that shipping
a possibly-broken compiler there is not too problematic.
r? @nikomatsakis
We're not quite ready to ship parallel compilers by default, but the alt
builders are not used too much (in theory), so we believe that shipping
a possibly-broken compiler there is not too problematic.
azure: Convert Windows installations scripts to `bash`
Looks like `script`, which uses `cmd.exe`, doesn't have fail-fast
behavior and if a leading command fails the script doesn't actually fail
so long as the last command succeeds. We instead want the opposite
behavior where if any step fails the whole script fails.
I don't really know `cmd.exe` that well, nor powershell, so I've opted
to move everything to `bash` which should be a good common denominator
amongst all platforms to work with. Additionally I know that `set -e`
works to cause scripts to fail fast.
Closes#64551
Looks like `script`, which uses `cmd.exe`, doesn't have fail-fast
behavior and if a leading command fails the script doesn't actually fail
so long as the last command succeeds. We instead want the opposite
behavior where if any step fails the whole script fails.
I don't really know `cmd.exe` that well, nor powershell, so I've opted
to move everything to `bash` which should be a good common denominator
amongst all platforms to work with. Additionally I know that `set -e`
works to cause scripts to fail fast.
Note that some scripts remain as `script` since they don't appear to
work in` bash`. I'm not really sure why but I reorganized them slightly
to have the "meaty command" run at the end.
This commit changes our CI to use two separate IAM users to
authenticate with AWS:
* ci--rust-lang--rust--sccache: has access to the rust-lang-ci-sccache2
S3 bucket and its credentials are available during the whole build.
* ci--rust-lang--rust--upload: has access to the rust-lang-ci2 S3 bucket
and its credentials are available just during the upload step.
The new tokens are available in the `prod-credentials` library.
azure: Disable more LLVM/debug assertions in
This commit disables LLVM/debug assertions in our 5 slowest builders:
* i686-gnu
* i686-gnu-nopt
* i686-msvc-1
* i686-msvc-2
* x86_64-msvc-cargo
This is reducing the amount of test coverage for LLVM/debug assertions,
but we're just unfortunately running out of time on CI too many times.
Some test builds have shown that i686-gnu drops nearly an hour of CI
time by disabling these two assertions. Perhaps when we eventually get
4-core machines we can reenable these, but for now turn them off and
hook them up to the tracking issue at #59637 which will ideally be
repurposes to tracking all of these.
This commit disables LLVM/debug assertions in our 5 slowest builders:
* i686-gnu
* i686-gnu-nopt
* i686-msvc-1
* i686-msvc-2
* x86_64-msvc-cargo
This is reducing the amount of test coverage for LLVM/debug assertions,
but we're just unfortunately running out of time on CI too many times.
Some test builds have shown that i686-gnu drops nearly an hour of CI
time by disabling these two assertions. Perhaps when we eventually get
4-core machines we can reenable these, but for now turn them off and
hook them up to the tracking issue at #59637 which will ideally be
repurposes to tracking all of these.
GitHub only allows people explicitly listed as collaborators on the
repository or who commented on the issue/PR to be assignees, failing to
create the issue if non-assignable people are assigned.
This adds an extra check on CI to make sure all the people listed as
tool maintainers can be assigned to toolstate issues. The check won't be
executed on PR builds due to the lack of a valid token.
ci: move linkcheck from mingw-2 to mingw-1
Running UI tests now takes a huge amount of time on mingw builders
(between 40 and 50 minutes), with mingw-1 builders taking even an hour
less to finish than mingw-2. This PR moves linkcheck from mingw-2 to
mingw-1, removing between 10 and 20 minutes of runtime on the -2
builders.
r? @alexcrichton
Running UI tests now takes a huge amount of time on mingw builders
(between 40 and 50 minutes), with mingw-1 builders taking even an hour
less to finish than mingw-2. This PR moves linkcheck from mingw-2 to
mingw-1, removing between 10 and 20 minutes of runtime on the -2
builders.
ci: add a check for clock drift
Recently we encountered multiple spurious failures where the crates.io
certificate was reported as expired, even though it's currently due to
expire in a few months. This adds some code to our CI to check for clock
drifts, to possibly find the cause or rule out a bad VM clock.
cc https://github.com/rust-lang/rust/issues/63510
r? @Mark-Simulacrum
Recently we encountered multiple spurious failures where the crates.io
certificate was reported as expired, even though it's currently due to
expire in a few months. This adds some code to our CI to check for clock
drifts, to possibly find the cause or rule out a bad VM clock.
Currently mirrors are stored in the rust-lang-ci2 S3 bucket along with
CI toolchains. This is problematic for multiple reasons:
- CI IAM credentials are allowed to both edit and delete those files.
A malicious user gaining access to those credentials would be able to
change our mirrored dependencies, possibly backdooring the compiler.
- Contents of the rust-lang-ci2 bucket are disposable except for the
mirrors' content. When we implement backups for S3 buckets we'd have
to replicate just that part of the bucket, complicating the backup
logic and increasing the chance of mistakes. A standalone bucket will
be way easier to backup.
This commit switches our CI to use the new rust-lang-ci-mirrors bucket.
Added support for armv7-unknown-linux-gnueabi/musleabi
Fixes#63101
Some things that are not done and I hope someone can help me with:
* During the ci build of `armv7-unknown-linux-gnueabi` `openssl` must be built (to build cargo) but `openssl` does not yet support this target. This feels slightly like a chicken-and-egg problem, any feedback is welcome.
* Should I add any tests for any of these targets?
ci: fix toolstate not pushing data for Linux
A recent commit modified toolstate to only push updated data when the `TOOLSTATE_PUBLISH` environment variable is present. This worked fine on Windows but failed on Linux, since Linux jobs run inside Docker containers and the variable wasn't forwarded inside it.
This changes the Docker startup code to set the `TOOLSTATE_PUBLISH` enviornment variable inside the container if it's present outside.
r? @alexcrichton
fixes https://github.com/rust-lang/rust/issues/63190
A recent commit modified toolstate to only push updated data when the
TOOLSTATE_PUBLISH environment variable is present. This worked fine on
Windows but failed on Linux, since Linux jobs run inside Docker
containers and the variable wasn't forwarded inside it.
This changes the Docker startup code to set the TOOLSTATE_PUBLISH
enviornment variable inside the container if it's present outside.
ci: download awscli from our mirror
This fixes multiple network issues we had when downloading awscli from PyPI on Azure Pipelines by vendoring awscli itself and its dependencies in our S3 bucket. Instructions on how to update the cache are present at the top of `src/ci/install-awscli.sh`.
r? @alexcrichton or @Mark-Simulacrum
fixes#62967
This fixes multiple network issues we had when downloading awscli from
PyPI on Azure Pipelines by vendoring awscli itself and its dependencies
in our S3 bucket. Instructions on how to update the cache are present at
the top of src/ci/install-awscli.sh
ci: gate toolstate repo pushes on the TOOLSTATE_PUBLISH envvar
This PR fixes toolstate failing to push on the LinuxTools PR builder by gating the pushes on the new `TOOLSTATE_PUBLISH` environment variable, which is set on prod credentials but not on the PR ones. The old code checked whether the access token was set, but that doesn't work due to an Azure quirk.
For a bit of background, secret environment variables are not available by default, but each step needs to explicitly declare which secret vars to load:
```yaml
- bash: echo foo
env:
SECRET_VAR: $(SECRET_VAR)
```
This works fine when the variable is present but when it's missing, instead of setting `SECRET_VAR` to an empty string or just not setting it at all, Azure Pipelines puts the literal `$(SECRET_VAR)` as the content, which completly breaks the old check we had. I tried almost every thing to make this work in a sensible way, and the only conclusion I reached is to set the variable at the top level with the runtime expression evaluation syntax, which sets the variable to an empty string if missing:
```yaml
# At the top:
variables:
- name: MAYBE_SECRET_VAR
value: $[ variables.MAYBE_SECRET_VAR ]
# In the step:
- bash: echo foo
env:
SECRET_VAR: $(MAYBE_SECRET_VAR)
```
While that *could've worked* it was ugly and messy, so I just opted to add yet another non-secret variable.
r? @alexcrichton
fixes#62811
Unfortunately due to an Azure quirk the TOOLSTATE_REPO_ACCESS_TOKEN is
not suitable to gate whether to push new commits to the repo, as if it's
not defined on the Azure side it will actually be set to the literal
`$(TOOLSTATE_REPO_ACCESS_TOKEN)`, which screws everything up.
This instead adds another, non-secret environment variable to gate
publishing: TOOLSTATE_PUBLISH. As non-secret environment variables
behave correctly this fixes the issue.
Require a value for configure --debuginfo-level
In `configure.py`, using the `o` function creates an enable/disable
boolean setting, and writes `true` or `false` in `config.toml`. However,
rustbuild is expecting to parse a `u32` debuginfo level. We can change
to the `v` function to have the options require a value.
In `configure.py`, using the `o` function creates an enable/disable
boolean setting, and writes `true` or `false` in `config.toml`. However,
rustbuild is expecting to parse a `u32` debuginfo level. We can change
to the `v` function to have the options require a value.
docutils 0.15, a dependency of awscli, broke our CI since it's not
compatible with Python 2 due to a bug. This pins all the dependencies of
awscli with docutils 0.14, to make sure this kind of regressions doesn't
happen again.
This commit updates some of our assorted Azure/CI configuration to
prepare for some 4-core machines coming online. We're still in the
process of performance testing them to get final numbers, but some
changes are worth landing ahead of this. The updates here are:
* Use `C:/` instead of `D:/` for submodule checkout since it should have
plenty of space and the 4-core machines won't have `D:/`
* Update `lzma-sys` to 0.1.14 which has support for VS2019, where 0.1.10
doesn't.
* Update `src/ci/docker/run.sh` to work when it itself is running inside
of a docker container (see the comment in the file for more info)
* Print step timings on the `try` branch in addition to the `auto`
branch in. The logs there should be seen by similarly many humans (not
many) and can be useful for performance analysis after a `try` build
runs.
* Install the WIX and InnoSetup tools manually on Windows instead of
relying on pre-installed copies on the VM. This gives us more control
over what's being used on the Azure cloud right now (we control the
version) and in the 4-core machines these won't be pre-installed. Note
that on AppVeyor we actually already were installing InnoSetup, we
just didn't carry that over on Azure!
Update to LLVM 9 trunk
Following the preparatory changes in #62474, this updates the LLVM submodule to https://github.com/rust-lang/llvm-project/tree/rustc/9.0-2019-07-12 and:
* Changes the LLVM Rust bindings to account for the new SubtargetSubTypeKV.
* Adjusts a codegen test for the new form of the byval attribute that takes a type.
* Makes a PGO codegen test more liberal with regard to order and linkage.
* Builds InstrProfilingPlatformWindows.c as part of libprofiler_builtins.
* Moves registration of additional passes (in particular sanitizers) to the end of the module pass manager.
* Disables LLDB on builders.
r? @alexcrichton
Now that we've fully moved to Azure Pipelines and bors has been updated
to only gate on Azure this commit removes the remaining Travis/AppVeyor
support contained in this repository. Most of the deletions here are
related to producing better output on Travis by folding certain
sections. This isn't supported by Azure so there's no need to keep it
around, and if Azure ever adds support we can always add it back!
Update musl to 1.1.22
This is the latest available version. I noticed Rust wasn't using the
latest version when I attempted to have Cargo link object files produced
outside of Cargo / Rust's toolchain and was getting missing symbol
errors. Those missing symbols were added in 1.1.21 and 1.1.22.
I'm not fully sure of the implications of this change or how
comprehensive the test coverage is. Upstream changes in 1.1.21 and
1.1.22 can be found at
https://git.musl-libc.org/cgit/musl/tree/WHATSNEW?h=v1.1.22&id=e97681d6f2c44bf5fa9ecdd30607cb63c780062e#n1989.
The PR builder on Azure currently takes 2.5h which is a bit long, so
this commit disables debug assertions and llvm assertions in an attempt
to speed up that builder and have PR builds come back a bit more
quickly. Other builders continue to enable debug assertions and test the
compiler there.
Limit dylib symbols
This makes `windows-gnu` match the behavior of `windows-msvc`. It probably doesn't make sense to export these symbols on other platforms either.
Currently just run it through its paces but don't actually push to
official locations. Instead let's just push to a separate fork (mine) as
well as open issues in a separate fork (mine). Make sure that people
aren't pinged for these issues as well!
This should hopefully ensure that everything is working on Azure and
give us a chance to work through any issues that come up.
ci: Disable LLVM/debug assertions for distcheck
The purpose of distcheck is to test `./x.py test` from a tarball, not to
test that all assertions pass all the time. These assertions are largely
just redundant with other builders, so skip the assertions for now and
save a good chunk of time on CI.
cc #61185
ci: Collect CPU usage statistics on Azure
This commit adds a script which we'll execute on Azure Pipelines which
is intended to run in the background and passively collect CPU usage
statistics for our builders. The intention here is that we can use this
information over time to diagnose issues with builders, see where we can
optimize our build, fix parallelism issues, etc. This might not end up
being too useful in the long run but it's data we've wanted to collect
for quite some time now, so here's a stab at it!
Comments about how this is intended to work can be found in the python
script used here to collect CPU usage statistics.
Closes#48828
This commit adds a script which we'll execute on Azure Pipelines which
is intended to run in the background and passively collect CPU usage
statistics for our builders. The intention here is that we can use this
information over time to diagnose issues with builders, see where we can
optimize our build, fix parallelism issues, etc. This might not end up
being too useful in the long run but it's data we've wanted to collect
for quite some time now, so here's a stab at it!
Comments about how this is intended to work can be found in the python
script used here to collect CPU usage statistics.
Closes#48828
checktools: unify grepping the TOOLSTATE file
The file was grepped twice but in a different way. This unifies the code to make sure it is consistent. Or were these deliberately not doing the same thing? That seems strange though.
I wouldn't know how to test these changes.
The purpose of distcheck is to test `./x.py test` from a tarball, not to
test that all assertions pass all the time. These assertions are largely
just redundant with other builders, so skip the assertions for now and
save a good chunk of time on CI.
cc #61185
This shaves of 50 minutes of cycle time on Azure and will likely also
save a significant chunk of time on Travis. The assertions here aren't
really buying us much over other builders with assertions already
enabled, so let's disable them for this builder.
cc #61185
This was accidentally regressed in #60777 by accident, and we've stopped
printing out step timings on AppVeyor recently reducing the ability for
us to track build times over time!
Update musl-cross-make to 0.9.8
This version uses musl 1.1.22 and GCC 6.4.0 by default. It also
contains support for newer binutils and GCC versions, should we
want to bump those as well. But I'm purposefully limiting this
patch in order to reduce the surface area for controversy.
ci: Favor SCRIPT instead of RUST_CHECK_TARGET
Since #61212 we've been timing out on OSX, and this looks to be because
we're building tools like Cargo and the RLS twice instead of once. This
turns out to be a slight bug in our configuration. CI builders using the
`RUST_CHECK_TARGET` directive actually execute `make all` just before
their acual target. In `make all` we're building a stage2 cargo, and
then in `make dist` we're building a stage1 cargo.
Other builders use `SCRIPT` which provides explicit control over what
`x.py` script, for example, is used to execute the build. This moves
almost all targets to using `SCRIPT` to ensure that we're explicitly
specifying what's being built where. Additionally this updates the logic
of `RUST_CHECK_TARGET` to remove the pre-flight tidy as well as the
pre-flight `make all`. The system LLVM builder (run on PRs) now
explicitly runs tidy first and then runs the rest of the test suite.
Since #61212 we've been timing out on OSX, and this looks to be because
we're building tools like Cargo and the RLS twice instead of once. This
turns out to be a slight bug in our configuration. CI builders using the
`RUST_CHECK_TARGET` directive actually execute `make all` just before
their acual target. In `make all` we're building a stage2 cargo, and
then in `make dist` we're building a stage1 cargo.
Other builders use `SCRIPT` which provides explicit control over what
`x.py` script, for example, is used to execute the build. This moves
almost all targets to using `SCRIPT` to ensure that we're explicitly
specifying what's being built where. Additionally this updates the logic
of `RUST_CHECK_TARGET` to remove the pre-flight tidy as well as the
pre-flight `make all`. The system LLVM builder (run on PRs) now
explicitly runs tidy first and then runs the rest of the test suite.
Update all s3 URLs used on CI with subdomains
Ensure that they're all forwards-compatible with AWS updates happening
next year by ensuring the bucket name shows up in the domain name.
Closes#61168
This version uses musl 1.1.22 and GCC 6.4.0 by default. It also
contains support for newer binutils and GCC versions, should we
want to bump those as well. But I'm purposefully limiting this
patch in order to reduce the surface area for controversy.
This is the latest available version. I noticed Rust wasn't using the
latest version when I attempted to have Cargo link object files produced
outside of Cargo / Rust's toolchain and was getting missing symbol
errors. Those missing symbols were added in 1.1.21 and 1.1.22.
I'm not fully sure of the implications of this change or how
comprehensive the test coverage is. Upstream changes in 1.1.21 and
1.1.22 can be found at
https://git.musl-libc.org/cgit/musl/tree/WHATSNEW?h=v1.1.22&id=e97681d6f2c44bf5fa9ecdd30607cb63c780062e#n1989.
Add Azure Pipelines configuration
Huge thanks to ＠johnterickson and ＠willsmythe for writing the initial config! ❤️
I applied some changes to the initial config and disabled most of the builders since we're not going to run all of them during the initial step for the evaluation.
[More details about our plans for the Azure Pipelines evaluation.](https://internals.rust-lang.org/t/update-on-the-ci-investigation/10056)
r? ＠alexcrichton ＠kennytm
cc ＠rust-lang/infra ＠ethomson ＠rylev
Solaris CI: Build with dilos2 stable
dilos2-testing has problems since the last repository update, so get the packages from dilos2 stable.
Fixes#61022.
This builder already is very close to the 2 hour mark and the debug
assertions aren't really buying us anything over what all the other
containers are enabling, so let's disable them for this slow builder.
* Clean up the `install-windows-build-deps.yml` file and add some more
comments where appropriate.
* Add some comments to `run.yml`
* Don't fast path the `rustfmt` submodule, but we'll take care of that
later if necessary.
Update books
## nomicon
1 commits in fb29b147be4d9a1f8e24aba753a7e1de537abf61..c656171b749b7307f21371dd0d3278efee5573b8
2019-04-22 19:10:29 -0400 to 2019-04-25 15:31:26 -0400
- Lifetimes: Updates to incorporate NLL (rust-lang-nursery/nomicon#101)
## reference
5 commits in 2a2de9ce095979978ad7b582daecf94e4070b916..862b669c395822bb0938781d74f860e5762ad4fb
2019-04-22 10:25:52 -0700 to 2019-05-04 23:41:35 -0700
- Typo (rust-lang-nursery/reference#606)
- Added missing ? to Generics from InherentImpl and TraitImpl (rust-lang-nursery/reference#604)
- Add missing ( to ExternalFunctionItem (rust-lang-nursery/reference#603)
- Remove unneeded | from AssignmentExpression (rust-lang-nursery/reference#601)
- Remove unneeded ( from TypePathSegment (rust-lang-nursery/reference#602)
## book
6 commits in db919bc6bb9071566e9c4f05053672133eaac33e..29fe982990e43b9367be0ff47abc82fb2123fd03
2019-04-15 20:11:03 -0400 to 2019-05-15 17:48:40 -0400
- Ignore a non-compiling test listing and add code to fix a test listing
- Remove nostarch snapshots I've checked in layout
- Reword error messages to maybe not need to wrap lines in print
- This example doesn't compile but wasn't marked as such
- Update install instructions for VS 2019 (rust-lang/book#1923)
- Switch IRC to Discord
## rust-by-example
9 commits in 1ff0f8e018838a710ebc0cc1a7bf74ebe73ad9f1..811c697b232c611ed754d279ed20643a0c4096f6
2019-04-15 08:15:32 -0300 to 2019-04-28 18:56:42 -0300
- Fix typo in dsl.md (rust-lang/rust-by-example#1187)
- File read lines (rust-lang/rust-by-example#1186)
- For https://github.com/rust-lang/rust-by-example/issues/1184closesrust-lang/rust-by-example#1184 (rust-lang/rust-by-example#1185)
- Link to Reference for macro_rules designators (rust-lang/rust-by-example#1182)
- Improve section Meta/Docs (rust-lang/rust-by-example#1183)
- Small improvements to various files (rust-lang/rust-by-example#1173)
- 19.2 Vectors Error in Code Example (rust-lang/rust-by-example#1178)
- For https://github.com/rust-lang/rust-by-example/issues/1175 (rust-lang/rust-by-example#1176)
- For https://github.com/rust-lang/rust-by-example/issues/1179 (rust-lang/rust-by-example#1180)
## rustc-guide
12 commits in 99e1b1d53656be08654df399fc200584aebb50e4..3cb727b62b953d59b4360d39aa68b6dc8f157655
2019-04-20 09:57:54 -0500 to 2019-05-07 09:53:32 -0500
- Fix typo, 'which' repeated twice
- [canonicalization] fix result canonicalization example (rust-lang/rustc-guide#304)
- Rename to RUSTC_LOG
- Added mention of universal ctags
- Fix link in walkthrough
- Remove IRC from discussion chats
- Bring the updating LLVM guide up to date
- use nightly rust for ci
- Fixed broken chalk links
- Add documentation for two-phase borrows
- Explain new powers of the `treat-err-as-bug` flag
- Update lowering-module test case
## edition-guide
3 commits in c413d42a207bd082f801ec0137c31b71e4bfed4c..581c6cccfaf995394ea9dcac362dc8e731c18558
2019-04-22 01:14:56 +0200 to 2019-05-06 12:47:44 -0700
- Fix typo in controlling-panics-with-std-panic.md (rust-lang-nursery/edition-guide#158)
- Fix links for book editions. (rust-lang-nursery/edition-guide#149)
- Update now that NLL is enabled in 2015. (rust-lang-nursery/edition-guide#157)
## embedded-book
3 commits in de3d55f521e657863df45260ebbca1b10527f662..9858872bd1b7dbba5ec27dc30d34eba00acd7ef9
2019-04-22 12:58:28 +0000 to 2019-05-02 18:56:54 +0000
- Update linux.md (rust-embedded/book#167)
- Clarify list of available targets for installation (rust-embedded/book#165)
- minor grammar fix (rust-embedded/book#188)
Omit the vendor component in the WASI triple
This renames wasm32-unknown-wasi to wasm32-wasi, omitting the vendor
component. This follows aarch64-linux-android, x86_64-fuchsia, and others in
omitting the vendor field, which has the advantage of aligning with the
[multiarch tuple](https://wiki.debian.org/Multiarch/Tuples), and of being
less noisy.
r? @alexcrichton
This renames wasm32-unknown-wasi to wasm32-wasi, omitting the vendor
component. This follows aarch64-linux-android, x86_64-fuchsia, and others in
omitting the vendor field, which has the advantage of aligning with the
[multiarch tuple](https://wiki.debian.org/Multiarch/Tuples), and of being
less noisy.
The canonical repository on fuchsia.googlesource.com is not accessible
anymore, neither for anonymous access nor logged in access. This commit
switches our CI to fetch the repository from our own mirror.
ci: use a custom android sdk manager with pinning and mirroring
Google's own sdkmanager has two issues that make it unsuitable for us:
* Mirroring has to be done manually, which is annoying because we need to figure out on our own all the URLs to copy (I couldn't find any documentation when building this PR, had to use mitmproxy).
* There is no support for pinning, which means an update on Google's side can break our CI, as it happened multiple times.
This PR replaces all our usage of sdkmanager with a custom Python script which mimics its behavior, but with the two issues fixes.
sdkmanager's logic for installing packages is thankfully very simple: the package name (like `system-images;android-18;default;armeabi-v7a`) is the directory where the package should live (with `;` replaced with `/`), so to install a package we only need to extract its contents in the right directory.
r? @alexcrichton
cc @kennytm
fixes https://github.com/rust-lang/rust/issues/59778
musl: do not compress debug section
This should be beta nominated.
Fixes https://github.com/rust-lang/rust/issues/59411 (this time for real).
Test with `DEPLOY=1 ./src/ci/docker/run.sh dist-x86_64-musl`, without `DEPLOY=1` libs are built without debuginfo.
r? @alexcrichton
ci: pin android emulator to 28.0.23
Apparently Android Emulator 28.0.25 (the latest one) is broken for us, and `sdkmanager` doesn't have a way to pin a package to a specific version. This PR stops downloading the emulator from `sdkmanager`, downloading it manually instead.
Tested this locally and confirmed it works. I'll open an issue on Google's issue tracker later today.
Fixes https://github.com/rust-lang/rust/issues/59757
r? @kennytm
wasi: Use shared API for preopened fds
This commit updates the wasi target with supported added in
CraneStation/wasi-sysroot#10. That function allows both C and Rust to
cooperate in how preopened files are managed, enabling us to learn about
propened files through the same interface. The `open_parent` function in
the wasi `fs` module was updated to avoid its own initialization of a
global preopened map and instead delegate to libc to perform this
functionality.
This should both be more robust into the future in terms of handling
path logic as well as ensuring the propened map is correctly set up at
process boot time. This does currently require some unfortunate
allocations on our side, but if that becomes an issue we can always
paper over those in time!
This commit updates the wasi target with supported added in
CraneStation/wasi-sysroot#10. That function allows both C and Rust to
cooperate in how preopened files are managed, enabling us to learn about
propened files through the same interface. The `open_parent` function in
the wasi `fs` module was updated to avoid its own initialization of a
global preopened map and instead delegate to libc to perform this
functionality.
This should both be more robust into the future in terms of handling
path logic as well as ensuring the propened map is correctly set up at
process boot time. This does currently require some unfortunate
allocations on our side, but if that becomes an issue we can always
paper over those in time!
This commit adds the Armv8-M Baseline and Armv8-M Mainline with
FPU targets in the list of targets that
get their dist components built. It also update the build-manifest
so that this target gets also its dist components uploaded.
This commit adds a new wasm32-based target distributed through rustup,
supported in the standard library, and implemented in the compiler. The
`wasm32-unknown-wasi` target is intended to be a WebAssembly target
which matches the [WASI proposal recently announced.][LINK]. In summary
the WASI target is an effort to define a standard set of syscalls for
WebAssembly modules, allowing WebAssembly modules to not only be
portable across architectures but also be portable across environments
implementing this standard set of system calls.
The wasi target in libstd is still somewhat bare bones. This PR does not
fill out the filesystem, networking, threads, etc. Instead it only
provides the most basic of integration with the wasi syscalls, enabling
features like:
* `Instant::now` and `SystemTime::now` work
* `env::args` is hooked up
* `env::vars` will look up environment variables
* `println!` will print to standard out
* `process::{exit, abort}` should be hooked up appropriately
None of these APIs can work natively on the `wasm32-unknown-unknown`
target, but with the assumption of the WASI set of syscalls we're able
to provide implementations of these syscalls that engines can implement.
Currently the primary engine implementing wasi is [wasmtime], but more
will surely emerge!
In terms of future development of libstd, I think this is something
we'll probably want to discuss. The purpose of the WASI target is to
provide a standardized set of syscalls, but it's *also* to provide a
standard C sysroot for compiling C/C++ programs. This means it's
intended that functions like `read` and `write` are implemented for this
target with a relatively standard definition and implementation. It's
unclear, therefore, how we want to expose file descriptors and how we'll
want to implement system primitives. For example should `std::fs::File`
have a libc-based file descriptor underneath it? The raw wasi file
descriptor? We'll see! Currently these details are all intentionally
hidden and things we can change over time.
A `WasiFd` sample struct was added to the standard library as part of
this commit, but it's not currently used. It shows how all the wasi
syscalls could be ergonomically bound in Rust, and they offer a possible
implementation of primitives like `std::fs::File` if we bind wasi file
descriptors exactly.
Apart from the standard library, there's also the matter of how this
target is integrated with respect to its C standard library. The
reference sysroot, for example, provides managment of standard unix file
descriptors and also standard APIs like `open` (as opposed to the
relative `openat` inspiration for the wasi ssycalls). Currently the
standard library relies on the C sysroot symbols for operations such as
environment management, process exit, and `read`/`write` of stdio fds.
We want these operations in Rust to be interoperable with C if they're
used in the same process. Put another way, if Rust and C are linked into
the same WebAssembly binary they should work together, but that requires
that the same C standard library is used.
We also, however, want the `wasm32-unknown-wasi` target to be
usable-by-default with the Rust compiler without requiring a separate
toolchain to get downloaded and configured. With that in mind, there's
two modes of operation for the `wasm32-unknown-wasi` target:
1. By default the C standard library is statically provided inside of
`liblibc.rlib` distributed as part of the sysroot. This means that
you can `rustc foo.wasm --target wasm32-unknown-unknown` and you're
good to go, a fully workable wasi binary pops out. This is
incompatible with linking in C code, however, which may be compiled
against a different sysroot than the Rust code was previously
compiled against. In this mode the default of `rust-lld` is used to
link binaries.
2. For linking with C code, the `-C target-feature=-crt-static` flag
needs to be passed. This takes inspiration from the musl target for
this flag, but the idea is that you're no longer using the provided
static C runtime, but rather one will be provided externally. This
flag is intended to also get coupled with an external `clang`
compiler configured with its own sysroot. Therefore you'll typically
use this flag with `-C linker=/path/to/clang-script-wrapper`. Using
this mode the Rust code will continue to reference standard C
symbols, but the definition will be pulled in by the linker configured.
Alright so that's all the current state of this PR. I suspect we'll
definitely want to discuss this before landing of course! This PR is
coupled with libc changes as well which I'll be posting shortly.
[LINK]:
[wasmtime]:
This writes an extra `dist/image-$image.txt` which contains the S3 URL
of the cached image and the `sha256` digest of the docker entry point.
This will be uploaded with the rest of the deployed artifacts in the
Travis `after_success` script.
Introduce assembly tests suite
The change introduces a new test suite - **Assembly** tests. The motivation behind this is an ability to perform end-to-end codegen testing with LLVM backend. Turned out, NVPTX backend sometimes missing common Rust features (`i128` and libcalls in the past, and still full atomics support) due to different reasons.
Prior to this change, basic NVPTX assembly tests were implemented within `run-make` suite. Now, it's easier to write additional and maintain existing tests for the target.
cc @gnzlbg @peterhj
cc @eddyb I adjusted mangling scheme expectation, so there is no need to change the tests for #57967
Calculate Docker cache hash precisely from Dockerfile's dependencies
#58549 changed the Docker cache calculation to include every file under `src/ci/docker`, so that when files under `dist-x86_64-linux` is changed, its dependent image `dist-i686-linux` will also be rebuilt.
However, this ultraconservative solution caused the `dist-i686-linux` to be rebuilt every time an irrelevant Dockerfile (e.g. the PowerPC ones) is changed, which increases the building time beyond 3 hours and forcing a spurious but expected failure.
This commit instead parses the Dockerfile itself and look for the actual dependencies. The scripts needs to be copied into the Docker image, which must be done with the COPY command, so we just need to find all lines with a COPY command and add the source file into the hash calculator.
Note: this script only handles single-lined COPY command in the form `COPY src1 src2 src3 dst`, since these are the only variant used inside this repository.
Track embedded-book in the toolstate
The embedded book was tested in the tools job but the test result was never published. This PR adds maintainer information of embedded-book. This PR also requires the next update to embedded-book to pass the all tests, currently its state is test-fail.
rust-lang-nursery/rust-toolstate#10 should be merged before this PR.
[CI] Update binutils for powerpc64 and powerpc64le
Cargo powerpc64 and powerpc64le are seeing `SIGILL` crashes in openssl,
which was found to be a linking problem, fixed by newer binutils. See
<https://github.com/rust-lang/rust/issues/57345#issuecomment-462094555>
For powerpc64 we're using crosstool-ng, which doesn't offer a newer
binutils version, but we can just compile it separately. On powerpc64le
we're already building binutils. Both are now updated to binutils 2.32.
Closesrust-lang/cargo#6320Closesrust-lang/rust#57345Closesrust-lang/rustup.rs#1620
r? @alexcrichton
`src/ci/docker`, so that when files under `dist-x86_64-linux` is changed,
its dependent image `dist-i686-linux` will also be rebuilt.
However, this ultraconservative solution caused the `dist-i686-linux` to
be rebuilt every time an irrelevant Dockerfile (e.g. the PowerPC ones) is
changed, which increases the building time beyond 3 hours and forcing
a spurious but expected failure.
This commit instead parses the Dockerfile itself and look for the actual
dependencies. The scripts needs to be copied into the Docker image, which
must be done with the COPY command, so we just need to find all lines with
a COPY command and add the source file into the hash calculator.
Note: this script only handles single-lined COPY command in the form
`COPY src1 src2 src3 dst`, since these are the only variant used inside
this repository.
The musl-target doesn't automatically disable static linking of musl when building a dylib, and then complains it can't build a dylib.
As a workaround, disable static linking via RUSTFLAGS, to see how far the build gets.
The proper fix is to have rustc figure that out automagically.
This removes some tests from the i686-gnu job. This job clocks in at 2hr 56min, and removing these should cut about 10 to 15 minutes, giving a little more breathing room. I suspect these don't need to be tested on every platform.
We'll use this as a temporary measure to get an LLVM update landed, but
we'll have to go through and update images later to make sure they've
got the right toolchains.
ci: fix docker cache hash collision
#58416 uncovered a bug in our caching for docker images: if the image `foo` pulls files from the image `bar` and a file in `bar` changed, the hash of `foo` will be the same even though it should be different. In that PR's case, `dist-i686-linux` pulls scripts from `dist-x86_64-linux`, and the PR only changed those scripts, causing an hash collision for `dist-i686-linux`.
We have to fix this, since the image will be rebuilt every time bors switches from testing master to testing beta/stable (and when it switches back), making CI way more painful than it currently is.
The approach used by this PR is to just include all the files in `src/ci/docker` in the hash. It's a bit heavy-handed and it will cause a rebuild of all the images every time a single image changes, but it's the best I can think of.
r? @Mark-Simulacrum
cc @alexcrichton @kennytm
Before this commit the hash used to cache docker images was calculated
from the image's files and the files in the scripts/ directory. This
worked fine when all the files used by an image were in those
directories, but some images pull files from other images, causing hash
collisions in some cases.
This commit changes the hash to include the files of all the docker
images, causing a rebuild of all the images when a single one changes.
That's a bit heavy-handed, but we have no way to track which files an
image pulls in and hash collisions are really painful to deal with.
[CI] Update GCC in the dist-linux builders
Upstream LLVM is planning to raise their minimum toolchain requirements, so they may start using C++14 features. This new policy has already landed in the form of a "soft" error.
http://lists.llvm.org/pipermail/llvm-dev/2019-February/130117.html
For GCC, they will require at least version 5.1. This PR moves our crosstool-ng builders to their max GCC 5.2, with a few small patches to fix compatibility. The dist-x86_64-linux builder is updated to GCC 5.5 and LLVM/Clang 8.0.0-rc2, which also affects dist-i686-linux sharing the same scripts.
r? @alexcrichton
The Ubuntu keyserver is more reliable than the MIT PGP server, which is
prone to going down. This commit also explicitly uses port 80 on the
keyserver for reasons outlined in #57844.
NVPTX target specification
This change adds a built-in `nvptx64-nvidia-cuda` GPGPU no-std target specification and a basic PTX assembly smoke tests.
The approach is taken here and the target spec is based on `ptx-linker`, a project started about 1.5 years ago. Key feature: bitcode object files being linked with LTO into the final module on the linker's side.
Prior to this change, the linker used a `ld` linker-flavor, but I think, having the special CLI convention is a more reliable way.
Questions about further progress on reliable CUDA workflow with Rust:
1. Is it possible to create a test suite `codegen-asm` to verify end-to-end integration with LLVM backend?
1. How would it be better to organise no-std `compile-fail` tests: add `#![no_std]` where possible and mark others as `ignore-nvptx` directive, or alternatively, introduce `compile-fail-no-std` test suite?
1. Can we have the `ptx-linker` eventually be integrated as `rls` or `clippy`? Hopefully, this should allow to statically link against LLVM used in Rust and get rid of the [current hacky solution](https://github.com/denzp/rustc-llvm-proxy).
1. Am I missing some methods from `rustc_codegen_ssa:🔙:linker::Linker` that can be useful for bitcode-only linking?
Currently, there are no major public CUDA projects written in Rust I'm aware of, but I'm expecting to have a built-in target will create a solid foundation for further experiments and awesome crates.
Related to #38789Fixes#38787Fixes#38786
compiletest: Support opt-in Clang-based run-make tests and use them for testing xLTO.
Some cross-language run-make tests need a Clang compiler that matches the LLVM version of `rustc`. Since such a compiler usually isn't available these tests (marked with the `needs-matching-clang`
directive) are ignored by default.
For some CI jobs we do need these tests to run unconditionally though. In order to support this a `--force-clang-based-tests` flag is added to compiletest. If this flag is specified, `compiletest` will fail if it can't detect an appropriate version of Clang.
@rust-lang/infra The PR doesn't yet enable the tests yet. Do you have any recommendation for which jobs to enable them?
cc #57438
r? @alexcrichton
