Required moving all fulldeps tests depending on `rand` to different locations as
now there's multiple `rand` crates that can't be implicitly linked against.
rustc: Migrate to `termcolor` crate from `term`
This crate moves the compiler's error reporting to using the `termcolor` crate
from crates.io. Previously rustc used a super-old version of the `term` crate
in-tree which is basically unmaintained at this point, but Cargo has been using
`termcolor` for some time now and tools like `rg` are using `termcolor` as well,
so it seems like a good strategy to take!
Note that the `term` crate remains in-tree for libtest. Changing libtest will be
a bit tricky due to how the build works, but we can always tackle that later.
cc #45728
This crate moves the compiler's error reporting to using the `termcolor` crate
from crates.io. Previously rustc used a super-old version of the `term` crate
in-tree which is basically unmaintained at this point, but Cargo has been using
`termcolor` for some time now and tools like `rg` are using `termcolor` as well,
so it seems like a good strategy to take!
Note that the `term` crate remains in-tree for libtest. Changing libtest will be
a bit tricky due to how the build works, but we can always tackle that later.
cc #45728
It looks like this cuts down on the number of dependencies in env_logger and
notably cuts out a difference between a shared dependency of rls/cargo. My goal
here is to ensure that when we compile the RLS/Cargo on CI we only compile Cargo
once, and this is one step towards that!
This commit imports the LLD project from LLVM to serve as the default linker for
the `wasm32-unknown-unknown` target. The `binaryen` submoule is consequently
removed along with "binaryen linker" support in rustc.
Moving to LLD brings with it a number of benefits for wasm code:
* LLD is itself an actual linker, so there's no need to compile all wasm code
with LTO any more. As a result builds should be *much* speedier as LTO is no
longer forcibly enabled for all builds of the wasm target.
* LLD is quickly becoming an "official solution" for linking wasm code together.
This, I believe at least, is intended to be the main supported linker for
native code and wasm moving forward. Picking up support early on should help
ensure that we can help LLD identify bugs and otherwise prove that it works
great for all our use cases!
* Improvements to the wasm toolchain are currently primarily focused around LLVM
and LLD (from what I can tell at least), so it's in general much better to be
on this bandwagon for bugfixes and new features.
* Historical "hacks" like `wasm-gc` will soon no longer be necessary, LLD
will [natively implement][gc] `--gc-sections` (better than `wasm-gc`!) which
means a postprocessor is no longer needed to show off Rust's "small wasm
binary size".
LLD is added in a pretty standard way to rustc right now. A new rustbuild target
was defined for building LLD, and this is executed when a compiler's sysroot is
being assembled. LLD is compiled against the LLVM that we've got in tree, which
means we're currently on the `release_60` branch, but this may get upgraded in
the near future!
LLD is placed into rustc's sysroot in a `bin` directory. This is similar to
where `gcc.exe` can be found on Windows. This directory is automatically added
to `PATH` whenever rustc executes the linker, allowing us to define a `WasmLd`
linker which implements the interface that `wasm-ld`, LLD's frontend, expects.
Like Emscripten the LLD target is currently only enabled for Tier 1 platforms,
notably OSX/Windows/Linux, and will need to be installed manually for compiling
to wasm on other platforms. LLD is by default turned off in rustbuild, and
requires a `config.toml` option to be enabled to turn it on.
Finally the unstable `#![wasm_import_memory]` attribute was also removed as LLD
has a native option for controlling this.
[gc]: https://reviews.llvm.org/D42511
These changes were meant to be in
2b18d8fe9dc05415a8e6b7cadf879c7f7ebe020a (rebased from
12a230562e), but I messed up the rebase a
bit as the file had been moved.
Is it really time? Have our months, no, *years* of suffering come to an end? Are we finally able to cast off the pall of Hoedown? The weight which has dragged us down for so long?
-----
So, timeline for those who need to catch up:
* Way back in December 2016, [we decided we wanted to switch out the markdown renderer](https://github.com/rust-lang/rust/issues/38400). However, this was put on hold because the build system at the time made it difficult to pull in dependencies from crates.io.
* A few months later, in March 2017, [the first PR was done, to switch out the renderers entirely](https://github.com/rust-lang/rust/pull/40338). The PR itself was fraught with CI and build system issues, but eventually landed.
* However, not all was well in the Rustdoc world. During the PR and shortly after, we noticed [some differences in the way the two parsers handled some things](https://github.com/rust-lang/rust/issues/40912), and some of these differences were major enough to break the docs for some crates.
* A couple weeks afterward, [Hoedown was put back in](https://github.com/rust-lang/rust/pull/41290), at this point just to catch tests that Pulldown was "spuriously" running. This would at least provide some warning about spurious tests, rather than just breaking spontaneously.
* However, the problems had created enough noise by this point that just a few days after that, [Hoedown was switched back to the default](https://github.com/rust-lang/rust/pull/41431) while we came up with a solution for properly warning about the differences.
* That solution came a few weeks later, [as a series of warnings when the HTML emitted by the two parsers was semantically different](https://github.com/rust-lang/rust/pull/41991). But that came at a cost, as now rustdoc needed proc-macro support (the new crate needed some custom derives farther down its dependency tree), and the build system was not equipped to handle it at the time. It was worked on for three months as the issue stumped more and more people.
* In that time, [bootstrap was completely reworked](https://github.com/rust-lang/rust/pull/43059) to change how it ordered compilation, and [the method by which it built rustdoc would change](https://github.com/rust-lang/rust/pull/43482), as well. This allowed it to only be built after stage1, when proc-macros would be available, allowing the "rendering differences" PR to finally land.
* The warnings were not perfect, and revealed a few [spurious](https://github.com/rust-lang/rust/pull/44368) [differences](https://github.com/rust-lang/rust/pull/45421) between how we handled the renderers.
* Once these were handled, [we flipped the switch to turn on the "rendering difference" warnings all the time](https://github.com/rust-lang/rust/pull/45324), in October 2017. This began the "warning cycle" for this change, and landed in stable in 1.23, on 2018-01-04.
* Once those warnings hit stable, and after a couple weeks of seeing whether we would get any more reports than what we got from sitting on nightly/beta, [we switched the renderers](https://github.com/rust-lang/rust/pull/47398), making Pulldown the default but still offering the option to use Hoedown.
And that brings us to the present. We haven't received more new issues from this in the meantime, and the "switch by default" is now on beta. Our reasoning is that, at this point, anyone who would have been affected by this has run into it already.
Use time crate in bootstrap dist instead of date
`bootstrap dist` command is trying to run *NIX specific `date` command to get current month and year. This command keep failing when it's called on a Windows command prompt. This patch is making it use time crate.
Closes: #47908
Building on the work of # 45684 this commit updates the compiler to
unconditionally load the `rustc_trans` crate at runtime instead of linking to it
at compile time. The end goal of this work is to implement # 46819 where rustc
will have multiple backends available to it to load.
This commit starts off by removing the `extern crate rustc_trans` from the
driver. This involved moving some miscellaneous functionality into the
`TransCrate` trait and also required an implementation of how to locate and load
the trans backend. This ended up being a little tricky because the sysroot isn't
always the right location (for example `--sysroot` arguments) so some extra code
was added as well to probe a directory relative to the current dll (the
rustc_driver dll).
Rustbuild has been updated accordingly as well to have a separate compilation
invocation for the `rustc_trans` crate and assembly it accordingly into the
sysroot. Finally, the distribution logic for the `rustc` package was also
updated to slurp up the trans backends folder.
A number of assorted fallout changes were included here as well to ensure tests
pass and such, and they should all be commented inline.
Check for deadlinks from the summary during book generation
Previously, any deadlinks from a book's SUMMARY.md wouldn't
cause any errors or warnings or similar but mdbook would simply
create a page with blank content.
This has kept bug #47394 hidden. It should have been detected
back in the PR when those wrongly named files got added to the
book.
PR #47414 was one component of the solution. This change
is a second line of defense for the unstable book and a first
line of defense for any other book.
We also update mdbook to the most recent version.
Update rust-installer for streaming parallelism
Pull in rust-lang/rust-installer#76 to get streamed tarball generation,
rather than batching it all in memory, while still getting the benefit
of compressing in parallel.
Pull in rust-lang/rust-installer#76 to get streamed tarball generation,
rather than batching it all in memory, while still getting the benefit
of compressing in parallel.
Previously, any deadlinks from a book's SUMMARY.md wouldn't
cause any errors or warnings or similar but mdbook would simply
create a page with blank content.
This has kept bug #47394 hidden. It should have been detected
back in the PR when those wrongly named files got added to the
book.
PR #47414 was one component of the solution. This change
is a second line of defense for the unstable book and a first
line of defense for any other book.
We also update mdbook to the most recent version.
Update `rand` crate to `0.3.19`.
Update `log` crate to `0.3.9` and `0.4.1`.
Update `parking_lot_core` crate to `0.2.9`.
Upgrade all flate2 dependencies to `1.0.1`.
- Update `rust-installer` submodule.
Remove librustdoc dependency on log
This change should have been included in PR #46386.
Since librustdoc doesn't explicitly depend on internal crates
(such as librustc_driver) through its Cargo.toml, it ends up using the
sysroot to resolve them. By removing the dependency on 'log',
we ensure that the syroot is used to resolve `log` as well. This ensures
that only one version of log is in use, so that `env_logger::init()`
enables all uses of `log!` macros.
This change should have been included in PR #46386.
Since librustdoc doesn't explicitly depend on internal crates
(such as librustc_driver) through its Cargo.toml, it ends up using the
sysroot to resolve them. By removing the dependency on 'log',
we ensure that the syroot is used to resolve `log` as well. This ensures
that only one version of log is in use, so that `env_logger::init()`
enables all uses of `log!` macros.
Remove librustdoc dependency on env_logger
We want librustdoc to pickup the env_logger dependency from
the sysroot. This ensures that the same copy of env_logger is used
for both internal crates (e.g. librustc_driver, libsyntax) and
librustdoc
Closes#46383
We want librustdoc to pickup the env_logger dependency from
the sysroot. This ensures that the same copy of env_logger is used
for both internal crates (e.g. librustc_driver, libsyntax) and
librustdoc
Closes#46383
Allow filtering analysis by reachability
Fixes#43521.
Fixes https://github.com/nrc/rls-analysis/issues/79.
This PR allows a user to filter items present in the save-analysis data by setting the `reachable_only` config option. This option is intended for use by the new rustdoc. The PR isn't quite finished, because it's dependent on a new release of rls-data, but I want to make sure that the approach is valid.
https://github.com/nrc/rls-analysis/issues/79 mentions that `pub use` might need to be handled, but my thinking is that the consumer of the analysis data would be able to infer which imports are `pub use`, and which items are only reachable through `pub use`, so that doesn't need to be handled here.
r? @nrc
std: Add a new wasm32-unknown-unknown target
This commit adds a new target to the compiler: wasm32-unknown-unknown. This target is a reimagining of what it looks like to generate WebAssembly code from Rust. Instead of using Emscripten which can bring with it a weighty runtime this instead is a target which uses only the LLVM backend for WebAssembly and a "custom linker" for now which will hopefully one day be direct calls to lld.
Notable features of this target include:
* There is zero runtime footprint. The target assumes nothing exists other than the wasm32 instruction set.
* There is zero toolchain footprint beyond adding the target. No custom linker is needed, rustc contains everything.
* Very small wasm modules can be generated directly from Rust code using this target.
* Most of the standard library is stubbed out to return an error, but anything related to allocation works (aka `HashMap`, `Vec`, etc).
* Naturally, any `#[no_std]` crate should be 100% compatible with this new target.
This target is currently somewhat janky due to how linking works. The "linking" is currently unconditional whole program LTO (aka LLVM is being used as a linker). Naturally that means compiling programs is pretty slow! Eventually though this target should have a linker.
This target is also intended to be quite experimental. I'm hoping that this can act as a catalyst for further experimentation in Rust with WebAssembly. Breaking changes are very likely to land to this target, so it's not recommended to rely on it in any critical capacity yet. We'll let you know when it's "production ready".
### Building yourself
First you'll need to configure the build of LLVM and enable this target
```
$ ./configure --target=wasm32-unknown-unknown --set llvm.experimental-targets=WebAssembly
```
Next you'll want to remove any previously compiled LLVM as it needs to be rebuilt with WebAssembly support. You can do that with:
```
$ rm -rf build
```
And then you're good to go! A `./x.py build` should give you a rustc with the appropriate libstd target.
### Test support
Currently testing-wise this target is looking pretty good but isn't complete. I've got almost the entire `run-pass` test suite working with this target (lots of tests ignored, but many passing as well). The `core` test suite is [still getting LLVM bugs fixed](https://reviews.llvm.org/D39866) to get that working and will take some time. Relatively simple programs all seem to work though!
In general I've only tested this with a local fork that makes use of LLVM 5 rather than our current LLVM 4 on master. The LLVM 4 WebAssembly backend AFAIK isn't broken per se but is likely missing bug fixes available on LLVM 5. I'm hoping though that we can decouple the LLVM 5 upgrade and adding this wasm target!
### But the modules generated are huge!
It's worth nothing that you may not immediately see the "smallest possible wasm module" for the input you feed to rustc. For various reasons it's very difficult to get rid of the final "bloat" in vanilla rustc (again, a real linker should fix all this). For now what you'll have to do is:
cargo install --git https://github.com/alexcrichton/wasm-gc
wasm-gc foo.wasm bar.wasm
And then `bar.wasm` should be the smallest we can get it!
---
In any case for now I'd love feedback on this, particularly on the various integration points if you've got better ideas of how to approach them!
This commit adds a new target to the compiler: wasm32-unknown-unknown. This
target is a reimagining of what it looks like to generate WebAssembly code from
Rust. Instead of using Emscripten which can bring with it a weighty runtime this
instead is a target which uses only the LLVM backend for WebAssembly and a
"custom linker" for now which will hopefully one day be direct calls to lld.
Notable features of this target include:
* There is zero runtime footprint. The target assumes nothing exists other than
the wasm32 instruction set.
* There is zero toolchain footprint beyond adding the target. No custom linker
is needed, rustc contains everything.
* Very small wasm modules can be generated directly from Rust code using this
target.
* Most of the standard library is stubbed out to return an error, but anything
related to allocation works (aka `HashMap`, `Vec`, etc).
* Naturally, any `#[no_std]` crate should be 100% compatible with this new
target.
This target is currently somewhat janky due to how linking works. The "linking"
is currently unconditional whole program LTO (aka LLVM is being used as a
linker). Naturally that means compiling programs is pretty slow! Eventually
though this target should have a linker.
This target is also intended to be quite experimental. I'm hoping that this can
act as a catalyst for further experimentation in Rust with WebAssembly. Breaking
changes are very likely to land to this target, so it's not recommended to rely
on it in any critical capacity yet. We'll let you know when it's "production
ready".
---
Currently testing-wise this target is looking pretty good but isn't complete.
I've got almost the entire `run-pass` test suite working with this target (lots
of tests ignored, but many passing as well). The `core` test suite is still
getting LLVM bugs fixed to get that working and will take some time. Relatively
simple programs all seem to work though!
---
It's worth nothing that you may not immediately see the "smallest possible wasm
module" for the input you feed to rustc. For various reasons it's very difficult
to get rid of the final "bloat" in vanilla rustc (again, a real linker should
fix all this). For now what you'll have to do is:
cargo install --git https://github.com/alexcrichton/wasm-gc
wasm-gc foo.wasm bar.wasm
And then `bar.wasm` should be the smallest we can get it!
---
In any case for now I'd love feedback on this, particularly on the various
integration points if you've got better ideas of how to approach them!
Miscellaneous changes for CI, Docker and compiletest.
This PR contains 7 independent commits that improves interaction with CI, Docker and compiletest.
1. a4e5c91cb8 — Forces a newline every 100 dots when testing in quiet mode. Prevents spurious timeouts when abusing the CI to test Android jobs.
2. 1b5aaf22e8 — Use vault.centos.org for dist-powerpc64le-linux, see #45744.
3. 33400fbbcd — Modify `src/ci/docker/run.sh` so that the docker images can be run from Docker Toolbox for Windows on Windows 7. I haven't checked the behavior of the newer Docker for Windows on Windows 10. Also, "can run" does not mean all the test can pass successfully (the UDP tests failed last time I checked)
4. d517668a08 — Don't emit a real warning the linker segfault, which affects UI tests like https://github.com/rust-lang/rust/pull/45489#issuecomment-340134944. Log it instead.
5. 51e2247948 — During run-pass, trim the output if stdout/stderr exceeds 416 KB (top 160 KB + bottom 256 KB). This is an attempt to avoid spurious failures like https://github.com/rust-lang/rust/pull/45384#issuecomment-341755788
6. 9cfdabaf3c — Force `gem update --system` before deploy. This is an attempt to prevent spurious error #44159.
7. eee10cc482 — Tries to print the crash log on macOS on failure. This is an attempt to debug #45230.
This commit removes the `rand` crate from the standard library facade as
well as the `__rand` module in the standard library. Neither of these
were used in any meaningful way in the standard library itself. The only
need for randomness in libstd is to initialize the thread-local keys of
a `HashMap`, and that unconditionally used `OsRng` defined in the
standard library anyway.
The cruft of the `rand` crate and the extra `rand` support in the
standard library makes libstd slightly more difficult to port to new
platforms, namely WebAssembly which doesn't have any randomness at all
(without interfacing with JS). The purpose of this commit is to clarify
and streamline randomness in libstd, focusing on how it's only required
in one location, hashmap seeds.
Note that the `rand` crate out of tree has almost always been a drop-in
replacement for the `rand` crate in-tree, so any usage (accidental or
purposeful) of the crate in-tree should switch to the `rand` crate on
crates.io. This then also has the further benefit of avoiding
duplication (mostly) between the two crates!
This affects regular code generation as well as constant evaluation in trans,
but not the HIR constant evaluator because that one returns an error for
overflowing casts and NaN-to-int casts. That error is conservatively
correct and we should be careful to not accept more code in constant
expressions.
The changes to code generation are guarded by a new -Z flag, to be able
to evaluate the performance impact. The trans constant evaluation changes
are unconditional because they have no run time impact and don't affect
type checking either.
These tools have been corrected in their upstream repo's, and the
submodules have been updated here to reflect that. I also had to update
Cargo to match what the RLS is expecting.
The tool states for `rustfmt` and `rls` where both changed from "Broken"
to "Testing" in this commit, thus enabling testing and distribution
again.
Make last structs indexes definitions use newtype_index macro
This PR makes the last two index structs not using newtype_index macro to use it and also fixes this https://github.com/rust-lang/rust/issues/45763 issue.
This is intended to prevent the spurious OOM error from
run-pass/rustc-rust-log.rs, by skipping the output in the middle when the
size is over 416 KB, so that the log output will not be overwhelmed.
Bump to 1.23 and update bootstrap
This commit updates the bootstrap compiler, bumps the version to 1.23, updates
Cargo, updates books, and updates crates.io dependencies
Don't emit the same compiler diagnostic twice.
This PR makes the compiler filter out diagnostic messages that have already been emitted during the same compilation session.
update jobserver version to work around macos bug
Update `jobserver` crate to fixrust-lang/cargo#4643, a panic which can't happen according to `libc::poll`'s man page but was nevertheless reported on macOS 10.9 and 10.10.
r? @alexcrichton
Reactivate clippy in toolstate.toml
The `Cargo.lock` changes are b/c clippy is not its own workspace anymore, but is part of the main workspace
r? @Manishearth
rustc: Remove `used_mut_nodes` from `TyCtxt`
This updates the borrowck query to return a result, and this result is then used
to incrementally check for unused mutable nodes given sets of all the used
mutable nodes.
Closes#42384
Add x86_64-unknown-linux-gnux32 target
This adds X32 ABI support for Linux on X86_64. Let's package and dist it so we can star testing libc, libstd, etc.
Fixes https://github.com/rust-lang/rfcs/issues/1339
This updates the borrowck query to return a result, and this result is then used
to incrementally check for unused mutable nodes given sets of all the used
mutable nodes.
Closes#42384
Initial support for `..=` syntax
#28237
This PR adds `..=` as a synonym for `...` in patterns and expressions.
Since `...` in expressions was never stable, we now issue a warning.
cc @durka
r? @aturon
