This ensures that each build will support the testing design of "dry
running" builds. It's also checked that a dry run build is equivalent
step-wise to a "wet" run build; the graphs we generate when running are
directly compared node/node and edge/edge, both for order and contents.
For LLD integration the path to `llvm-config` needed to change to inside the
build directory itself (for whatever reason) but the build directory is
different on MSBuild than it is on `ninja` for MSVC builds, so the path to
`llvm-config.exe` was actually wrong and not working!
This commit removes the `Build::llvm_config` function in favor of the source of
truth, the `Llvm` build step itself. The build step was then updated to find the
right build directory for MSBuild as well as `ninja` for where `llvm-config.exe`
is located.
Closes#48749
Currently, we are producing headers for markdown files,
which is generally not what we want. As such, passing this
flag causes them to render normally.
Previously, a Step would be able to tell on its own when it was invoked
"by-default" (that is, `./x.py test` was called instead of `./x.py test
some/path`). This commit replaces that functionality, invoking each Step
with each of the paths it has specified as "should be invoked by."
For example, if a step calls `path("src/tools/cargo")` and
`path("src/doc/cargo")` then it's make_run will be called twice, with
"src/tools/cargo" and "src/doc/cargo." This makes it so that default
handling logic is in builder, instead of spread across various Steps.
However, this meant that some Step specifications needed to be updated,
since for example `rustdoc` can be built by `./x.py build
src/librustdoc` or `./x.py build src/tools/rustdoc`. A `PathSet`
abstraction is added that handles this: now, each Step can not only list
`path(...)` but also `paths(&[a, b, ...])` which will make it so that we
don't invoke it with each of the individual paths, instead invoking it
with the first path in the list (though this shouldn't be depended on).
Future work likely consists of implementing a better/easier way for a
given Step to work with "any" crate in-tree, especially those that want
to run tests, build, or check crates in the std, test, or rustc crate
trees. Currently this is rather painful to do as most of the logic is
duplicated across should_run and make_run. It seems likely this can be
abstracted away into builder somehow.
Building on the work of # 45684 this commit updates the compiler to
unconditionally load the `rustc_trans` crate at runtime instead of linking to it
at compile time. The end goal of this work is to implement # 46819 where rustc
will have multiple backends available to it to load.
This commit starts off by removing the `extern crate rustc_trans` from the
driver. This involved moving some miscellaneous functionality into the
`TransCrate` trait and also required an implementation of how to locate and load
the trans backend. This ended up being a little tricky because the sysroot isn't
always the right location (for example `--sysroot` arguments) so some extra code
was added as well to probe a directory relative to the current dll (the
rustc_driver dll).
Rustbuild has been updated accordingly as well to have a separate compilation
invocation for the `rustc_trans` crate and assembly it accordingly into the
sysroot. Finally, the distribution logic for the `rustc` package was also
updated to slurp up the trans backends folder.
A number of assorted fallout changes were included here as well to ensure tests
pass and such, and they should all be commented inline.
Test rustdoc js
Add tests for the rustdoc search. It was heavily required because of all the recent breaking changes that happened while I went through improvements in doc search (add search in/for generic search for example).
Fix broken CSS for book redirect pages
rust.css has to be next to the font files so we shouldn't copy it for
only the book redirect pages, instead just use the version that is
already there.
This also removes the duplicate code creating version_info.html.
Fixes: #45974
rust.css has to be next to the font files so we shouldn't copy it for
only the book redirect pages, instead just use the version that is
already there.
This also removes the duplicate code creating version_info.html.
rustdoc is built separately to rustc now so the docs would need to be
generated separately as well. Also rustdoc doesn't build at stage 1
which prevented the compiler docs being built at stage 1.
As discussed in
https://github.com/rust-lang/cargo/issues/4040#issuecomment-321639074
This is a redirect meant to be replaced once cargo docs have been
converted to mdbook. We just want *a* URL to ride the trains for now so
that we can print doc.rust-lang.org/cargo in the paper book and
guarantee that it will go *somewhere* useful by the time the book is
printed.
Implemented as a meta redirect in HTML because we don't currently have
any google juice at doc.rust-lang.org/cargo to lose.
Autogenerate stubs and SUMMARY.md in the unstable book
Removes a speed bump in compiler development by autogenerating stubs for features in the unstable book. See #42454 for discussion.
The PR contains three commits, separated in order to make review easy:
* The first commit converts the tidy tool from a binary crate to a crate that contains both a library and a binary. In the second commit, we'll use the tidy library
* The second and main commit introduces autogeneration of SUMMARY.md and feature stub files
* The third commit turns off the tidy lint that checks for features without a stub, and removes the stub files. A separate commit due to the large number of files touched
Members of the doc team who wish to document some features can either do this (where `$rustsrc` is the root of the rust repo git checkout):
1. cd to `$rustsrc/src/tools/unstable-book-gen` and then do `cargo run $rustsrc/src $rustsrc/src/doc/unstable-book` to put the stubs into the unstable book
2. cd to `$rustsrc` and run `git ls-files --others --exclude-standard` to list the newly added stubs
3. choose a file to edit, then `git add` it and `git commit`
4. afterwards, remove all changes by the tool by doing `git --reset hard` and `git clean -f`
Or they can do this:
1. remove the comment marker in `src/tools/tidy/src/unstable_book.rs` line 122
2. run `./x.py test src/tools/tidy` to list the unstable features which only have stubs
3. revert the change in 1
3. document one of the chosen unstable features
The changes done by this PR also allow for further development:
* tidy obtains information about tracking issues. We can now forbid differing tracking issues between differing `#![unstable]` annotations. I haven't done this but plan to in a future PR
* we now have a general framework for generating stuff for the unstable book at build time. Further changes can autogenerate a list of the API a given library feature exposes.
The old way to simply click through the documentation after it has been uploaded to rust-lang.org works as well.
r? @nagisa
Fixes#42454
* Make sure std docs are generated before compiler docs so rustdoc uses
relative links.
* Don't document the rustc and rustdoc binary crates as they overwrite
the real rustc and rustdoc crates.
This commit fixes#38749 by building documentation for the `proc_macro` crate by
default for configured hosts. Unfortunately did not turn out to be a trivial
fix. Currently rustbuild generates documentation into multiple locations: one
for std, one for test, and one for rustc. The initial fix for this issue simply
actually executed `cargo doc -p proc_macro` which was otherwise completely
elided before.
Unfortunately rustbuild was the left to merge two documentation trees together.
One for the standard library and one for the rustc tree (which only had docs for
the `proc_macro` crate). Rustdoc itself knows how to merge documentation files
(specifically around search indexes, etc) but rustbuild was unaware of this, so
an initial fix ended up destroying the sidebar and the search bar from the
libstd docs.
To solve this issue the method of documentation has been tweaked slightly in
rustbuild. The build system will not use symlinks (or directory junctions on
Windows) to generate all documentation into the same location initially. This'll
rely on rustdoc's logic to weave together all the output and ensure that it ends
up all consistent.
Closes#38749
This commit adds support to rustbuild for compiling Cargo as part of the release
process. Previously rustbuild would simply download a Cargo snapshot and
repackage it. With this change we should be able to turn off artifacts from the
rust-lang/cargo repository and purely rely on the artifacts Cargo produces here.
The infrastructure added here is intended to be extensible to other components,
such as the RLS. It won't exactly be a one-line addition, but the addition of
Cargo didn't require too much hooplah anyway.
The process for release Cargo will now look like:
* The rust-lang/rust repository has a Cargo submodule which is used to build a
Cargo to pair with the rust-lang/rust release
* Periodically we'll update the cargo submodule as necessary on rust-lang/rust's
master branch
* When branching beta we'll create a new branch of Cargo (as we do today), and
the first commit to the beta branch will be to update the Cargo submodule to
this exact revision.
* When branching stable, we'll ensure that the Cargo submodule is updated and
then make a stable release.
Backports to Cargo will look like:
* Send a PR to cargo's master branch
* Send a PR to cargo's release branch (e.g. rust-1.16.0)
* Send a PR to rust-lang/rust's beta branch updating the submodule
* Eventually send a PR to rust-lang/rust's master branch updating the submodule
For reference, the process to add a new component to the rust-lang/rust release
would look like:
* Add `$foo` as a submodule in `src/tools`
* Add a `tool-$foo` step which compiles `$foo` with the specified compiler,
likely mirroring what Cargo does.
* Add a `dist-$foo` step which uses `src/tools/$foo` and the `tool-$foo` output
to create a rust-installer package for `$foo` likely mirroring what Cargo
does.
* Update the `dist-extended` step with a new dependency on `dist-$foo`
* Update `src/tools/build-manifest` for the new component.
test: Verify all sysroot crates are unstable
As we continue to add more crates to the compiler and use them to implement
various features we want to be sure we're not accidentally expanding the API
surface area of the compiler! To that end this commit adds a new `run-make` test
which will attempt to `extern crate foo` all crates in the sysroot, verifying
that they're all unstable.
This commit discovered that the `std_shim` and `test_shim` crates were
accidentally stable and fixes the situation by deleting those shims. The shims
are no longer necessary due to changes in Cargo that have happened since they
were originally incepted.
As we continue to add more crates to the compiler and use them to implement
various features we want to be sure we're not accidentally expanding the API
surface area of the compiler! To that end this commit adds a new `run-make` test
which will attempt to `extern crate foo` all crates in the sysroot, verifying
that they're all unstable.
This commit discovered that the `std_shim` and `test_shim` crates were
accidentally stable and fixes the situation by deleting those shims. The shims
are no longer necessary due to changes in Cargo that have happened since they
were originally incepted.
mdbook has a lot of optional dependencies that we don't want, so instead
of using it directly, we re-build rustbook to use mdbook as a library.
For convenience' sake, we keep the same CLI interface as mdbook; the
only difference is that it only accepts build and test subcommands,
rather than the full range.
This commit optimizes the compile time for creating tarballs of cross-host
compilers and as a proof of concept adds two to the standard Travis matrix. Much
of this commit is further refactoring and refining of the `step.rs` definitions
along with the interpretation of `--target` and `--host` flags. This has gotten
confusing enough that I've also added a small test suite to
`src/bootstrap/step.rs` to ensure what we're doing works and doesn't regress.
After this commit when you execute:
./x.py dist --host $MY_HOST --target $MY_HOST
the build system will compile two compilers. The first is for the build platform
and the second is for the host platform. This second compiler is then packaged
up and placed into `build/dist` and is ready to go. With a fully cached LLVM and
docker image I was able to create a cross-host compiler in around 20 minutes
locally.
Eventually we plan to add a whole litany of cross-host entries to the Travis
matrix, but for now we're just adding a few before we eat up all the extra
capacity.
cc #38531
rustbuild: Compile all support tools in stage0
This commit changes all tools and such to get compiled in stage0, not in
later stages. The purpose of this commit is to cut down dependencies on later
stages for future modifications to the build system. Notably we're going to be
adding builders that produce a full suite of cross-compiled artifacts for a
particular host, and that shouldn't compile the `x86_64-unknown-linux-gnu`
compiler more than once. Currently dependencies on, for example, the error index
end up compiling the `x86_64-unknown-linux-gnu` compiler more than necessary.
As a result here we move many dependencies on these tools to being produced by a
stage0 compiler, not a stage1+ compiler. None of these tools actually need to be
staged at all, so they'll exhibit consistent behavior across the stages.
This commit changes all tools and such to get compiled in stage0, not in
later stages. The purpose of this commit is to cut down dependencies on later
stages for future modifications to the build system. Notably we're going to be
adding builders that produce a full suite of cross-compiled artifacts for a
particular host, and that shouldn't compile the `x86_64-unknown-linux-gnu`
compiler more than once. Currently dependencies on, for example, the error index
end up compiling the `x86_64-unknown-linux-gnu` compiler more than necessary.
As a result here we move many dependencies on these tools to being produced by a
stage0 compiler, not a stage1+ compiler. None of these tools actually need to be
staged at all, so they'll exhibit consistent behavior across the stages.
This commit switches the rustbuild build system to compiling the
compiler twice for a normal bootstrap rather than the historical three
times.
Rust is a bootstrapped language which means that a previous version of
the compiler is used to build the next version of the compiler. Over
time, however, we change many parts of compiler artifacts such as the
metadata format, symbol names, etc. These changes make artifacts from
one compiler incompatible from another compiler. Consequently if a
compiler wants to be able to use some artifacts then it itself must have
compiled the artifacts.
Historically the rustc build system has achieved this by compiling the
compiler three times:
* An older compiler (stage0) is downloaded to kick off the chain.
* This compiler now compiles a new compiler (stage1)
* The stage1 compiler then compiles another compiler (stage2)
* Finally, the stage2 compiler needs libraries to link against, so it
compiles all the libraries again.
This entire process amounts in compiling the compiler three times.
Additionally, this process always guarantees that the Rust source tree
can compile itself because the stage2 compiler (created by a freshly
created compiler) would successfully compile itself again. This
property, ensuring Rust can compile itself, is quite important!
In general, though, this third compilation is not required for general
purpose development on the compiler. The third compiler (stage2) can
reuse the libraries that were created during the second compile. In
other words, the second compilation can produce both a compiler and the
libraries that compiler will use. These artifacts *must* be compatible
due to the way plugins work today anyway, and they were created by the
same source code so they *should* be compatible as well.
So given all that, this commit switches the default build process to
only compile the compiler three times, avoiding this third compilation
by copying artifacts from the previous one. Along the way a new entry in
the Travis matrix was also added to ensure that our full bootstrap can
succeed. This entry does not run tests, though, as it should not be
necessary.
To restore the old behavior of a full bootstrap (three compiles) you can
either pass:
./configure --enable-full-bootstrap
or if you're using config.toml:
[build]
full-bootstrap = true
Overall this will hopefully be an easy 33% win in build times of the
compiler. If we do 33% less work we should be 33% faster! This in turn
should affect cycle times and such on Travis and AppVeyor positively as
well as making it easier to work on the compiler itself.
This commit is a rewrite of the user-facing interface to the rustbuild build
system. The intention here is to make it much easier to compile/test the project
without having to remember weird rule names and such. An overall view of the new
interface is:
# build everything
./x.py build
# document everyting
./x.py doc
# test everything
./x.py test
# test libstd
./x.py test src/libstd
# build libcore stage0
./x.py build src/libcore --stage 0
# run stage1 run-pass tests
./x.py test src/test/run-pass --stage 1
The `src/bootstrap/bootstrap.py` script is now aliased as a top-level `x.py`
script. This `x` was chosen to be both short and easily tab-completable (no
collisions in that namespace!). The build system now accepts a "subcommand" of
what to do next, the main ones being build/doc/test.
Each subcommand then receives an optional list of arguments. These arguments are
paths in the source repo of what to work with. That is, if you want to test a
directory, you just pass that directory as an argument.
The purpose of this rewrite is to do away with all of the arcane renames like
"rpass" is the "run-pass" suite, "cfail" is the "compile-fail" suite, etc. By
simply working with directories and files it's much more intuitive of how to run
a test (just pass it as an argument).
The rustbuild step/dependency management was also rewritten along the way to
make this easy to work with and define, but that's largely just a refactoring of
what was there before.
The *intention* is that this support is extended for arbitrary files (e.g.
`src/test/run-pass/my-test-case.rs`), but that isn't quite implemented just yet.
Instead directories work for now but we can follow up with stricter path
filtering logic to plumb through all the arguments.
The organization in rustbuild was a little odd at the moment where the `lib.rs`
was quite small but the binary `main.rs` was much larger. Unfortunately as well
there was a `build/` directory with the implementation of the build system, but
this directory was ignored by GitHub on the file-search prompt which was a
little annoying.
This commit reorganizes rustbuild slightly where all the library files (the
build system) is located directly inside of `src/bootstrap` and all the binaries
now live in `src/bootstrap/bin` (they're small). Hopefully this should allow
GitHub to index and allow navigating all the files while maintaining a
relatively similar layout to the other libraries in `src/`.
