Implement #85440 (Random test ordering)
This PR adds `--shuffle` and `--shuffle-seed` options to `libtest`. The options are similar to the [`-shuffle` option](c894b442d1/src/testing/testing.go (L1482-L1499)) that was recently added to Go.
Here are the relevant parts of the help message:
```
--shuffle Run tests in random order
--shuffle-seed SEED
Run tests in random order; seed the random number
generator with SEED
...
By default, the tests are run in alphabetical order. Use --shuffle or set
RUST_TEST_SHUFFLE to run the tests in random order. Pass the generated
"shuffle seed" to --shuffle-seed (or set RUST_TEST_SHUFFLE_SEED) to run the
tests in the same order again. Note that --shuffle and --shuffle-seed do not
affect whether the tests are run in parallel.
```
Is an RFC needed for this?
Rename `std:🧵:available_conccurrency` to `std:🧵:available_parallelism`
_Tracking issue: https://github.com/rust-lang/rust/issues/74479_
This PR renames `std:🧵:available_conccurrency` to `std:🧵:available_parallelism`.
## Rationale
The API was initially named `std:🧵:hardware_concurrency`, mirroring the [C++ API of the same name](https://en.cppreference.com/w/cpp/thread/thread/hardware_concurrency). We eventually decided to omit any reference to the word "hardware" after [this comment](https://github.com/rust-lang/rust/pull/74480#issuecomment-662045841). And so we ended up with `available_concurrency` instead.
---
For a talk I was preparing this week I was reading through ["Understanding and expressing scalable concurrency" (A. Turon, 2013)](http://aturon.github.io/academic/turon-thesis.pdf), and the following passage stood out to me (emphasis mine):
> __Concurrency is a system-structuring mechanism.__ An interactive system that deals with disparate asynchronous events is naturally structured by division into concurrent threads with disparate responsibilities. Doing so creates a better fit between problem and solution, and can also decrease the average latency of the system by preventing long-running computations from obstructing quicker ones.
> __Parallelism is a resource.__ A given machine provides a certain capacity for parallelism, i.e., a bound on the number of computations it can perform simultaneously. The goal is to maximize throughput by intelligently using this resource. For interactive systems, parallelism can decrease latency as well.
_Chapter 2.1: Concurrency is not Parallelism. Page 30._
---
_"Concurrency is a system-structuring mechanism. Parallelism is a resource."_ — It feels like this accurately captures the way we should be thinking about these APIs. What this API returns is not "the amount of concurrency available to the program" which is a property of the program, and thus even with just a single thread is effectively unbounded. But instead it returns "the amount of _parallelism_ available to the program", which is a resource hard-constrained by the machine's capacity (and can be further restricted by e.g. operating systems).
That's why I'd like to propose we rename this API from `available_concurrency` to `available_parallelism`. This still meets the criteria we previously established of not attempting to define what exactly we mean by "hardware", "threads", and other such words. Instead we only talk about "concurrency" as an abstract resource available to our program.
r? `@joshtriplett`
make junit output more consistent with default format
The default format of libtest includes new-lines between each section to ensure the label output from cargo is on it's own line
<pre><font color="#A1B56C"><b>❯</b></font> <font color="#A1B56C">cargo</font><font color="#D8D8D8"> </font><font color="#A1B56C">test</font>
<font color="#A1B56C"><b> Compiling</b></font> test-test v0.1.0 (/home/jlusby/tmp/test-test)
<font color="#A1B56C"><b> Finished</b></font> test [unoptimized + debuginfo] target(s) in 0.59s
<font color="#A1B56C"><b> Running</b></font> unittests (target/debug/deps/test_test-639f369234319c09)
running 1 test
test tests::it_works ... <font color="#A1B56C">ok</font>
test result: <font color="#A1B56C">ok</font>. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
<font color="#A1B56C"><b> Doc-tests</b></font> test-test
running 0 tests
test result: <font color="#A1B56C">ok</font>. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
</pre>
But when the junit outputter was added to libtest these newlines were omitted, resulting in some "fun" output when run via cargo.
Note the `Doc-tests` text at the end of the first line of xml.
<pre><font color="#A1B56C"><b>❯</b></font> <font color="#A1B56C">cargo</font><font color="#D8D8D8"> </font><font color="#A1B56C">test</font><font color="#D8D8D8"> </font><font color="#A1B56C">--</font><font color="#D8D8D8"> </font><font color="#A1B56C">-Zunstable-options</font><font color="#D8D8D8"> </font><font color="#A1B56C">--format</font><font color="#D8D8D8"> </font><font color="#A1B56C">junit</font>
<font color="#A1B56C"><b> Finished</b></font> test [unoptimized + debuginfo] target(s) in 0.00s
<font color="#A1B56C"><b> Running</b></font> unittests (target/debug/deps/test_test-639f369234319c09)
<?xml version="1.0" encoding="UTF-8"?><testsuites><testsuite name="test" package="test" id="0" errors="0" failures="0" tests="1" skipped="0" ><testcase classname="tests" name="it_works" time="0"/><system-out/><system-err/></testsuite></testsuites><font color="#A1B56C"><b> Doc-tests</b></font> test-test
<?xml version="1.0" encoding="UTF-8"?><testsuites><testsuite name="test" package="test" id="0" errors="0" failures="0" tests="0" skipped="0" ><system-out/><system-err/></testsuite></testsuites>
</pre>
After this PR the junit output includes the same style of newlines as the pretty format
<pre><font color="#A1B56C"><b>❯</b></font> <font color="#A1B56C">cargo</font><font color="#D8D8D8"> </font><font color="#A1B56C">test</font><font color="#D8D8D8"> </font><font color="#A1B56C">--</font><font color="#D8D8D8"> </font><font color="#A1B56C">-Zunstable-options</font><font color="#D8D8D8"> </font><font color="#A1B56C">--format</font><font color="#D8D8D8"> </font><font color="#A1B56C">junit</font>
<font color="#A1B56C"><b> Compiling</b></font> test-test v0.1.0 (/home/jlusby/tmp/test-test)
<font color="#A1B56C"><b> Finished</b></font> test [unoptimized + debuginfo] target(s) in 0.39s
<font color="#A1B56C"><b> Running</b></font> unittests (target/debug/deps/test_test-42c2320bb9450c69)
<?xml version="1.0" encoding="UTF-8"?><testsuites><testsuite name="test" package="test" id="0" errors="0" failures="0" tests="1" skipped="0" ><testcase classname="tests" name="it_works" time="0"/><system-out/><system-err/></testsuite></testsuites>
<font color="#A1B56C"><b> Doc-tests</b></font> test-test
<?xml version="1.0" encoding="UTF-8"?><testsuites><testsuite name="test" package="test" id="0" errors="0" failures="0" tests="0" skipped="0" ><system-out/><system-err/></testsuite></testsuites>
</pre>
Since RFC 3052 soft deprecated the authors field anyway, hiding it from
crates.io, docs.rs, and making Cargo not add it by default, and it is
not generally up to date/useful information, we should remove it from
crates in this repo.
Show test type during prints
Test output can sometimes be confusing. For example doctest with the no_run argument are displayed the same way than test that are run.
During #83857 I got the feedback that test output can be confusing.
For the moment test output is
```
test $DIR/test-type.rs - f (line 12) ... ignored
test $DIR/test-type.rs - f (line 15) ... ok
test $DIR/test-type.rs - f (line 21) ... ok
test $DIR/test-type.rs - f (line 6) ... ok
```
I propose to change output by indicating the test type as
```
test $DIR/test-type.rs - f (line 12) ... ignored
test $DIR/test-type.rs - f (line 15) - compile ... ok
test $DIR/test-type.rs - f (line 21) - compile fail ... ok
test $DIR/test-type.rs - f (line 6) ... ok
```
by indicating the test type after the test name (and in the case of doctest after the function name and line) and before the "...".
------------
Note: this is a proof of concept, the implementation is probably not optimal as the properties added in `TestDesc` are only use in the display and does not represent actual change of behavior, maybe `TestType::DocTest` could have fields
## User-facing changes
- Intra-doc links to primitives that currently go to rust-lang.org/nightly/std/primitive.x.html will start going to channel that rustdoc was built with. Nightly will continue going to /nightly; Beta will link to /beta; stable compilers will link to /1.52.1 (or whatever version they were built as).
- Cross-crate links from std to core currently go to /nightly unconditionally. They will start going to /1.52.0 on stable channels (but remain the same on nightly channels).
- Intra-crate links from std to std (or core to core) currently go to the same URL they are hosted at; they will continue to do so. Notably, this is different from everything else because it can preserve the distinction between /stable and /1.52.0 by using relative links.
Note that "links" includes both intra-doc links and rustdoc's own
automatically generated hyperlinks.
## Implementation changes
- Update the testsuite to allow linking to /beta and /1.52.1 in docs
- Use an html_root_url for the standard library that's dependent on the channel
This avoids linking to nightly docs on stable.
- Update rustdoc to use channel-dependent links for primitives from an
unknown crate
- Set DOC_RUST_LANG_ORG_CHANNEL from bootstrap to ensure it's in sync
- Include doc.rust-lang.org in the channel
Add a chapter on the test harness.
There isn't really any online documentation on the test harness, so this adds a chapter to the rustc book which provides information on how the harness works and details on the command-line options.
test: Print test name only once on timeout
Pretty formatter when using multiple test threads displays test name twice on
timeout event. This implicitly suggest that those are two different events,
while in fact they are always printed together.
Print test name only once.
Before:
```
running 3 tests
test src/lib.rs - c (line 16) ... ok
test src/lib.rs - a (line 3) ... ok
test src/lib.rs - b (line 9) ... test src/lib.rs - b (line 9) has been running for over 60 seconds
test src/lib.rs - b (line 9) ... ok
```
After:
```
running 3 tests
test src/lib.rs - c (line 16) ... ok
test src/lib.rs - a (line 3) ... ok
test src/lib.rs - b (line 9) has been running for over 60 seconds
test src/lib.rs - b (line 9) ... ok
```
Pretty formatter when using multiple test threads displays test name twice on
timeout event. This implicitly suggest that those are two different events,
while in fact they are always printed together.
Print test name only once.
Before:
```
running 3 tests
test src/lib.rs - c (line 16) ... ok
test src/lib.rs - a (line 3) ... ok
test src/lib.rs - b (line 9) ... test src/lib.rs - b (line 9) has been running for over 60 seconds
test src/lib.rs - b (line 9) ... ok
```
After:
```
running 3 tests
test src/lib.rs - c (line 16) ... ok
test src/lib.rs - a (line 3) ... ok
test src/lib.rs - b (line 9) has been running for over 60 seconds
test src/lib.rs - b (line 9) ... ok
```
libtest: Fix unwrap panic on duplicate TestDesc
It is possible for different tests to collide to the same `TestDesc` when macros are involved. That is a bug, but it didn’t cause a panic until #81367. For now, change the code to ignore this problem.
Fixes#81852.
This will need to be applied to `beta` too.
It is possible for different tests to collide to the same TestDesc
when macros are involved. That is a bug, but it didn’t cause a panic
until #81367. For now, change the code to ignore this problem.
Fixes#81852.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
[libtest] Run the test synchronously when hitting thread limit
libtest currently panics if it hits the thread limit. This often results in spurious test failures (<code>thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }'</code> ... `error: test failed, to rerun pass '--lib'`). This PR makes it continue to run the test synchronously if it runs out of threads.
Closes#78165.
``@rustbot`` label: A-libtest T-libs
libtest: allow multiple filters
Libtest ignores any filters after the first. This changes it so that if multiple filters are passed, it will test against all of them.
This also affects compiletest to do the same.
Closes#30422
Upgrade wasm32 image to Ubuntu 20.04
This switches the wasm32 image, which is used to test
wasm32-unknown-emscripten, to Ubuntu 20.04. While at it, enable
most of the excluded tests, as they seem to work fine with some
minor fixes.
This switches the wasm32 image, which is used to test
wasm32-unknown-emscripten to Ubuntu 20.04. While at it, enable
most of the excluded tests, as they seem to work fine with some
minor fixes.
