mirror of
https://github.com/rust-lang/rust.git
synced 2024-10-30 22:12:15 +00:00
Auto merge of #82350 - ehuss:test-chapter, r=jyn514
Add a chapter on the test harness. There isn't really any online documentation on the test harness, so this adds a chapter to the rustc book which provides information on how the harness works and details on the command-line options.
This commit is contained in:
bors
commit
247337f409
@ -51,14 +51,8 @@ fn optgroups() -> getopts::Options {
|
||||
.optflag("", "test", "Run tests and not benchmarks")
|
||||
.optflag("", "bench", "Run benchmarks instead of tests")
|
||||
.optflag("", "list", "List all tests and benchmarks")
|
||||
.optflag("h", "help", "Display this message (longer with --help)")
|
||||
.optopt(
|
||||
"",
|
||||
"logfile",
|
||||
"Write logs to the specified file instead \
|
||||
of stdout",
|
||||
"PATH",
|
||||
)
|
||||
.optflag("h", "help", "Display this message")
|
||||
.optopt("", "logfile", "Write logs to the specified file", "PATH")
|
||||
.optflag(
|
||||
"",
|
||||
"nocapture",
|
||||
|
||||
@ -11,6 +11,7 @@
|
||||
- [Deny-by-default lints](lints/listing/deny-by-default.md)
|
||||
- [Codegen options](codegen-options/index.md)
|
||||
- [JSON Output](json.md)
|
||||
- [Tests](tests/index.md)
|
||||
- [Platform Support](platform-support.md)
|
||||
- [Targets](targets/index.md)
|
||||
- [Built-in Targets](targets/built-in.md)
|
||||
|
||||
@ -204,7 +204,8 @@ out a longer explanation of a given error.
|
||||
## `--test`: build a test harness
|
||||
|
||||
When compiling this crate, `rustc` will ignore your `main` function
|
||||
and instead produce a test harness.
|
||||
and instead produce a test harness. See the [Tests chapter](tests/index.md)
|
||||
for more information about tests.
|
||||
|
||||
<a id="option-target"></a>
|
||||
## `--target`: select a target triple to build
|
||||
|
||||
284
src/doc/rustc/src/tests/index.md
Normal file
284
src/doc/rustc/src/tests/index.md
Normal file
@ -0,0 +1,284 @@
|
||||
# Tests
|
||||
|
||||
`rustc` has a built-in facility for building and running tests for a crate.
|
||||
More information about writing and running tests may be found in the [Testing
|
||||
Chapter] of the Rust Programming Language book.
|
||||
|
||||
Tests are written as free functions with the [`#[test]`
|
||||
attribute][attribute-test]. For example:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn it_works() {
|
||||
assert_eq!(2 + 2, 4);
|
||||
}
|
||||
```
|
||||
|
||||
Tests "pass" if they return without an error. They "fail" if they [panic], or
|
||||
return a [`Result`] with an error.
|
||||
|
||||
By passing the [`--test` option] to `rustc`, the compiler will build the crate
|
||||
in a special mode to construct an executable that will run the tests in the
|
||||
crate. The `--test` flag will make the following changes:
|
||||
|
||||
* The crate will be built as a `bin` [crate type], forcing it to be an
|
||||
executable.
|
||||
* Links the executable with [`libtest`], the test harness that is part of the
|
||||
standard library, which handles running the tests.
|
||||
* Synthesizes a [`main` function] which will process command-line arguments
|
||||
and run the tests. This new `main` function will replace any existing `main`
|
||||
function as the entry point of the executable, though the existing `main`
|
||||
will still be compiled.
|
||||
* Enables the [`test` cfg option], which allows your code to use conditional
|
||||
compilation to detect if it is being built as a test.
|
||||
* Enables building of functions annotated with the [`test`][attribute-test]
|
||||
and [`bench`](#benchmarks) attributes, which will be run by the test
|
||||
harness.
|
||||
|
||||
After the executable is created, you can run it to execute the tests and
|
||||
receive a report on what passes and fails. If you are using [Cargo] to manage
|
||||
your project, it has a built-in [`cargo test`] command which handles all of
|
||||
this automatically. An example of the output looks like this:
|
||||
|
||||
```text
|
||||
running 4 tests
|
||||
test it_works ... ok
|
||||
test check_valid_args ... ok
|
||||
test invalid_characters ... ok
|
||||
test walks_the_dog ... ok
|
||||
|
||||
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
|
||||
```
|
||||
|
||||
> **Note**: Tests must be built with the [`unwind` panic
|
||||
> strategy][panic-strategy]. This is because all tests run in the same
|
||||
> process, and they are intended to catch panics, which is not possible with
|
||||
> the `abort` strategy. See the unstable [`-Z panic-abort-tests`] option for
|
||||
> experimental support of the `abort` strategy by spawning tests in separate
|
||||
> processes.
|
||||
|
||||
## Test attributes
|
||||
|
||||
Tests are indicated using attributes on free functions. The following
|
||||
attributes are used for testing, see the linked documentation for more
|
||||
details:
|
||||
|
||||
* [`#[test]`][attribute-test] — Indicates a function is a test to be run.
|
||||
* `#[bench]` — Indicates a function is a benchmark to be
|
||||
run. Benchmarks are currently unstable and only available in the nightly
|
||||
channel, see the [unstable docs][bench-docs] for more details.
|
||||
* [`#[should_panic]`][attribute-should_panic] — Indicates that the test
|
||||
function will only pass if the function [panics][panic].
|
||||
* [`#[ignore]`][attribute-ignore] — Indicates that the test function will be
|
||||
compiled, but not run by default. See the [`--ignored`](#--ignored) and
|
||||
[`--include-ignored`](#--include-ignored) options to run these tests.
|
||||
|
||||
## CLI arguments
|
||||
|
||||
The libtest harness has several command-line arguments to control its
|
||||
behavior.
|
||||
|
||||
> Note: When running with [`cargo test`], the libtest CLI arguments must be
|
||||
> passed after the `--` argument to differentiate between flags for Cargo and
|
||||
> those for the harness. For example: `cargo test -- --nocapture`
|
||||
|
||||
### Filters
|
||||
|
||||
Positional arguments (those without a `-` prefix) are treated as filters which
|
||||
will only run tests whose name matches one of those strings. The filter will
|
||||
match any substring found in the full path of the test function. For example,
|
||||
if the test function `it_works` is located in the module
|
||||
`utils::paths::tests`, then any of the filters `works`, `path`, `utils::`, or
|
||||
`utils::paths::tests::it_works` will match that test.
|
||||
|
||||
See [Selection options](#selection-options) for more options to control which
|
||||
tests are run.
|
||||
|
||||
### Action options
|
||||
|
||||
The following options perform different actions other than running tests.
|
||||
|
||||
#### `--list`
|
||||
|
||||
Prints a list of all tests and benchmarks. Does not run any of the tests.
|
||||
[Filters](#filters) can be used to list only matching tests.
|
||||
|
||||
#### `-h`, `--help`
|
||||
|
||||
Displays usage information and command-line options.
|
||||
|
||||
### Selection options
|
||||
|
||||
The following options change how tests are selected.
|
||||
|
||||
#### `--test`
|
||||
|
||||
This is the default mode where all tests will be run as well as running all
|
||||
benchmarks with only a single iteration (to ensure the benchmark works,
|
||||
without taking the time to actually perform benchmarking). This can be
|
||||
combined with the `--bench` flag to run both tests and perform full
|
||||
benchmarking.
|
||||
|
||||
#### `--bench`
|
||||
|
||||
This runs in a mode where tests are ignored, and only runs benchmarks. This
|
||||
can be combined with `--test` to run both benchmarks and tests.
|
||||
|
||||
#### `--exact`
|
||||
|
||||
This forces [filters](#filters) to match the full path of the test exactly.
|
||||
For example, if the test `it_works` is in the module `utils::paths::tests`,
|
||||
then only the string `utils::paths::tests::it_works` will match that test.
|
||||
|
||||
#### `--skip` _FILTER_
|
||||
|
||||
Skips any tests whose name contains the given _FILTER_ string. This flag may
|
||||
be passed multiple times.
|
||||
|
||||
#### `--ignored`
|
||||
|
||||
Runs only tests that are marked with the [`ignore`
|
||||
attribute][attribute-ignore].
|
||||
|
||||
#### `--include-ignored`
|
||||
|
||||
Runs both [ignored](#--ignored) and non-ignored tests.
|
||||
|
||||
#### `--exclude-should-panic`
|
||||
|
||||
Excludes tests marked with the [`should_panic`
|
||||
attribute][attribute-should_panic].
|
||||
|
||||
⚠️ 🚧 This option is [unstable](#unstable-options), and requires the `-Z
|
||||
unstable-options` flag. See [tracking issue
|
||||
#82348](https://github.com/rust-lang/rust/issues/82348) for more information.
|
||||
|
||||
### Execution options
|
||||
|
||||
The following options affect how tests are executed.
|
||||
|
||||
#### `--test-threads` _NUM_THREADS_
|
||||
|
||||
Sets the number of threads to use for running tests in parallel. By default,
|
||||
uses the amount of concurrency available on the hardware as indicated by
|
||||
[`available_concurrency`].
|
||||
|
||||
This can also be specified with the `RUST_TEST_THREADS` environment variable.
|
||||
|
||||
#### `--force-run-in-process`
|
||||
|
||||
Forces the tests to run in a single process when using the [`abort` panic
|
||||
strategy][panic-strategy].
|
||||
|
||||
⚠️ 🚧 This only works with the unstable [`-Z panic-abort-tests`] option, and
|
||||
requires the `-Z unstable-options` flag. See [tracking issue
|
||||
#67650](https://github.com/rust-lang/rust/issues/67650) for more information.
|
||||
|
||||
#### `--ensure-time`
|
||||
|
||||
⚠️ 🚧 This option is [unstable](#unstable-options), and requires the `-Z
|
||||
unstable-options` flag. See [tracking issue
|
||||
#64888](https://github.com/rust-lang/rust/issues/64888) and the [unstable
|
||||
docs](../../unstable-book/compiler-flags/report-time.html) for more information.
|
||||
|
||||
### Output options
|
||||
|
||||
The following options affect the output behavior.
|
||||
|
||||
#### `-q`, `--quiet`
|
||||
|
||||
Displays one character per test instead of one line per test. This is an alias
|
||||
for [`--format=terse`](#--format-format).
|
||||
|
||||
#### `--nocapture`
|
||||
|
||||
Does not capture the stdout and stderr of the test, and allows tests to print
|
||||
to the console. Usually the output is captured, and only displayed if the test
|
||||
fails.
|
||||
|
||||
This may also be specified by setting the `RUST_TEST_NOCAPTURE` environment
|
||||
variable set to anything but `0`.
|
||||
|
||||
#### `--show-output`
|
||||
|
||||
Displays the stdout and stderr of successful tests after all tests have run.
|
||||
|
||||
Contrast this with [`--nocapture`](#--nocapture) which allows tests to print
|
||||
*while they are running*, which can cause interleaved output if there are
|
||||
multiple tests running in parallel, `--show-output` ensures the output is
|
||||
contiguous, but requires waiting for all tests to finish.
|
||||
|
||||
#### `--color` _COLOR_
|
||||
|
||||
Control when colored terminal output is used. Valid options:
|
||||
|
||||
* `auto`: Colorize if stdout is a tty and [`--nocapture`](#--nocapture) is not
|
||||
used. This is the default.
|
||||
* `always`: Always colorize the output.
|
||||
* `never`: Never colorize the output.
|
||||
|
||||
#### `--format` _FORMAT_
|
||||
|
||||
Controls the format of the output. Valid options:
|
||||
|
||||
* `pretty`: This is the default format, with one line per test.
|
||||
* `terse`: Displays only a single character per test. [`--quiet`](#-q---quiet)
|
||||
is an alias for this option.
|
||||
* `json`: Emits JSON objects, one per line. ⚠️ 🚧 This option is
|
||||
[unstable](#unstable-options), and requires the `-Z unstable-options` flag.
|
||||
See [tracking issue #49359](https://github.com/rust-lang/rust/issues/49359)
|
||||
for more information.
|
||||
|
||||
#### `--logfile` _PATH_
|
||||
|
||||
Writes the results of the tests to the given file.
|
||||
|
||||
#### `--report-time` _FORMAT_
|
||||
|
||||
⚠️ 🚧 This option is [unstable](#unstable-options), and requires the `-Z
|
||||
unstable-options` flag. See [tracking issue
|
||||
#64888](https://github.com/rust-lang/rust/issues/64888) and the [unstable
|
||||
docs](../../unstable-book/compiler-flags/report-time.html) for more information.
|
||||
|
||||
### Unstable options
|
||||
|
||||
Some CLI options are added in an "unstable" state, where they are intended for
|
||||
experimentation and testing to determine if the option works correctly, has
|
||||
the right design, and is useful. The option may not work correctly, break, or
|
||||
change at any time. To signal that you acknowledge that you are using an
|
||||
unstable option, they require passing the `-Z unstable-options` command-line
|
||||
flag.
|
||||
|
||||
## Benchmarks
|
||||
|
||||
The libtest harness supports running benchmarks for functions annotated with
|
||||
the `#[bench]` attribute. Benchmarks are currently unstable, and only
|
||||
available on the [nightly channel]. More information may be found in the
|
||||
[unstable book][bench-docs].
|
||||
|
||||
## Custom test frameworks
|
||||
|
||||
Experimental support for using custom test harnesses is available on the
|
||||
[nightly channel]. See [tracking issue
|
||||
#50297](https://github.com/rust-lang/rust/issues/50297) and the
|
||||
[custom_test_frameworks documentation] for more information.
|
||||
|
||||
[`--test` option]: ../command-line-arguments.md#option-test
|
||||
[`-Z panic-abort-tests`]: https://github.com/rust-lang/rust/issues/67650
|
||||
[`available_concurrency`]: ../../std/thread/fn.available_concurrency.html
|
||||
[`cargo test`]: ../../cargo/commands/cargo-test.html
|
||||
[`libtest`]: ../../test/index.html
|
||||
[`main` function]: ../../reference/crates-and-source-files.html#main-functions
|
||||
[`Result`]: ../../std/result/index.html
|
||||
[`test` cfg option]: ../../reference/conditional-compilation.html#test
|
||||
[attribute-ignore]: ../../reference/attributes/testing.html#the-ignore-attribute
|
||||
[attribute-should_panic]: ../../reference/attributes/testing.html#the-should_panic-attribute
|
||||
[attribute-test]: ../../reference/attributes/testing.html#the-test-attribute
|
||||
[bench-docs]: ../../unstable-book/library-features/test.html
|
||||
[Cargo]: ../../cargo/index.html
|
||||
[crate type]: ../../reference/linkage.html
|
||||
[custom_test_frameworks documentation]: ../../unstable-book/language-features/custom-test-frameworks.html
|
||||
[nightly channel]: ../../book/appendix-07-nightly-rust.html
|
||||
[panic-strategy]: ../../book/ch09-01-unrecoverable-errors-with-panic.html
|
||||
[panic]: ../../book/ch09-01-unrecoverable-errors-with-panic.html
|
||||
[Testing Chapter]: ../../book/ch11-00-testing.html
|
||||
Loading…
Reference in New Issue
Block a user