From 6121d82d476c43b533a24e3ec697d681199df497 Mon Sep 17 00:00:00 2001 From: Steve Klabnik Date: Fri, 25 Jul 2014 19:48:05 -0400 Subject: [PATCH] Guide: testing --- src/doc/guide.md | 575 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 574 insertions(+), 1 deletion(-) diff --git a/src/doc/guide.md b/src/doc/guide.md index 2216d829e4b..2ee926d08a1 100644 --- a/src/doc/guide.md +++ b/src/doc/guide.md @@ -2808,9 +2808,582 @@ By just bringing the module into scope, we can keep one level of namespacing. # Testing +Traditionally, testing has not been a strong suit of most systems programming +languages. Rust, however, has very basic testing built into the language +itself. While automated testing cannot prove that your code is bug-free, it is +useful for verifying that certain behaviors work as intended. + +Here's a very basic test: + +```{rust} +#[test] +fn is_one_equal_to_one() { + assert_eq!(1i, 1i); +} +``` + +You may notice something new: that `#[test]`. Before we get into the mechanics +of testing, let's talk about attributes. + ## Attributes -## Stability Markers +Rust's testing system uses **attribute**s to mark which functions are tests. +Attributes can be placed on any Rust **item**. Remember how most things in +Rust are an expression, but `let` is not? Item declarations are also not +expressions. Here's a list of things that qualify as an item: + +* functions +* modules +* type definitions +* structures +* enumerations +* static items +* traits +* implementations + +You haven't learned about all of these things yet, but that's the list. As +you can see, functions are at the top of it. + +Attributes can appear in three ways: + +1. A single identifier, the attribute name. `#[test]` is an example of this. +2. An identifier followed by an equals sign (`=`) and a literal. `#[cfg=test]` + is an example of this. +3. An identifier followed by a parenthesized list of sub-attribute arguments. + `#[cfg(unix, target_word_size = "32")]` is an example of this, where one of + the sub-arguments is of the second kind. + +There are a number of different kinds of attributes, enough that we won't go +over them all here. Before we talk about the testing-specific attributes, I +want to call out one of the most important kinds of attributes: stability +markers. + +## Stability attributes + +Rust provides six attributes to indicate the stability level of various +parts of your library. The six levels are: + +* deprecated: this item should no longer be used. No guarantee of backwards + compatibility. +* experimental: This item was only recently introduced or is otherwise in a + state of flux. It may change significantly, or even be removed. No guarantee + of backwards-compatibility. +* unstable: This item is still under development, but requires more testing to + be considered stable. No guarantee of backwards-compatibility. +* stable: This item is considered stable, and will not change significantly. + Guarantee of backwards-compatibility. +* frozen: This item is very stable, and is unlikely to change. Guarantee of + backwards-compatibility. +* locked: This item will never change unless a serious bug is found. Guarantee + of backwards-compatibility. + +All of Rust's standard library uses these attribute markers to communicate +their relative stability, and you should use them in your code, as well. +There's an associated attribute, `warn`, that allows you to warn when you +import an item marked with certain levels: deprecated, experimental and +unstable. For now, only deprecated warns by default, but this will change once +the standard library has been stabilized. + +You can use the `warn` attribute like this: + +```{rust,ignore} +#![warn(unstable)] +``` + +And later, when you import a crate: + +```{rust,ignore} +extern crate some_crate; +``` + +You'll get a warning if you use something marked unstable. + +You may have noticed an exclamation point in the `warn` attribute declaration. +The `!` in this attribute means that this attribute applies to the enclosing +item, rather than to the item that follows the attribute. So this `warn` +attribute declaration applies to the enclosing crate itself, rather than +to whatever item statement follows it: + +```{rust,ignore} +// applies to the crate we're in +#![warn(unstable)] + +extern crate some_crate; + +// applies to the following `fn`. +#[test] +fn a_test() { + // ... +} +``` + +## Writing tests + +Let's write a very simple crate in a test-driven manner. You know the drill by +now: make a new project: + +```{bash,ignore} +$ cd ~/projects +$ mkdir testing +$ cd testing +$ mkdir test +``` + +In `src/main.rs`: + +```{rust} +fn main() { + println!("Hello, world!"); +} +``` + +And in `Cargo.toml`: + +```{notrust,ignore} +[package] + +name = "testing" +version = "0.1.0" +authors = [ "someone@example.com" ] +``` + +And try it out: + +```{notrust,ignore} +$ cargo run + Compiling testing v0.1.0 (file:/home/you/projects/testing) +Hello, world! +$ +``` + +Great. Rust's infrastructure supports tests in two sorts of places, and they're +for two kinds of tests: you include **unit test**s inside of the crate itself, +and you place **integration test**s inside a `tests` directory. "Unit tests" +are small tests that test one focused unit, "integration tests" tests multiple +units in integration. That said, this is a social convention, they're no different +in syntax. Let's make a `tests` directory: + +```{bash,ignore} +$ mkdir tests +``` + +Next, let's create an integration test in `tests/lib.rs`: + +```{rust,no_run} +#[test] +fn foo() { + assert!(false); +} +``` + +It doesn't matter what you name your test functions, though it's nice if +you give them descriptive names. You'll see why in a moment. We then use a +macro, `assert!`, to assert that something is true. In this case, we're giving +it `false`, so this test should fail. Let's try it! + +```{notrust,ignore} +$ cargo test + Compiling testing v0.1.0 (file:/home/you/projects/testing) +/home/you/projects/testing/src/main.rs:1:1: 3:2 warning: code is never used: `main`, #[warn(dead_code)] on by default +/home/you/projects/testing/src/main.rs:1 fn main() { +/home/you/projects/testing/src/main.rs:2 println!("Hello, world"); +/home/you/projects/testing/src/main.rs:3 } + +running 0 tests + +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured + + +running 1 test +test foo ... FAILED + +failures: + +---- foo stdout ---- + task 'foo' failed at 'assertion failed: false', /home/you/projects/testing/tests/lib.rs:3 + + + +failures: + foo + +test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured + +task '
' failed at 'Some tests failed', /home/you/src/rust/src/libtest/lib.rs:242 +``` + +Lots of output! Let's break this down: + +```{notrust,ignore} +$ cargo test + Compiling testing v0.1.0 (file:/home/you/projects/testing) +``` + +You can run all of your tests with `cargo test`. This runs both your tests in +`tests`, as well as the tests you put inside of your crate. + +```{notrust,ignore} +/home/you/projects/testing/src/main.rs:1:1: 3:2 warning: code is never used: `main`, #[warn(dead_code)] on by default +/home/you/projects/testing/src/main.rs:1 fn main() { +/home/you/projects/testing/src/main.rs:2 println!("Hello, world"); +/home/you/projects/testing/src/main.rs:3 } +``` + +Rust has a **lint** called 'warn on dead code' used by default. A lint is a +bit of code that checks your code, and can tell you things about it. In this +case, Rust is warning us that we've written some code that's never used: our +`main` function. Of course, since we're running tests, we don't use `main`. +We'll turn this lint off for just this function soon. For now, just ignore this +output. + +```{notrust,ignore} +running 0 tests + +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured +``` + +Wait a minute, zero tests? Didn't we define one? Yup. This output is from +attempting to run the tests in our crate, of which we don't have any. +You'll note that Rust reports on several kinds of tests: passed, failed, +ignored, and measured. The 'measured' tests refer to benchmark tests, which +we'll cover soon enough! + +```{notrust,ignore} +running 1 test +test foo ... FAILED +``` + +Now we're getting somewhere. Remember when we talked about naming our tests +with good names? This is why. Here, it says 'test foo' because we called our +test 'foo.' If we had given it a good name, it'd be more clear which test +failed, especially as we accumulate more tests. + +```{notrust,ignore} +failures: + +---- foo stdout ---- + task 'foo' failed at 'assertion failed: false', /home/you/projects/testing/tests/lib.rs:3 + + + +failures: + foo + +test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured + +task '
' failed at 'Some tests failed', /home/you/src/rust/src/libtest/lib.rs:242 +``` + +After all the tests run, Rust will show us any output from our failed tests. +In this instance, Rust tells us that our assertion failed, with false. This was +what we expected. + +Whew! Let's fix our test: + +```{rust} +#[test] +fn foo() { + assert!(true); +} +``` + +And then try to run our tests again: + +```{notrust,ignore} +$ cargo test + Compiling testing v0.1.0 (file:/home/you/projects/testing) +/home/you/projects/testing/src/main.rs:1:1: 3:2 warning: code is never used: `main`, #[warn(dead_code)] on by default +/home/you/projects/testing/src/main.rs:1 fn main() { +/home/you/projects/testing/src/main.rs:2 println!("Hello, world"); +/home/you/projects/testing/src/main.rs:3 } + +running 0 tests + +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured + + +running 1 test +test foo ... ok + +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured +$ +``` + +Nice! Our test passes, as we expected. Let's get rid of that warning for our `main` +function. Change your `src/main.rs` to look like this: + +```{rust} +#[cfg(not(test))] +fn main() { + println!("Hello, world"); +} +``` + +This attribute combines two things: `cfg` and `not`. The `cfg` attribute allows +you to conditionally compile code based on something. The following item will +only be compiled if the configuration says it's true. And when Cargo compiles +our tests, it sets things up so that `cfg(test)` is true. But we want to only +include `main` when it's _not_ true. So we use `not` to negate things: +`cfg(not(test))` will only compile our code when the `cfg(test)` is false. + +With this attribute, we won't get the warning: + +```{notrust,ignore} +$ cargo test + Compiling testing v0.1.0 (file:/home/you/projects/testing) + +running 0 tests + +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured + + +running 1 test +test foo ... ok + +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured +``` + +Nice. Okay, let's write a real test now. Change your `tests/lib.rs` +to look like this: + +```{rust,ignore} +#[test] +fn math_checks_out() { + let result = add_three_times_four(5i); + + assert_eq!(32i, result); +} +``` + +And try to run the test: + +```{notrust,ignore} +$ cargo test + Compiling testing v0.1.0 (file:/home/youg/projects/testing) +/home/youg/projects/testing/tests/lib.rs:3:18: 3:38 error: unresolved name `add_three_times_four`. +/home/youg/projects/testing/tests/lib.rs:3 let result = add_three_times_four(5i); + ^~~~~~~~~~~~~~~~~~~~ +error: aborting due to previous error +Build failed, waiting for other jobs to finish... +Could not compile `testing`. + +To learn more, run the command again with --verbose. +``` + +Rust can't find this function. That makes sense, as we didn't write it yet! + +In order to share this codes with our tests, we'll need to make a library crate. +This is also just good software design: as we mentioned before, it's a good idea +to put most of your functionality into a library crate, and have your executable +crate use that library. This allows for code re-use. + +To do that, we'll need to make a new module. Make a new file, `src/lib.rs`, +and put this in it: + +```{rust} +fn add_three_times_four(x: int) -> int { + (x + 3) * 4 +} +``` + +We're calling this file `lib.rs` because it has the same name as our project, +and so it's named this, by convention. + +We'll then need to use this crate in our `src/main.rs`: + +```{rust,ignore} +extern crate testing; + +#[cfg(not(test))] +fn main() { + println!("Hello, world"); +} +``` + +Finally, let's import this function in our `tests/lib.rs`: + +```{rust,ignore} +extern crate testing; +use testing::add_three_times_four; + +#[test] +fn math_checks_out() { + let result = add_three_times_four(5i); + + assert_eq!(32i, result); +} +``` + +Let's give it a run: + +```{ignore,notrust} +$ cargo test + Compiling testing v0.1.0 (file:/home/you/projects/testing) + +running 0 tests + +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured + + +running 0 tests + +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured + + +running 1 test +test math_checks_out ... ok + +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured +``` + +Great! One test passed. We've got an integration test showing that our public +method works, but maybe we want to test some of the internal logic as well. +While this function is simple, if it were more complicated, you can imagine +we'd need more tests. So let's break it up into two helper functions, and +write some unit tests to test those. + +Change your `src/lib.rs` to look like this: + +```{rust,ignore} +pub fn add_three_times_four(x: int) -> int { + times_four(add_three(x)) +} + +fn add_three(x: int) -> int { x + 3 } + +fn times_four(x: int) -> int { x * 4 } +``` + +If you run `cargo test`, you should get the same output: + +```{ignore,notrust} +$ cargo test + Compiling testing v0.1.0 (file:/home/you/projects/testing) + +running 0 tests + +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured + + +running 0 tests + +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured + + +running 1 test +test math_checks_out ... ok + +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured +``` + +If we tried to write a test for these two new functions, it wouldn't +work. For example: + +```{rust,ignore} +extern crate testing; +use testing::add_three_times_four; +use testing::add_three; + +#[test] +fn math_checks_out() { + let result = add_three_times_four(5i); + + assert_eq!(32i, result); +} + +#[test] +fn test_add_three() { + let result = add_three(5i); + + assert_eq!(8i, result); +} +``` + +We'd get this error: + +```{notrust,ignore} + Compiling testing v0.1.0 (file:/home/you/projects/testing) +/home/you/projects/testing/tests/lib.rs:3:5: 3:24 error: function `add_three` is private +/home/you/projects/testing/tests/lib.rs:3 use testing::add_three; + ^~~~~~~~~~~~~~~~~~~ +``` + +Right. It's private. So external, integration tests won't work. We need a +unit test. Open up your `src/lib.rs` and add this: + +```{rust,ignore} +pub fn add_three_times_four(x: int) -> int { + times_four(add_three(x)) +} + +fn add_three(x: int) -> int { x + 3 } + +fn times_four(x: int) -> int { x * 4 } + +#[cfg(test)] +mod test { + use super::add_three; + use super::add_four; + + #[test] + fn test_add_three() { + let result = add_three(5i); + + assert_eq!(8i, result); + } + + #[test] + fn test_times_four() { + let result = times_four(5i); + + assert_eq!(20i, result); + } +} +``` + +Let's give it a shot: + +```{ignore,notrust} +$ cargo test + Compiling testing v0.1.0 (file:/home/you/projects/testing) + +running 1 test +test test::test_times_four ... ok +test test::test_add_three ... ok + +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured + + +running 0 tests + +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured + + +running 1 test +test math_checks_out ... ok + +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured +``` + +Cool! We now have two tests of our internal functions. You'll note that there +are three sets of output now: one for `src/main.rs`, one for `src/lib.rs`, and +one for `tests/lib.rs`. There's one interesting thing that we haven't talked +about yet, and that's these lines: + +```{rust,ignore} +use super::add_three; +use super::add_four; +``` + +Because we've made a nested module, we can import functions from the parent +module by using `super`. Sub-modules are allowed to 'see' private functions in +the parent. We sometimes call this usage of `use` a 're-export,' because we're +exporting the name again, somewhere else. + +We've now covered the basics of testing. Rust's tools are primitive, but they +work well in the simple cases. There are some Rustaceans working on building +more complicated frameworks on top of all of this, but thery're just starting +out. # Pointers