libtest: also measure time in Miri
A long time ago we disabled timekeeping of the default test harness in Miri, as otherwise it would fail to run without `-Zmiri-disable-isolation`. However, since then Miri gained a "fake clock" that lets it present some deterministic notion of time when isolation is enabled.
So we could now let libtest do timekeeping again when running in Miri. That's nice as it can help detect tests that run too long. However it can also be confusing as the results with isolation can be quite different than the real time.
``@rust-lang/miri`` what do you think?
Previously we left the panic hook we allocated
on main termination. Doing so makes Valgrind
report it as a reachable unfreed block.
In order to fix that use `panic::take_hook()` before
examining test results.
Example backtrace:
```
==146594== 16 bytes in 1 blocks are still reachable in loss record 1 of 1
==146594== at 0x4A390C5: malloc (vg_replace_malloc.c:442)
==146594== by 0x151336: alloc (alloc.rs:98)
==146594== by 0x151336: alloc_impl (alloc.rs:181)
==146594== by 0x151336: allocate (alloc.rs:241)
==146594== by 0x151336: exchange_malloc (alloc.rs:330)
==146594== by 0x151336: new<test::test_main::{closure_env#0}> (boxed.rs:217)
==146594== by 0x151336: test::test_main (lib.rs:124)
==146594== by 0x1522F9: test::test_main_static (lib.rs:160)
==146594== by 0x11E102: reachable_block_with_cargo_test::main (lib.rs:1)
==146594== by 0x11EABA: core::ops::function::FnOnce::call_once (function.rs:250)
==146594== by 0x11E76D: std::sys_common::backtrace::__rust_begin_short_backtrace (backtrace.rs:154)
==146594== by 0x11DFC0: std::rt::lang_start::{{closure}} (rt.rs:166)
==146594== by 0x177D3A: call_once<(), (dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe)> (function.rs:284)
==146594== by 0x177D3A: do_call<&(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe), i32> (panicking.rs:504)
==146594== by 0x177D3A: try<i32, &(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe)> (panicking.rs:468)
==146594== by 0x177D3A: catch_unwind<&(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe), i32> (panic.rs:142)
==146594== by 0x177D3A: {closure#2} (rt.rs:148)
==146594== by 0x177D3A: do_call<std::rt::lang_start_internal::{closure_env#2}, isize> (panicking.rs:504)
==146594== by 0x177D3A: try<isize, std::rt::lang_start_internal::{closure_env#2}> (panicking.rs:468)
==146594== by 0x177D3A: catch_unwind<std::rt::lang_start_internal::{closure_env#2}, isize> (panic.rs:142)
==146594== by 0x177D3A: std::rt::lang_start_internal (rt.rs:148)
==146594== by 0x11DF99: std::rt::lang_start (rt.rs:165)
```
Signed-off-by: Tal Gelbard <talgelbard1@gmail.com>
This commit updates the libtest conditionals to use `std::time::Instant`
on WASI targets where it's implemented. Previously all wasm targets
wouldn't use this type.
Previously, libtest would wait until all tests finished running to print the progress, which made it
annoying to run many tests at once (since you don't know which have failed). Change it to print the
names as soon as they fail.
This also adds a test for the terse output; previously it was untested.
This also adds changes in the rust test suite in order to get a few of them to
pass.
Co-authored-by: Frank Laub <flaub@risc0.com>
Co-authored-by: Urgau <3616612+Urgau@users.noreply.github.com>
libtest: Fix padding of benchmarks run as tests
### Summary
The first commit adds regression tests for libtest padding.
The second commit fixes padding for benches run as tests and updates the blessed output of the regression tests to make it clear what effect the fix has on padding.
Closes#104092 which is **E-help-wanted** and **regression-from-stable-to-stable**
### More details
Before this fix we applied padding _before_ manually doing what `convert_benchmarks_to_tests()` does which affects padding calculations. Instead use `convert_benchmarks_to_tests()` first if applicable and then apply padding afterwards so it becomes correct.
Benches should only be padded when run as benches to make it easy to compare the benchmark numbers. Not when run as tests.
r? `@ghost` until CI passes.
detects redundant imports that can be eliminated.
for #117772 :
In order to facilitate review and modification, split the checking code and
removing redundant imports code into two PR.
Before this fix we applied padding before manually doing what
`convert_benchmarks_to_tests()` does. Instead use
`convert_benchmarks_to_tests()` if applicable and then apply padding
afterwards so it becomes correct. (Benches should only be padded when
run as benches to make it easy to compare the benchmark numbers.)
test_get_dbpath_for_term(): handle non-utf8 paths (fix FIXME)
Removes a FIXME for #9639
Part of #44366 which is E-help-wanted
The remaining two FIXMEs for #9639 are considerably more complicated, so I will create separate PRs for them.
It lints against features that are inteded to be internal to the
compiler and standard library. Implements MCP #596.
We allow `internal_features` in the standard library and compiler as those
use many features and this _is_ the standard library from the "internal to the compiler and
standard library" after all.
Marking some features as internal wasn't exactly the most scientific approach, I just marked some
mostly obvious features. While there is a categorization in the macro,
it's not very well upheld (should probably be fixed in another PR).
We always pass `-Ainternal_features` in the testsuite
About 400 UI tests and several other tests use internal features.
Instead of throwing the attribute on each one, just always allow them.
There's nothing wrong with testing internal features^^
Test benchmarks with `-Z panic-abort-tests`
During test execution, when a `#[bench]` benchmark is encountered it's executed once to check whether it works. Unfortunately that was not compatible with `-Z panic-abort-tests`: the feature works by spawning a subprocess for each test, which prevents the use of dynamic tests as we cannot pass closures to child processes, and before this PR the conversion from benchmark to test was done by turning benchmarks into dynamic tests whose closures execute the benchmark once.
The approach this PR took was to add two new kinds of `TestFn`s: `StaticBenchAsTestFn` and `DynBenchAsTestFn` (⚠️ **this is a breaking change** ⚠️). With that change, a `StaticBenchFn` can be converted into a `StaticBenchAsTestFn` without creating dynamic tests, and making it possible to test `#[bench]` functions with `-Z panic-abort-tests`. The subprocess test runner also had to be updated to perform the conversion from benchmark to test when appropriate.
Along with the bug fix, in the first commit I refactored how tests are executed: rather than executing the test function in multiple places across `libtest`, there is now a private `TestFn::into_runnable()` method, which returns either a `RunnableTest` or `RunnableBench`, on which you can call the `run()` method. This simplified the rest of the changes in the PR.
This PR is best reviewed commit-by-commit.
Fixes https://github.com/rust-lang/rust/issues/73509
Before this commit, both static and dynamic benches were converted to a
DynTestFn, with a boxed closure that ran the benchmarks exactly once.
While this worked, it conflicted with -Z panic-abort-tests as the flag
does not support dynamic tests. With this change, a StaticBenchFn is
converted to a StaticBenchAsTestFn, avoiding any dynamic test creation.
DynBenchFn is also converted to DynBenchAsTestFn for completeness.
Before this commit, tests were invoked in multiple places, especially
due to `-Z panic-abort-tests`, and adding a new test kind meant having
to chase down and update all these places.
This commit creates a new Runnable enum, and its children RunnableTest
and RunnableBench. The rest of the harness will now pass around the enum
rather than constructing and passing around boxed functions. The enum
has two children enums because invoking tests and invoking benchmarks
requires different parameters.
Add a `sysroot` crate to represent the standard library crates
This adds a dummy crate named `sysroot` to represent the standard library target instead of using the `test` crate. This allows the removal of `proc_macro` as a dependency of `test` allowing these 2 crates to build in parallel saving around 9 seconds locally.
By placing the stdout in a CDATA block we avoid almost all escaping, as
there's only two byte sequences you can't sneak into a CDATA and you can
handle that with some only slightly regrettable CDATA-splitting. I've
done this in at least two other implementations of the junit xml format
over the years and it's always worked out. The only quirk new to this
(for me) is smuggling newlines as 
 to avoid literal newlines in the
output.
