Tools, tests, and experimenting with MIR-derived coverage counters
Adds a new mir_dump output file in HTML/CSS to visualize code regions
and the MIR features that they came from (including overlapping spans).
See example below:
Includes a basic, MIR-block-based implementation of coverage injection,
available via `-Zexperimental-coverage`. This implementation has known
flaws and omissions, but is simple enough to validate the new tools and
tests.
The existing `-Zinstrument-coverage` option currently enables
function-level coverage only, which at least appears to generate
accurate coverage reports at that level.
Experimental coverage is not accurate at this time. When branch coverage
works as intended, the `-Zexperimental-coverage` option should be
removed.
This PR replaces the bulk of PR #75828, with the remaining parts of
that PR distributed among other separate and indentpent PRs.
This PR depends on three of those other PRs: #76000, #76002, and
Rust compiler MCP rust-lang/compiler-team#278
Relevant issue: #34701 - Implement support for LLVMs code coverage
instrumentation
2020-08-27 21:13:04 +00:00
|
|
|
# needs-profiler-support
|
2020-08-24 09:58:04 +00:00
|
|
|
# ignore-windows-gnu
|
|
|
|
|
|
|
|
|
|
# FIXME(mati865): MinGW GCC miscompiles compiler-rt profiling library but with Clang it works
|
|
|
|
|
# properly. Since we only have GCC on the CI ignore the test for now.
|
Tools, tests, and experimenting with MIR-derived coverage counters
Adds a new mir_dump output file in HTML/CSS to visualize code regions
and the MIR features that they came from (including overlapping spans).
See example below:
Includes a basic, MIR-block-based implementation of coverage injection,
available via `-Zexperimental-coverage`. This implementation has known
flaws and omissions, but is simple enough to validate the new tools and
tests.
The existing `-Zinstrument-coverage` option currently enables
function-level coverage only, which at least appears to generate
accurate coverage reports at that level.
Experimental coverage is not accurate at this time. When branch coverage
works as intended, the `-Zexperimental-coverage` option should be
removed.
This PR replaces the bulk of PR #75828, with the remaining parts of
that PR distributed among other separate and indentpent PRs.
This PR depends on three of those other PRs: #76000, #76002, and
Rust compiler MCP rust-lang/compiler-team#278
Relevant issue: #34701 - Implement support for LLVMs code coverage
instrumentation
2020-08-27 21:13:04 +00:00
|
|
|
|
2020-10-05 16:26:30 +00:00
|
|
|
-include ../coverage/coverage_tools.mk
|
Tools, tests, and experimenting with MIR-derived coverage counters
Adds a new mir_dump output file in HTML/CSS to visualize code regions
and the MIR features that they came from (including overlapping spans).
See example below:
Includes a basic, MIR-block-based implementation of coverage injection,
available via `-Zexperimental-coverage`. This implementation has known
flaws and omissions, but is simple enough to validate the new tools and
tests.
The existing `-Zinstrument-coverage` option currently enables
function-level coverage only, which at least appears to generate
accurate coverage reports at that level.
Experimental coverage is not accurate at this time. When branch coverage
works as intended, the `-Zexperimental-coverage` option should be
removed.
This PR replaces the bulk of PR #75828, with the remaining parts of
that PR distributed among other separate and indentpent PRs.
This PR depends on three of those other PRs: #76000, #76002, and
Rust compiler MCP rust-lang/compiler-team#278
Relevant issue: #34701 - Implement support for LLVMs code coverage
instrumentation
2020-08-27 21:13:04 +00:00
|
|
|
|
2020-12-01 07:37:49 +00:00
|
|
|
BASEDIR=../coverage-reports
|
2020-10-05 16:26:30 +00:00
|
|
|
SOURCEDIR=../coverage
|
Tools, tests, and experimenting with MIR-derived coverage counters
Adds a new mir_dump output file in HTML/CSS to visualize code regions
and the MIR features that they came from (including overlapping spans).
See example below:
Includes a basic, MIR-block-based implementation of coverage injection,
available via `-Zexperimental-coverage`. This implementation has known
flaws and omissions, but is simple enough to validate the new tools and
tests.
The existing `-Zinstrument-coverage` option currently enables
function-level coverage only, which at least appears to generate
accurate coverage reports at that level.
Experimental coverage is not accurate at this time. When branch coverage
works as intended, the `-Zexperimental-coverage` option should be
removed.
This PR replaces the bulk of PR #75828, with the remaining parts of
that PR distributed among other separate and indentpent PRs.
This PR depends on three of those other PRs: #76000, #76002, and
Rust compiler MCP rust-lang/compiler-team#278
Relevant issue: #34701 - Implement support for LLVMs code coverage
instrumentation
2020-08-27 21:13:04 +00:00
|
|
|
|
2020-11-04 05:45:32 +00:00
|
|
|
# The `llvm-cov show` flag `--debug`, used to generate the `counters` output files, is only enabled
|
2020-11-08 19:06:05 +00:00
|
|
|
# if LLVM assertions are enabled. Requires Rust config `llvm/optimize` and not
|
|
|
|
|
# `llvm/release_debuginfo`. Note that some CI builds disable debug assertions (by setting
|
|
|
|
|
# `NO_LLVM_ASSERTIONS=1`), so it is not OK to fail the test, but `bless`ed test results cannot be
|
|
|
|
|
# generated without debug assertions.
|
2020-11-24 19:50:24 +00:00
|
|
|
LLVM_COV_DEBUG := $(shell \
|
|
|
|
|
"$(LLVM_BIN_DIR)"/llvm-cov show --debug 2>&1 | \
|
|
|
|
|
grep -q "Unknown command line argument '--debug'"; \
|
|
|
|
|
echo $$?)
|
2020-11-08 19:06:05 +00:00
|
|
|
ifeq ($(LLVM_COV_DEBUG), 1)
|
2020-11-04 05:45:32 +00:00
|
|
|
DEBUG_FLAG=--debug
|
|
|
|
|
endif
|
|
|
|
|
|
2020-12-02 17:43:46 +00:00
|
|
|
# FIXME(richkadel): I'm adding `--ignore-filename-regex=` line(s) for specific test(s) that produce
|
|
|
|
|
# `llvm-cov` results for multiple files (for example `uses_crate.rs` and `used_crate/mod.rs`) as a
|
|
|
|
|
# workaround for two problems causing tests to fail on Windows:
|
|
|
|
|
#
|
|
|
|
|
# 1. When multiple files appear in the `llvm-cov show` results, each file's coverage results can
|
|
|
|
|
# appear in different a different order. Whether this is random or, somehow, platform-specific,
|
|
|
|
|
# the Windows output flips the order of the files, compared to Linux. In the `uses_crate.rs`
|
|
|
|
|
# test, the only test-unique (interesting) results we care about are the results for only one
|
|
|
|
|
# of the two files, `mod/uses_crate.rs`, so the workaround is to ignore all but this one file.
|
|
|
|
|
# In the future, we may want a more sophisticated solution that splits apart `llvm-cov show`
|
|
|
|
|
# results into separate results files for each result (taking care not to create new file
|
|
|
|
|
# paths that might be too long for Windows MAX_PATH limits when creating these new sub-results,
|
|
|
|
|
# as well).
|
|
|
|
|
# 2. When multiple files appear in the `llvm-cov show` results, the results for each file are
|
|
|
|
|
# prefixed with their filename, including platform-specific path separators (`\` for Windows,
|
|
|
|
|
# and `/` everywhere else). This could be filtered or normalized of course, but by ignoring
|
|
|
|
|
# coverage results for all but one of the file, the filenames are no longer included anyway.
|
|
|
|
|
# If this changes (if/when we decide to support `llvm-cov show` results for multiple files),
|
|
|
|
|
# the file path separator differences may need to be addressed.
|
|
|
|
|
#
|
|
|
|
|
# Since this is only a workaround, I decided to implement the override by adding an option for
|
|
|
|
|
# each file to be ignored, using a `--ignore-filename-regex=` entry for each one, rather than
|
|
|
|
|
# implement some more sophisticated solution with a new custom test directive in the test file
|
|
|
|
|
# itself (similar to `expect-exit-status`) because that would add a lot of complexity and still
|
|
|
|
|
# be a workaround, with the same result, with no benefit.
|
|
|
|
|
#
|
|
|
|
|
# Yes these `--ignore-filename-regex=` options are included in all invocations of `llvm-cov show`
|
|
|
|
|
# for now, but it is effectively ignored for all tests that don't include this file anyway.
|
|
|
|
|
#
|
|
|
|
|
# Note that it's also possible the `_counters.<test>.txt` and `<test>.json` files may order
|
|
|
|
|
# results from multiple files inconsistently, which might also have to be accomodated if and when
|
|
|
|
|
# we allow `llvm-cov` to produce results for multiple files. (The path separators appear to be
|
|
|
|
|
# normalized to `/` in those files, thankfully.) But since we are ignoring results for all but one
|
|
|
|
|
# file, this workaround addresses those potential issues as well.
|
|
|
|
|
LLVM_COV_IGNORE_FILES=\
|
|
|
|
|
--ignore-filename-regex=uses_crate.rs
|
|
|
|
|
|
2020-11-04 05:45:32 +00:00
|
|
|
# When generating `expected_*` results (using `x.py test --bless`), the `--debug` flag is forced.
|
|
|
|
|
# If assertions are disabled, the command will fail with an error, rather than attempt to generate
|
|
|
|
|
# only partial results.
|
|
|
|
|
ifdef RUSTC_BLESS_TEST
|
|
|
|
|
DEBUG_FLAG=--debug
|
|
|
|
|
endif
|
|
|
|
|
|
2020-11-24 19:50:24 +00:00
|
|
|
ifeq ($(LLVM_VERSION_11_PLUS),true)
|
2020-12-02 23:39:40 +00:00
|
|
|
all: $(patsubst $(SOURCEDIR)/lib/%.rs,%,$(wildcard $(SOURCEDIR)/lib/*.rs)) $(patsubst $(SOURCEDIR)/%.rs,%,$(wildcard $(SOURCEDIR)/*.rs))
|
2020-11-24 19:50:24 +00:00
|
|
|
else
|
|
|
|
|
$(info Rust option `-Z instrument-coverage` requires LLVM 11 or higher. Test skipped.)
|
|
|
|
|
all:
|
|
|
|
|
endif
|
Tools, tests, and experimenting with MIR-derived coverage counters
Adds a new mir_dump output file in HTML/CSS to visualize code regions
and the MIR features that they came from (including overlapping spans).
See example below:
Includes a basic, MIR-block-based implementation of coverage injection,
available via `-Zexperimental-coverage`. This implementation has known
flaws and omissions, but is simple enough to validate the new tools and
tests.
The existing `-Zinstrument-coverage` option currently enables
function-level coverage only, which at least appears to generate
accurate coverage reports at that level.
Experimental coverage is not accurate at this time. When branch coverage
works as intended, the `-Zexperimental-coverage` option should be
removed.
This PR replaces the bulk of PR #75828, with the remaining parts of
that PR distributed among other separate and indentpent PRs.
This PR depends on three of those other PRs: #76000, #76002, and
Rust compiler MCP rust-lang/compiler-team#278
Relevant issue: #34701 - Implement support for LLVMs code coverage
instrumentation
2020-08-27 21:13:04 +00:00
|
|
|
|
|
|
|
|
# Ensure there are no `expected` results for tests that may have been removed or renamed
|
|
|
|
|
.PHONY: clear_expected_if_blessed
|
|
|
|
|
clear_expected_if_blessed:
|
|
|
|
|
ifdef RUSTC_BLESS_TEST
|
|
|
|
|
rm -f expected_export_coverage.*.json
|
Updates to experimental coverage counter injection
This is a combination of 18 commits.
Commit #2:
Additional examples and some small improvements.
Commit #3:
fixed mir-opt non-mir extensions and spanview title elements
Corrected a fairly recent assumption in runtest.rs that all MIR dump
files end in .mir. (It was appending .mir to the graphviz .dot and
spanview .html file names when generating blessed output files. That
also left outdated files in the baseline alongside the files with the
incorrect names, which I've now removed.)
Updated spanview HTML title elements to match their content, replacing a
hardcoded and incorrect name that was left in accidentally when
originally submitted.
Commit #4:
added more test examples
also improved Makefiles with support for non-zero exit status and to
force validation of tests unless a specific test overrides it with a
specific comment.
Commit #5:
Fixed rare issues after testing on real-world crate
Commit #6:
Addressed PR feedback, and removed temporary -Zexperimental-coverage
-Zinstrument-coverage once again supports the latest capabilities of
LLVM instrprof coverage instrumentation.
Also fixed a bug in spanview.
Commit #7:
Fix closure handling, add tests for closures and inner items
And cleaned up other tests for consistency, and to make it more clear
where spans start/end by breaking up lines.
Commit #8:
renamed "typical" test results "expected"
Now that the `llvm-cov show` tests are improved to normally expect
matching actuals, and to allow individual tests to override that
expectation.
Commit #9:
test coverage of inline generic struct function
Commit #10:
Addressed review feedback
* Removed unnecessary Unreachable filter.
* Replaced a match wildcard with remining variants.
* Added more comments to help clarify the role of successors() in the
CFG traversal
Commit #11:
refactoring based on feedback
* refactored `fn coverage_spans()`.
* changed the way I expand an empty coverage span to improve performance
* fixed a typo that I had accidently left in, in visit.rs
Commit #12:
Optimized use of SourceMap and SourceFile
Commit #13:
Fixed a regression, and synched with upstream
Some generated test file names changed due to some new change upstream.
Commit #14:
Stripping out crate disambiguators from demangled names
These can vary depending on the test platform.
Commit #15:
Ignore llvm-cov show diff on test with generics, expand IO error message
Tests with generics produce llvm-cov show results with demangled names
that can include an unstable "crate disambiguator" (hex value). The
value changes when run in the Rust CI Windows environment. I added a sed
filter to strip them out (in a prior commit), but sed also appears to
fail in the same environment. Until I can figure out a workaround, I'm
just going to ignore this specific test result. I added a FIXME to
follow up later, but it's not that critical.
I also saw an error with Windows GNU, but the IO error did not
specify a path for the directory or file that triggered the error. I
updated the error messages to provide more info for next, time but also
noticed some other tests with similar steps did not fail. Looks
spurious.
Commit #16:
Modify rust-demangler to strip disambiguators by default
Commit #17:
Remove std::process::exit from coverage tests
Due to Issue #77553, programs that call std::process::exit() do not
generate coverage results on Windows MSVC.
Commit #18:
fix: test file paths exceeding Windows max path len
2020-09-01 23:15:17 +00:00
|
|
|
rm -f expected_show_coverage.*.txt
|
2020-10-05 23:36:10 +00:00
|
|
|
rm -f expected_show_coverage_counters.*.txt
|
Tools, tests, and experimenting with MIR-derived coverage counters
Adds a new mir_dump output file in HTML/CSS to visualize code regions
and the MIR features that they came from (including overlapping spans).
See example below:
Includes a basic, MIR-block-based implementation of coverage injection,
available via `-Zexperimental-coverage`. This implementation has known
flaws and omissions, but is simple enough to validate the new tools and
tests.
The existing `-Zinstrument-coverage` option currently enables
function-level coverage only, which at least appears to generate
accurate coverage reports at that level.
Experimental coverage is not accurate at this time. When branch coverage
works as intended, the `-Zexperimental-coverage` option should be
removed.
This PR replaces the bulk of PR #75828, with the remaining parts of
that PR distributed among other separate and indentpent PRs.
This PR depends on three of those other PRs: #76000, #76002, and
Rust compiler MCP rust-lang/compiler-team#278
Relevant issue: #34701 - Implement support for LLVMs code coverage
instrumentation
2020-08-27 21:13:04 +00:00
|
|
|
endif
|
|
|
|
|
|
|
|
|
|
-include clear_expected_if_blessed
|
|
|
|
|
|
2020-12-02 23:39:40 +00:00
|
|
|
%: $(SOURCEDIR)/lib/%.rs
|
|
|
|
|
# Compile the test library with coverage instrumentation
|
|
|
|
|
$(RUSTC) $(SOURCEDIR)/lib/$@.rs \
|
|
|
|
|
$$( grep -q '^\/\/ require-rust-edition-2018' $(SOURCEDIR)/lib/$@.rs && \
|
|
|
|
|
echo "--edition=2018" \
|
|
|
|
|
) \
|
|
|
|
|
--crate-type rlib \
|
|
|
|
|
-Zinstrument-coverage
|
|
|
|
|
|
Tools, tests, and experimenting with MIR-derived coverage counters
Adds a new mir_dump output file in HTML/CSS to visualize code regions
and the MIR features that they came from (including overlapping spans).
See example below:
Includes a basic, MIR-block-based implementation of coverage injection,
available via `-Zexperimental-coverage`. This implementation has known
flaws and omissions, but is simple enough to validate the new tools and
tests.
The existing `-Zinstrument-coverage` option currently enables
function-level coverage only, which at least appears to generate
accurate coverage reports at that level.
Experimental coverage is not accurate at this time. When branch coverage
works as intended, the `-Zexperimental-coverage` option should be
removed.
This PR replaces the bulk of PR #75828, with the remaining parts of
that PR distributed among other separate and indentpent PRs.
This PR depends on three of those other PRs: #76000, #76002, and
Rust compiler MCP rust-lang/compiler-team#278
Relevant issue: #34701 - Implement support for LLVMs code coverage
instrumentation
2020-08-27 21:13:04 +00:00
|
|
|
%: $(SOURCEDIR)/%.rs
|
2020-12-02 23:39:40 +00:00
|
|
|
# Compile the test program with coverage instrumentation
|
Tools, tests, and experimenting with MIR-derived coverage counters
Adds a new mir_dump output file in HTML/CSS to visualize code regions
and the MIR features that they came from (including overlapping spans).
See example below:
Includes a basic, MIR-block-based implementation of coverage injection,
available via `-Zexperimental-coverage`. This implementation has known
flaws and omissions, but is simple enough to validate the new tools and
tests.
The existing `-Zinstrument-coverage` option currently enables
function-level coverage only, which at least appears to generate
accurate coverage reports at that level.
Experimental coverage is not accurate at this time. When branch coverage
works as intended, the `-Zexperimental-coverage` option should be
removed.
This PR replaces the bulk of PR #75828, with the remaining parts of
that PR distributed among other separate and indentpent PRs.
This PR depends on three of those other PRs: #76000, #76002, and
Rust compiler MCP rust-lang/compiler-team#278
Relevant issue: #34701 - Implement support for LLVMs code coverage
instrumentation
2020-08-27 21:13:04 +00:00
|
|
|
$(RUSTC) $(SOURCEDIR)/$@.rs \
|
Coverage tests for remaining TerminatorKinds and async, improve Assert
Tested and validate results for panic unwind, panic abort, assert!()
macro, TerminatorKind::Assert (for example, numeric overflow), and
async/await.
Implemented a previous documented idea to change Assert handling to be
the same as FalseUnwind and Goto, so it doesn't get its own
BasicCoverageBlock anymore. This changed a couple of coverage regions,
but I validated those changes are not any worse than the prior results,
and probably help assure some consistency (even if some people might
disagree with how the code region is consistently computed).
Fixed issue with async/await. AggregateKind::Generator needs to be
handled like AggregateKind::Closure; coverage span for the outer async
function should not "cover" the async body, which is actually executed
in a separate "closure" MIR.
2020-11-16 17:14:28 +00:00
|
|
|
$$( grep -q '^\/\/ require-rust-edition-2018' $(SOURCEDIR)/$@.rs && \
|
|
|
|
|
echo "--edition=2018" \
|
|
|
|
|
) \
|
2020-12-02 23:39:40 +00:00
|
|
|
-L "$(TMPDIR)" \
|
2020-12-01 07:58:08 +00:00
|
|
|
-Zinstrument-coverage
|
Tools, tests, and experimenting with MIR-derived coverage counters
Adds a new mir_dump output file in HTML/CSS to visualize code regions
and the MIR features that they came from (including overlapping spans).
See example below:
Includes a basic, MIR-block-based implementation of coverage injection,
available via `-Zexperimental-coverage`. This implementation has known
flaws and omissions, but is simple enough to validate the new tools and
tests.
The existing `-Zinstrument-coverage` option currently enables
function-level coverage only, which at least appears to generate
accurate coverage reports at that level.
Experimental coverage is not accurate at this time. When branch coverage
works as intended, the `-Zexperimental-coverage` option should be
removed.
This PR replaces the bulk of PR #75828, with the remaining parts of
that PR distributed among other separate and indentpent PRs.
This PR depends on three of those other PRs: #76000, #76002, and
Rust compiler MCP rust-lang/compiler-team#278
Relevant issue: #34701 - Implement support for LLVMs code coverage
instrumentation
2020-08-27 21:13:04 +00:00
|
|
|
|
|
|
|
|
# Run it in order to generate some profiling data,
|
|
|
|
|
# with `LLVM_PROFILE_FILE=<profdata_file>` environment variable set to
|
|
|
|
|
# output the coverage stats for this run.
|
|
|
|
|
LLVM_PROFILE_FILE="$(TMPDIR)"/$@.profraw \
|
Updates to experimental coverage counter injection
This is a combination of 18 commits.
Commit #2:
Additional examples and some small improvements.
Commit #3:
fixed mir-opt non-mir extensions and spanview title elements
Corrected a fairly recent assumption in runtest.rs that all MIR dump
files end in .mir. (It was appending .mir to the graphviz .dot and
spanview .html file names when generating blessed output files. That
also left outdated files in the baseline alongside the files with the
incorrect names, which I've now removed.)
Updated spanview HTML title elements to match their content, replacing a
hardcoded and incorrect name that was left in accidentally when
originally submitted.
Commit #4:
added more test examples
also improved Makefiles with support for non-zero exit status and to
force validation of tests unless a specific test overrides it with a
specific comment.
Commit #5:
Fixed rare issues after testing on real-world crate
Commit #6:
Addressed PR feedback, and removed temporary -Zexperimental-coverage
-Zinstrument-coverage once again supports the latest capabilities of
LLVM instrprof coverage instrumentation.
Also fixed a bug in spanview.
Commit #7:
Fix closure handling, add tests for closures and inner items
And cleaned up other tests for consistency, and to make it more clear
where spans start/end by breaking up lines.
Commit #8:
renamed "typical" test results "expected"
Now that the `llvm-cov show` tests are improved to normally expect
matching actuals, and to allow individual tests to override that
expectation.
Commit #9:
test coverage of inline generic struct function
Commit #10:
Addressed review feedback
* Removed unnecessary Unreachable filter.
* Replaced a match wildcard with remining variants.
* Added more comments to help clarify the role of successors() in the
CFG traversal
Commit #11:
refactoring based on feedback
* refactored `fn coverage_spans()`.
* changed the way I expand an empty coverage span to improve performance
* fixed a typo that I had accidently left in, in visit.rs
Commit #12:
Optimized use of SourceMap and SourceFile
Commit #13:
Fixed a regression, and synched with upstream
Some generated test file names changed due to some new change upstream.
Commit #14:
Stripping out crate disambiguators from demangled names
These can vary depending on the test platform.
Commit #15:
Ignore llvm-cov show diff on test with generics, expand IO error message
Tests with generics produce llvm-cov show results with demangled names
that can include an unstable "crate disambiguator" (hex value). The
value changes when run in the Rust CI Windows environment. I added a sed
filter to strip them out (in a prior commit), but sed also appears to
fail in the same environment. Until I can figure out a workaround, I'm
just going to ignore this specific test result. I added a FIXME to
follow up later, but it's not that critical.
I also saw an error with Windows GNU, but the IO error did not
specify a path for the directory or file that triggered the error. I
updated the error messages to provide more info for next, time but also
noticed some other tests with similar steps did not fail. Looks
spurious.
Commit #16:
Modify rust-demangler to strip disambiguators by default
Commit #17:
Remove std::process::exit from coverage tests
Due to Issue #77553, programs that call std::process::exit() do not
generate coverage results on Windows MSVC.
Commit #18:
fix: test file paths exceeding Windows max path len
2020-09-01 23:15:17 +00:00
|
|
|
$(call RUN,$@) || \
|
|
|
|
|
( \
|
|
|
|
|
status=$$?; \
|
|
|
|
|
grep -q "^\/\/ expect-exit-status-$$status" $(SOURCEDIR)/$@.rs || \
|
|
|
|
|
( >&2 echo "program exited with an unexpected exit status: $$status"; \
|
|
|
|
|
false \
|
|
|
|
|
) \
|
|
|
|
|
)
|
Tools, tests, and experimenting with MIR-derived coverage counters
Adds a new mir_dump output file in HTML/CSS to visualize code regions
and the MIR features that they came from (including overlapping spans).
See example below:
Includes a basic, MIR-block-based implementation of coverage injection,
available via `-Zexperimental-coverage`. This implementation has known
flaws and omissions, but is simple enough to validate the new tools and
tests.
The existing `-Zinstrument-coverage` option currently enables
function-level coverage only, which at least appears to generate
accurate coverage reports at that level.
Experimental coverage is not accurate at this time. When branch coverage
works as intended, the `-Zexperimental-coverage` option should be
removed.
This PR replaces the bulk of PR #75828, with the remaining parts of
that PR distributed among other separate and indentpent PRs.
This PR depends on three of those other PRs: #76000, #76002, and
Rust compiler MCP rust-lang/compiler-team#278
Relevant issue: #34701 - Implement support for LLVMs code coverage
instrumentation
2020-08-27 21:13:04 +00:00
|
|
|
|
|
|
|
|
# Postprocess the profiling data so it can be used by the llvm-cov tool
|
|
|
|
|
"$(LLVM_BIN_DIR)"/llvm-profdata merge --sparse \
|
|
|
|
|
"$(TMPDIR)"/$@.profraw \
|
|
|
|
|
-o "$(TMPDIR)"/$@.profdata
|
|
|
|
|
|
2020-11-04 05:45:32 +00:00
|
|
|
# Generate a coverage report using `llvm-cov show`.
|
Tools, tests, and experimenting with MIR-derived coverage counters
Adds a new mir_dump output file in HTML/CSS to visualize code regions
and the MIR features that they came from (including overlapping spans).
See example below:
Includes a basic, MIR-block-based implementation of coverage injection,
available via `-Zexperimental-coverage`. This implementation has known
flaws and omissions, but is simple enough to validate the new tools and
tests.
The existing `-Zinstrument-coverage` option currently enables
function-level coverage only, which at least appears to generate
accurate coverage reports at that level.
Experimental coverage is not accurate at this time. When branch coverage
works as intended, the `-Zexperimental-coverage` option should be
removed.
This PR replaces the bulk of PR #75828, with the remaining parts of
that PR distributed among other separate and indentpent PRs.
This PR depends on three of those other PRs: #76000, #76002, and
Rust compiler MCP rust-lang/compiler-team#278
Relevant issue: #34701 - Implement support for LLVMs code coverage
instrumentation
2020-08-27 21:13:04 +00:00
|
|
|
"$(LLVM_BIN_DIR)"/llvm-cov show \
|
2020-11-04 05:45:32 +00:00
|
|
|
$(DEBUG_FLAG) \
|
2020-12-02 17:43:46 +00:00
|
|
|
$(LLVM_COV_IGNORE_FILES) \
|
Tools, tests, and experimenting with MIR-derived coverage counters
Adds a new mir_dump output file in HTML/CSS to visualize code regions
and the MIR features that they came from (including overlapping spans).
See example below:
Includes a basic, MIR-block-based implementation of coverage injection,
available via `-Zexperimental-coverage`. This implementation has known
flaws and omissions, but is simple enough to validate the new tools and
tests.
The existing `-Zinstrument-coverage` option currently enables
function-level coverage only, which at least appears to generate
accurate coverage reports at that level.
Experimental coverage is not accurate at this time. When branch coverage
works as intended, the `-Zexperimental-coverage` option should be
removed.
This PR replaces the bulk of PR #75828, with the remaining parts of
that PR distributed among other separate and indentpent PRs.
This PR depends on three of those other PRs: #76000, #76002, and
Rust compiler MCP rust-lang/compiler-team#278
Relevant issue: #34701 - Implement support for LLVMs code coverage
instrumentation
2020-08-27 21:13:04 +00:00
|
|
|
--Xdemangler="$(RUST_DEMANGLER)" \
|
|
|
|
|
--show-line-counts-or-regions \
|
|
|
|
|
--instr-profile="$(TMPDIR)"/$@.profdata \
|
|
|
|
|
$(call BIN,"$(TMPDIR)"/$@) \
|
2020-10-05 23:36:10 +00:00
|
|
|
> "$(TMPDIR)"/actual_show_coverage.$@.txt \
|
2020-11-04 05:45:32 +00:00
|
|
|
2> "$(TMPDIR)"/show_coverage_stderr.$@.txt || \
|
|
|
|
|
( status=$$? ; \
|
|
|
|
|
>&2 cat "$(TMPDIR)"/show_coverage_stderr.$@.txt ; \
|
|
|
|
|
exit $$status \
|
|
|
|
|
)
|
2020-11-02 23:01:30 +00:00
|
|
|
|
2020-11-04 05:45:32 +00:00
|
|
|
ifdef DEBUG_FLAG
|
2020-11-02 23:01:30 +00:00
|
|
|
# The first line (beginning with "Args:" contains hard-coded, build-specific
|
|
|
|
|
# file paths. Strip that line and keep the remaining lines with counter debug
|
|
|
|
|
# data.
|
|
|
|
|
tail -n +2 "$(TMPDIR)"/show_coverage_stderr.$@.txt \
|
|
|
|
|
> "$(TMPDIR)"/actual_show_coverage_counters.$@.txt
|
2020-11-04 05:45:32 +00:00
|
|
|
endif
|
Tools, tests, and experimenting with MIR-derived coverage counters
Adds a new mir_dump output file in HTML/CSS to visualize code regions
and the MIR features that they came from (including overlapping spans).
See example below:
Includes a basic, MIR-block-based implementation of coverage injection,
available via `-Zexperimental-coverage`. This implementation has known
flaws and omissions, but is simple enough to validate the new tools and
tests.
The existing `-Zinstrument-coverage` option currently enables
function-level coverage only, which at least appears to generate
accurate coverage reports at that level.
Experimental coverage is not accurate at this time. When branch coverage
works as intended, the `-Zexperimental-coverage` option should be
removed.
This PR replaces the bulk of PR #75828, with the remaining parts of
that PR distributed among other separate and indentpent PRs.
This PR depends on three of those other PRs: #76000, #76002, and
Rust compiler MCP rust-lang/compiler-team#278
Relevant issue: #34701 - Implement support for LLVMs code coverage
instrumentation
2020-08-27 21:13:04 +00:00
|
|
|
|
|
|
|
|
ifdef RUSTC_BLESS_TEST
|
2020-10-05 23:36:10 +00:00
|
|
|
cp "$(TMPDIR)"/actual_show_coverage.$@.txt \
|
|
|
|
|
expected_show_coverage.$@.txt
|
|
|
|
|
cp "$(TMPDIR)"/actual_show_coverage_counters.$@.txt \
|
|
|
|
|
expected_show_coverage_counters.$@.txt
|
Tools, tests, and experimenting with MIR-derived coverage counters
Adds a new mir_dump output file in HTML/CSS to visualize code regions
and the MIR features that they came from (including overlapping spans).
See example below:
Includes a basic, MIR-block-based implementation of coverage injection,
available via `-Zexperimental-coverage`. This implementation has known
flaws and omissions, but is simple enough to validate the new tools and
tests.
The existing `-Zinstrument-coverage` option currently enables
function-level coverage only, which at least appears to generate
accurate coverage reports at that level.
Experimental coverage is not accurate at this time. When branch coverage
works as intended, the `-Zexperimental-coverage` option should be
removed.
This PR replaces the bulk of PR #75828, with the remaining parts of
that PR distributed among other separate and indentpent PRs.
This PR depends on three of those other PRs: #76000, #76002, and
Rust compiler MCP rust-lang/compiler-team#278
Relevant issue: #34701 - Implement support for LLVMs code coverage
instrumentation
2020-08-27 21:13:04 +00:00
|
|
|
else
|
|
|
|
|
# Compare the show coverage output (`--bless` refreshes `typical` files)
|
Updates to experimental coverage counter injection
This is a combination of 18 commits.
Commit #2:
Additional examples and some small improvements.
Commit #3:
fixed mir-opt non-mir extensions and spanview title elements
Corrected a fairly recent assumption in runtest.rs that all MIR dump
files end in .mir. (It was appending .mir to the graphviz .dot and
spanview .html file names when generating blessed output files. That
also left outdated files in the baseline alongside the files with the
incorrect names, which I've now removed.)
Updated spanview HTML title elements to match their content, replacing a
hardcoded and incorrect name that was left in accidentally when
originally submitted.
Commit #4:
added more test examples
also improved Makefiles with support for non-zero exit status and to
force validation of tests unless a specific test overrides it with a
specific comment.
Commit #5:
Fixed rare issues after testing on real-world crate
Commit #6:
Addressed PR feedback, and removed temporary -Zexperimental-coverage
-Zinstrument-coverage once again supports the latest capabilities of
LLVM instrprof coverage instrumentation.
Also fixed a bug in spanview.
Commit #7:
Fix closure handling, add tests for closures and inner items
And cleaned up other tests for consistency, and to make it more clear
where spans start/end by breaking up lines.
Commit #8:
renamed "typical" test results "expected"
Now that the `llvm-cov show` tests are improved to normally expect
matching actuals, and to allow individual tests to override that
expectation.
Commit #9:
test coverage of inline generic struct function
Commit #10:
Addressed review feedback
* Removed unnecessary Unreachable filter.
* Replaced a match wildcard with remining variants.
* Added more comments to help clarify the role of successors() in the
CFG traversal
Commit #11:
refactoring based on feedback
* refactored `fn coverage_spans()`.
* changed the way I expand an empty coverage span to improve performance
* fixed a typo that I had accidently left in, in visit.rs
Commit #12:
Optimized use of SourceMap and SourceFile
Commit #13:
Fixed a regression, and synched with upstream
Some generated test file names changed due to some new change upstream.
Commit #14:
Stripping out crate disambiguators from demangled names
These can vary depending on the test platform.
Commit #15:
Ignore llvm-cov show diff on test with generics, expand IO error message
Tests with generics produce llvm-cov show results with demangled names
that can include an unstable "crate disambiguator" (hex value). The
value changes when run in the Rust CI Windows environment. I added a sed
filter to strip them out (in a prior commit), but sed also appears to
fail in the same environment. Until I can figure out a workaround, I'm
just going to ignore this specific test result. I added a FIXME to
follow up later, but it's not that critical.
I also saw an error with Windows GNU, but the IO error did not
specify a path for the directory or file that triggered the error. I
updated the error messages to provide more info for next, time but also
noticed some other tests with similar steps did not fail. Looks
spurious.
Commit #16:
Modify rust-demangler to strip disambiguators by default
Commit #17:
Remove std::process::exit from coverage tests
Due to Issue #77553, programs that call std::process::exit() do not
generate coverage results on Windows MSVC.
Commit #18:
fix: test file paths exceeding Windows max path len
2020-09-01 23:15:17 +00:00
|
|
|
# Note `llvm-cov show` output for some programs can vary, but can be ignored
|
|
|
|
|
# by inserting `// ignore-llvm-cov-show-diffs` at the top of the source file.
|
2020-12-02 23:39:40 +00:00
|
|
|
#
|
|
|
|
|
# FIXME(richkadel): It looks like most past variations seem to have been mitigated. None of the
|
|
|
|
|
# Rust test source samples have the `// ignore-llvm-cov-show-diffs` anymore. The main variation
|
|
|
|
|
# I had seen (and is still present in the new `coverage/lib/used_crate.rs`) is the `llvm-cov show`
|
|
|
|
|
# reporting of multiple instantiations of a generic function with different type substitutions.
|
|
|
|
|
# For some reason, `llvm-cov show` can report these in a non-deterministic order, breaking the
|
|
|
|
|
# `diff` comparision. I was able to work around the problem with `diff --ignore-matching-lines=RE`
|
|
|
|
|
# to ignore each line prefixing each generic instantiation coverage code region.
|
|
|
|
|
|
|
|
|
|
$(DIFF) --ignore-matching-lines='::<.*>.*:$$' \
|
|
|
|
|
expected_show_coverage.$@.txt "$(TMPDIR)"/actual_show_coverage.$@.txt || \
|
Updates to experimental coverage counter injection
This is a combination of 18 commits.
Commit #2:
Additional examples and some small improvements.
Commit #3:
fixed mir-opt non-mir extensions and spanview title elements
Corrected a fairly recent assumption in runtest.rs that all MIR dump
files end in .mir. (It was appending .mir to the graphviz .dot and
spanview .html file names when generating blessed output files. That
also left outdated files in the baseline alongside the files with the
incorrect names, which I've now removed.)
Updated spanview HTML title elements to match their content, replacing a
hardcoded and incorrect name that was left in accidentally when
originally submitted.
Commit #4:
added more test examples
also improved Makefiles with support for non-zero exit status and to
force validation of tests unless a specific test overrides it with a
specific comment.
Commit #5:
Fixed rare issues after testing on real-world crate
Commit #6:
Addressed PR feedback, and removed temporary -Zexperimental-coverage
-Zinstrument-coverage once again supports the latest capabilities of
LLVM instrprof coverage instrumentation.
Also fixed a bug in spanview.
Commit #7:
Fix closure handling, add tests for closures and inner items
And cleaned up other tests for consistency, and to make it more clear
where spans start/end by breaking up lines.
Commit #8:
renamed "typical" test results "expected"
Now that the `llvm-cov show` tests are improved to normally expect
matching actuals, and to allow individual tests to override that
expectation.
Commit #9:
test coverage of inline generic struct function
Commit #10:
Addressed review feedback
* Removed unnecessary Unreachable filter.
* Replaced a match wildcard with remining variants.
* Added more comments to help clarify the role of successors() in the
CFG traversal
Commit #11:
refactoring based on feedback
* refactored `fn coverage_spans()`.
* changed the way I expand an empty coverage span to improve performance
* fixed a typo that I had accidently left in, in visit.rs
Commit #12:
Optimized use of SourceMap and SourceFile
Commit #13:
Fixed a regression, and synched with upstream
Some generated test file names changed due to some new change upstream.
Commit #14:
Stripping out crate disambiguators from demangled names
These can vary depending on the test platform.
Commit #15:
Ignore llvm-cov show diff on test with generics, expand IO error message
Tests with generics produce llvm-cov show results with demangled names
that can include an unstable "crate disambiguator" (hex value). The
value changes when run in the Rust CI Windows environment. I added a sed
filter to strip them out (in a prior commit), but sed also appears to
fail in the same environment. Until I can figure out a workaround, I'm
just going to ignore this specific test result. I added a FIXME to
follow up later, but it's not that critical.
I also saw an error with Windows GNU, but the IO error did not
specify a path for the directory or file that triggered the error. I
updated the error messages to provide more info for next, time but also
noticed some other tests with similar steps did not fail. Looks
spurious.
Commit #16:
Modify rust-demangler to strip disambiguators by default
Commit #17:
Remove std::process::exit from coverage tests
Due to Issue #77553, programs that call std::process::exit() do not
generate coverage results on Windows MSVC.
Commit #18:
fix: test file paths exceeding Windows max path len
2020-09-01 23:15:17 +00:00
|
|
|
( grep -q '^\/\/ ignore-llvm-cov-show-diffs' $(SOURCEDIR)/$@.rs && \
|
|
|
|
|
>&2 echo 'diff failed, but suppressed with `// ignore-llvm-cov-show-diffs` in $(SOURCEDIR)/$@.rs' \
|
|
|
|
|
) || \
|
|
|
|
|
( >&2 echo 'diff failed, and not suppressed without `// ignore-llvm-cov-show-diffs` in $(SOURCEDIR)/$@.rs'; \
|
|
|
|
|
false \
|
|
|
|
|
)
|
|
|
|
|
|
2020-11-04 05:45:32 +00:00
|
|
|
ifdef DEBUG_FLAG
|
2020-10-05 23:36:10 +00:00
|
|
|
$(DIFF) expected_show_coverage_counters.$@.txt "$(TMPDIR)"/actual_show_coverage_counters.$@.txt || \
|
|
|
|
|
( grep -q '^\/\/ ignore-llvm-cov-show-diffs' $(SOURCEDIR)/$@.rs && \
|
|
|
|
|
>&2 echo 'diff failed, but suppressed with `// ignore-llvm-cov-show-diffs` in $(SOURCEDIR)/$@.rs' \
|
|
|
|
|
) || \
|
|
|
|
|
( >&2 echo 'diff failed, and not suppressed without `// ignore-llvm-cov-show-diffs` in $(SOURCEDIR)/$@.rs'; \
|
2020-11-02 23:01:30 +00:00
|
|
|
>&2 echo '(Ignore anyway until mangled function names in "counters" files are demangled.)' \
|
2020-10-05 23:36:10 +00:00
|
|
|
)
|
|
|
|
|
|
2020-11-02 23:01:30 +00:00
|
|
|
# FIXME(richkadel): Apply the demangler to the `*_show_coverage_counters.*.txt` files,
|
|
|
|
|
# so the crate disambiguator differences will be stripped away. At that point, these files
|
|
|
|
|
# will be less likely to vary, and the last `echo` above (starting with "Ignore anyway")
|
|
|
|
|
# can be replaced with `false` to fail the test.
|
2020-11-04 05:45:32 +00:00
|
|
|
endif
|
2020-11-02 23:01:30 +00:00
|
|
|
|
Tools, tests, and experimenting with MIR-derived coverage counters
Adds a new mir_dump output file in HTML/CSS to visualize code regions
and the MIR features that they came from (including overlapping spans).
See example below:
Includes a basic, MIR-block-based implementation of coverage injection,
available via `-Zexperimental-coverage`. This implementation has known
flaws and omissions, but is simple enough to validate the new tools and
tests.
The existing `-Zinstrument-coverage` option currently enables
function-level coverage only, which at least appears to generate
accurate coverage reports at that level.
Experimental coverage is not accurate at this time. When branch coverage
works as intended, the `-Zexperimental-coverage` option should be
removed.
This PR replaces the bulk of PR #75828, with the remaining parts of
that PR distributed among other separate and indentpent PRs.
This PR depends on three of those other PRs: #76000, #76002, and
Rust compiler MCP rust-lang/compiler-team#278
Relevant issue: #34701 - Implement support for LLVMs code coverage
instrumentation
2020-08-27 21:13:04 +00:00
|
|
|
endif
|
|
|
|
|
|
|
|
|
|
# Generate a coverage report in JSON, using `llvm-cov export`, and fail if
|
|
|
|
|
# there are differences from the expected output.
|
|
|
|
|
"$(LLVM_BIN_DIR)"/llvm-cov export \
|
2020-12-02 17:43:46 +00:00
|
|
|
$(LLVM_COV_IGNORE_FILES) \
|
Tools, tests, and experimenting with MIR-derived coverage counters
Adds a new mir_dump output file in HTML/CSS to visualize code regions
and the MIR features that they came from (including overlapping spans).
See example below:
Includes a basic, MIR-block-based implementation of coverage injection,
available via `-Zexperimental-coverage`. This implementation has known
flaws and omissions, but is simple enough to validate the new tools and
tests.
The existing `-Zinstrument-coverage` option currently enables
function-level coverage only, which at least appears to generate
accurate coverage reports at that level.
Experimental coverage is not accurate at this time. When branch coverage
works as intended, the `-Zexperimental-coverage` option should be
removed.
This PR replaces the bulk of PR #75828, with the remaining parts of
that PR distributed among other separate and indentpent PRs.
This PR depends on three of those other PRs: #76000, #76002, and
Rust compiler MCP rust-lang/compiler-team#278
Relevant issue: #34701 - Implement support for LLVMs code coverage
instrumentation
2020-08-27 21:13:04 +00:00
|
|
|
--summary-only \
|
|
|
|
|
--instr-profile="$(TMPDIR)"/$@.profdata \
|
|
|
|
|
$(call BIN,"$(TMPDIR)"/$@) \
|
|
|
|
|
| "$(PYTHON)" $(BASEDIR)/prettify_json.py \
|
|
|
|
|
> "$(TMPDIR)"/actual_export_coverage.$@.json
|
2020-12-02 23:39:40 +00:00
|
|
|
# FIXME(richkadel): With the addition of `--ignore-matching-lines=RE` to ignore the
|
|
|
|
|
# non-deterministically-ordered coverage results for multiple instantiations of generics with
|
|
|
|
|
# differing type substitutions, I probably don't need the `.json` files anymore (and may not
|
|
|
|
|
# need `prettify_json.py` either).
|
Tools, tests, and experimenting with MIR-derived coverage counters
Adds a new mir_dump output file in HTML/CSS to visualize code regions
and the MIR features that they came from (including overlapping spans).
See example below:
Includes a basic, MIR-block-based implementation of coverage injection,
available via `-Zexperimental-coverage`. This implementation has known
flaws and omissions, but is simple enough to validate the new tools and
tests.
The existing `-Zinstrument-coverage` option currently enables
function-level coverage only, which at least appears to generate
accurate coverage reports at that level.
Experimental coverage is not accurate at this time. When branch coverage
works as intended, the `-Zexperimental-coverage` option should be
removed.
This PR replaces the bulk of PR #75828, with the remaining parts of
that PR distributed among other separate and indentpent PRs.
This PR depends on three of those other PRs: #76000, #76002, and
Rust compiler MCP rust-lang/compiler-team#278
Relevant issue: #34701 - Implement support for LLVMs code coverage
instrumentation
2020-08-27 21:13:04 +00:00
|
|
|
|
|
|
|
|
ifdef RUSTC_BLESS_TEST
|
|
|
|
|
cp "$(TMPDIR)"/actual_export_coverage.$@.json expected_export_coverage.$@.json
|
|
|
|
|
else
|
|
|
|
|
# Check that exported JSON coverage data matches what we expect (`--bless` refreshes `expected`)
|
|
|
|
|
$(DIFF) expected_export_coverage.$@.json "$(TMPDIR)"/actual_export_coverage.$@.json
|
|
|
|
|
endif
|