Commit Graph

109 Commits

Author SHA1 Message Date
bors
76c500ec6c Auto merge of #81635 - michaelwoerister:structured_def_path_hash, r=pnkfelix
Let a portion of DefPathHash uniquely identify the DefPath's crate.

This allows to directly map from a `DefPathHash` to the crate it originates from, without constructing side tables to do that mapping -- something that is useful for incremental compilation where we deal with `DefPathHash` instead of `DefId` a lot.

It also allows to reliably and cheaply check for `DefPathHash` collisions which allows the compiler to gracefully abort compilation instead of running into a subsequent ICE at some random place in the code.

The following new piece of documentation describes the most interesting aspects of the changes:

```rust
/// A `DefPathHash` is a fixed-size representation of a `DefPath` that is
/// stable across crate and compilation session boundaries. It consists of two
/// separate 64-bit hashes. The first uniquely identifies the crate this
/// `DefPathHash` originates from (see [StableCrateId]), and the second
/// uniquely identifies the corresponding `DefPath` within that crate. Together
/// they form a unique identifier within an entire crate graph.
///
/// There is a very small chance of hash collisions, which would mean that two
/// different `DefPath`s map to the same `DefPathHash`. Proceeding compilation
/// with such a hash collision would very probably lead to an ICE and, in the
/// worst case, to a silent mis-compilation. The compiler therefore actively
/// and exhaustively checks for such hash collisions and aborts compilation if
/// it finds one.
///
/// `DefPathHash` uses 64-bit hashes for both the crate-id part and the
/// crate-internal part, even though it is likely that there are many more
/// `LocalDefId`s in a single crate than there are individual crates in a crate
/// graph. Since we use the same number of bits in both cases, the collision
/// probability for the crate-local part will be quite a bit higher (though
/// still very small).
///
/// This imbalance is not by accident: A hash collision in the
/// crate-local part of a `DefPathHash` will be detected and reported while
/// compiling the crate in question. Such a collision does not depend on
/// outside factors and can be easily fixed by the crate maintainer (e.g. by
/// renaming the item in question or by bumping the crate version in a harmless
/// way).
///
/// A collision between crate-id hashes on the other hand is harder to fix
/// because it depends on the set of crates in the entire crate graph of a
/// compilation session. Again, using the same crate with a different version
/// number would fix the issue with a high probability -- but that might be
/// easier said then done if the crates in questions are dependencies of
/// third-party crates.
///
/// That being said, given a high quality hash function, the collision
/// probabilities in question are very small. For example, for a big crate like
/// `rustc_middle` (with ~50000 `LocalDefId`s as of the time of writing) there
/// is a probability of roughly 1 in 14,750,000,000 of a crate-internal
/// collision occurring. For a big crate graph with 1000 crates in it, there is
/// a probability of 1 in 36,890,000,000,000 of a `StableCrateId` collision.
```

Given the probabilities involved I hope that no one will ever actually see the error messages. Nonetheless, I'd be glad about some feedback on how to improve them. Should we create a GH issue describing the problem and possible solutions to point to? Or a page in the rustc book?

r? `@pnkfelix` (feel free to re-assign)
2021-03-07 23:45:57 +00:00
Dylan DPC
6d288c65df
Rollup merge of #82537 - wesleywiser:update_measureme, r=oli-obk
Update measureme dependency to the latest version

This version adds the ability to use `rdpmc` hardware-based performance
counters instead of wall-clock time for measuring duration. This also
introduces a dependency on the `perf-event-open-sys` crate on Linux
which is used when using hardware counters.

r? ```@oli-obk```
2021-02-27 21:56:20 +01:00
Dylan DPC
cabe97272d
Rollup merge of #82057 - upsuper-forks:cstr, r=davidtwco,wesleywiser
Replace const_cstr with cstr crate

This PR replaces the `const_cstr` macro inside `rustc_data_structures` with `cstr` macro from [cstr](https://crates.io/crates/cstr) crate.

The two macros basically serve the same purpose, which is to generate `&'static CStr` from a string literal. `cstr` is better because it validates the literal at compile time, while the existing `const_cstr` does it at runtime when `debug_assertions` is enabled. In addition, the value `cstr` generates can be used in constant context (which is seemingly not needed anywhere currently, though).
2021-02-27 02:34:21 +01:00
Wesley Wiser
e130e9cf77 Update measureme dependency to the latest version
This version adds the ability to use `rdpmc` hardware-based performance
counters instead of wall-clock time for measuring duration. This also
introduces a dependency on the `perf-event-open-sys` crate on Linux
which is used when using hardware counters.
2021-02-25 18:25:38 -05:00
Joshua Nelson
3733275854 Update the bootstrap compiler
Note this does not change `core::derive` since it was merged after the
beta bump.
2021-02-20 17:19:30 -05:00
Eduard-Mihai Burtescu
6165d1cc72 Print -Ztime-passes (and misc stats/logs) on stderr, not stdout. 2021-02-18 14:13:38 +02:00
bors
d1206f950f Auto merge of #81855 - cjgillot:ensure-cache, r=oli-obk
Check the result cache before the DepGraph when ensuring queries

Split out of https://github.com/rust-lang/rust/pull/70951

Calling `ensure` on already forced queries is a common operation.
Looking at the results cache first is faster than checking the DepGraph for a green node.
2021-02-15 12:11:59 +00:00
klensy
93c8ebe022 bumped smallvec deps 2021-02-14 18:03:11 +03:00
Xidorn Quan
38e4233a32 Replace const_cstr with cstr crate 2021-02-14 09:45:35 +11:00
Camille GILLOT
15b0bc6b83 Separate the query cache from the query state. 2021-02-13 21:14:58 +01:00
Dániel Buga
3c1d792f49 Only initialize what is used 2021-02-10 09:20:41 +01:00
Mara Bos
08d8fc14be
Rollup merge of #81771 - tgnottingham:time-passes-rss-delta, r=oli-obk
Indicate change in RSS from start to end of pass in time-passes output

Previously, this was omitted because it could be misleading, but the
functionality seems too useful not to include.

r? ``@oli-obk``
2021-02-05 12:26:08 +01:00
Tyson Nottingham
4253919f1d Indicate change in RSS from start to end of pass in time-passes output
Previously, this was omitted because it could be misleading, but the
functionality seems too useful not to include.
2021-02-05 01:11:52 -08:00
Michael Woerister
d4d8bdf52b Add documentation to Unhasher impl for Fingerprint. 2021-02-04 10:37:11 +01:00
Mara Bos
89882388d9 Revert stabilizing integer::BITS. 2021-02-03 22:23:58 +01:00
Michael Woerister
22d489be76 Let a portion of DefPathHash uniquely identify the DefPath's crate.
This allows to directly map from a DefPathHash to the crate it
originates from, without constructing side tables to do that mapping.

It also allows to reliably and cheaply check for DefPathHash collisions.
2021-02-02 17:40:29 +01:00
Jonas Schievink
82b00ec606
Rollup merge of #81536 - tgnottingham:time-passes-rss, r=oli-obk
Indicate both start and end of pass RSS in time-passes output

Previously, only the end of pass RSS was indicated. This could easily
lead one to believe that the change in RSS from one pass to the next was
attributable to the second pass, when in fact it occurred between the
end of the first pass and the start of the second.

Also, improve alignment of columns.

Sample of output:

```
time:   0.739; rss:   607MB ->   637MB	item_types_checking
time:   8.429; rss:   637MB ->   775MB	item_bodies_checking
time:  11.063; rss:   470MB ->   775MB	type_check_crate
time:   0.232; rss:   775MB ->   777MB	match_checking
time:   0.139; rss:   777MB ->   779MB	liveness_and_intrinsic_checking
time:   0.372; rss:   775MB ->   779MB	misc_checking_2
time:   8.188; rss:   779MB ->  1019MB	MIR_borrow_checking
time:   0.062; rss:  1019MB ->  1021MB	MIR_effect_checking
```
2021-02-01 14:29:40 +01:00
Ashley Mannix
8940a2652e stabilize int_bits_const 2021-01-31 21:50:47 +10:00
Tyson Nottingham
849dc1a20c Indicate both start and end of pass RSS in time-passes output
Previously, only the end of pass RSS was indicated. This could easily
lead one to believe that the change in RSS from one pass to the next was
attributable to the second pass, when in fact it occurred between the
end of the first pass and the start of the second.

Also, improve alignment of columns.
2021-01-29 12:46:29 -08:00
bors
a8f7075532 Auto merge of #80692 - Aaron1011:feature/query-result-debug, r=estebank
Enforce that query results implement Debug

Currently, we require that query keys implement `Debug`, but we do not do the same for query values. This can make incremental compilation bugs difficult to debug - there isn't a good place to print out the result loaded from disk.

This PR adds `Debug` bounds to several query-related functions, allowing us to debug-print the query value when an 'unstable fingerprint' error occurs. This required adding `#[derive(Debug)]` to a fairly large number of types - hopefully, this doesn't have much of an impact on compiler bootstrapping times.
2021-01-26 05:47:23 +00:00
Dániel Buga
f8416faaaf Clean up dominators_given_rpo 2021-01-24 13:32:18 +01:00
Joshua Nelson
394d7018b9 Add track_caller to .steal()
Before:

```
thread 'rustc' panicked at 'attempt to read from stolen value', /home/joshua/rustc/compiler/rustc_data_structures/src/steal.rs:43:15
```

After:

```
thread 'rustc' panicked at 'attempt to steal from stolen value', compiler/rustc_mir/src/transform/mod.rs:423:25
```
2021-01-17 12:27:20 -05:00
Aaron Hill
7afb32557d
Enforce that query results implement Debug 2021-01-16 17:53:02 -05:00
LingMan
a56bffb4f9 Use Option::map_or instead of .map(..).unwrap_or(..) 2021-01-14 19:23:59 +01:00
Tyson Nottingham
52f21791fb Serialize incr comp structures to file via fixed-size buffer
Reduce a large memory spike that happens during serialization by writing
the incr comp structures to file by way of a fixed-size buffer, rather
than an unbounded vector.

Effort was made to keep the instruction count close to that of the
previous implementation. However, buffered writing to a file inherently
has more overhead than writing to a vector, because each write may
result in a handleable error. To reduce this overhead, arrangements are
made so that each LEB128-encoded integer can be written to the buffer
with only one capacity and error check. Higher-level optimizations in
which entire composite structures can be written with one capacity and
error check are possible, but would require much more work.

The performance is mostly on par with the previous implementation, with
small to moderate instruction count regressions. The memory reduction is
significant, however, so it seems like a worth-while trade-off.
2021-01-11 12:13:22 -08:00
Tyson Nottingham
7c6274d464 rustc_serialize: have read_raw_bytes take MaybeUninit<u8> slice 2021-01-01 22:49:16 -08:00
Mark Rousskov
fe031180d0 Bump bootstrap compiler to 1.50 beta 2020-12-30 09:27:19 -05:00
Matthias Krüger
0c3af22e08 don't redundantly repeat field names 2020-12-29 22:26:58 +01:00
Bastian Kauschke
06cc9c26da stabilize min_const_generics 2020-12-26 18:24:10 +01:00
Yuki Okushi
c111404cb5
Rollup merge of #79612 - jyn514:compiler-links, r=Aaron1011
Switch some links in compiler/ to intra-doc links
2020-12-19 15:16:03 +09:00
Yuki Okushi
0765536c0b
Rollup merge of #78083 - ChaiTRex:master, r=m-ou-se
Stabilize or_insert_with_key

Stabilizes the `or_insert_with_key` feature from https://github.com/rust-lang/rust/issues/71024. This allows inserting key-derived values when a `HashMap`/`BTreeMap` entry is vacant.

The difference between this and  `.or_insert_with(|| ... )` is that this provides a reference to the key to the closure after it is moved with `.entry(key_being_moved)`, avoiding the need to copy or clone the key.
2020-12-19 15:15:57 +09:00
Joshua Nelson
35f16c60e7 Switch compiler/ to intra-doc links
rustc_lint and rustc_lint_defs weren't switched because they're included
in the compiler book and so can't use intra-doc links.
2020-12-18 15:22:51 -05:00
Camelid
810324d1f3 Rename optin_builtin_traits to auto_traits
They were originally called "opt-in, built-in traits" (OIBITs), but
people realized that the name was too confusing and a mouthful, and so
they were renamed to just "auto traits". The feature flag's name wasn't
updated, though, so that's what this PR does.

There are some other spots in the compiler that still refer to OIBITs,
but I don't think changing those now is worth it since they are internal
and not particularly relevant to this PR.

Also see <https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/opt-in.2C.20built-in.20traits.20(auto.20traits).20feature.20name>.
2020-11-23 14:14:06 -08:00
bors
8cfa7b4ec9 Auto merge of #78588 - HeroicKatora:sccc, r=nikomatsakis
Reworks Sccc computation to iteration instead of recursion

Linear graphs, producing as many scc's as nodes, would recurse once for every node when entered from the start of the list. This adds a test that exhausted the stack at least on my machine with error:

```
thread 'graph::scc::tests::test_deep_linear' has overflowed its stack
fatal runtime error: stack overflow
```

This may or may not be connected to #78567. I was only reminded that I started this rework some time ago. It might be plausible as borrow checking a long function with many borrow regions around each other—((((((…))))))— may produce the linear list setup to trigger this stack overflow ? I don't know enough about borrow check to say for sure.

This is best read in two separate commits. The first addresses only `find_state` internally. This is classical union phase from union-find. There's also a common solution of using the parent pointers in the (virtual) linked list to track the backreferences while traversing upwards and then following them backwards in a second path compression phase.

The second is more involved as it rewrites the mutually recursive `walk_node` and `walk_unvisited_node`. Firstly, the caller is required to handle the unvisited case of `walk_node` so a new `start_walk_from` method is added to handle that by walking the unvisited node if necessary. Then `walk_unvisited_node`, where we would previously recurse into in the missing case, is rewritten to construct a manual stack of its frames. The state fields consist of the previous stack slots.
2020-11-21 01:30:26 +00:00
Tyson Nottingham
142932ab19 Set unaligned_references lint to deny in rustc_data_structures
To detect misuse of private packed field in `PackedFingerprint`.
2020-11-20 01:13:15 -08:00
Tyson Nottingham
05dde137ca Make PackedFingerprint's Fingerprint private 2020-11-18 15:10:43 -08:00
Tyson Nottingham
f09d474836 Use PackedFingerprint in DepNode to reduce memory consumption 2020-11-18 12:49:09 -08:00
Mara Bos
fa45fce0d3
Rollup merge of #78702 - wesleywiser:self_profile_cgu_sizes, r=Mark-Simulacrum
[self-profiling] Include the estimated size of each cgu in the profile

This is helpful when looking for CGUs where the size estimate isn't a
good indicator of compilation time.

I verified that moving the profiling timer call doesn't affect the
results.

Results:

<img width="297" alt="Screen Shot 2020-11-03 at 7 25 04 AM" src="https://user-images.githubusercontent.com/831192/97985503-5901d100-1da6-11eb-9f10-f3e399702952.png">

`measureme` doesn't have support for custom arg names yet so `arg0` is the CGU name and `arg1` is the estimated size.
2020-11-17 16:13:49 +01:00
lcnr
a6cbd64dae words 2020-11-16 22:42:09 +01:00
Bastian Kauschke
2bf93bd852 compiler: fold by value 2020-11-16 22:34:57 +01:00
Bastian Kauschke
3ec6720bf1 add IdFunctor to rustc_data_structures 2020-11-16 22:27:20 +01:00
Jonas Schievink
ae1916b3b4
Rollup merge of #79058 - dtolnay:likelymacro, r=Mark-Simulacrum
Move likely/unlikely argument outside of invisible unsafe block

The previous `likely!`/`unlikely!` macros were unsound because it permits the caller's expr to contain arbitrary unsafe code.

```rust
pub fn huh() -> bool {
    likely!(std::ptr::read(&() as *const () as *const bool))
}
```

**Before:** compiles cleanly.
**After:**

```console
error[E0133]: call to unsafe function is unsafe and requires unsafe function or block
   |
70 |     likely!(std::ptr::read(&() as *const () as *const bool))
   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ call to unsafe function
   |
   = note: consult the function's documentation for information on how to avoid undefined behavior
```
2020-11-15 13:40:03 +01:00
David Tolnay
afb817054c
Move likely/unlikely argument outside of invisible unsafe block
The previous `likely!`/`unlikely!` macros were unsound because it
permits the caller's expr to contain arbitrary unsafe code.

    pub fn huh() -> bool {
        likely!(std::ptr::read(&() as *const () as *const bool))
    }

Before: compiles cleanly.
After:

    error[E0133]: call to unsafe function is unsafe and requires unsafe function or block
       |
    70 |     likely!(std::ptr::read(&() as *const () as *const bool))
       |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ call to unsafe function
       |
       = note: consult the function's documentation for information on how to avoid undefined behavior
2020-11-14 14:03:57 -08:00
Camille GILLOT
41c44b498f Move Steal to rustc_data_structures. 2020-11-14 01:30:56 +01:00
Andreas Molzer
eb597f5c4e Remove recursion from sccc walking
This allows constructing the sccc for large that visit many nodes before
finding a single cycle of sccc, for example lists. When used to find
dependencies in borrow checking the list case is what occurs in very
long functions.
2020-11-08 18:07:45 +01:00
Andreas Molzer
355904dca0 Add test for sccc of a long list 2020-11-05 19:24:49 +01:00
Andreas Molzer
a41e2fd963 Convert the recursive find_state to a loop
The basic conversion is a straightforward conversion of the linear
recursion to a loop forwards and backwards propagation of the result.
But this uses an optimization to avoid the need for extra space that
would otherwise be necessary to store the stack of unfinished states as
the function is not tail recursive.

Observe that only non-root-nodes in cycles have a recursive call and
that every such call overwrites their own node state. Thus we reuse the
node state itself as temporary storage for the stack of unfinished
states by inverting the links to a chain back to the previous state
update. When we hit the root or end of the full explored chain we
propagate the node state update backwards by following the chain until
a node with a link to itself.
2020-11-05 19:24:49 +01:00
Wesley Wiser
efe703a01a [self-profiling] Include the estimated size of each cgu in the profile
This is helpful when looking for CGUs where the size estimate isn't a
good indicator of compilation time.

I verified that moving the profiling timer call doesn't affect the
results.
2020-11-03 07:55:17 -05:00
Andreas Molzer
af72a70ee2 Move post order walk to iterative approach
The previous recursive approach might overflow the stack when walking a
particularly deep, list-like, graph. In particular, dominator
calculation for borrow checking does such a traversal and very long
functions might lead to a region dependency graph with in this
problematic structure.
2020-10-31 18:52:00 +01:00
Andreas Molzer
4fdf8a5630 Add a benchmark test for sccc finding
While a bit primitive, it should get us at least a better number than
nothing.
2020-10-31 01:05:15 +01:00