rustdoc-search: use set ops for ranking and filtering
This commit adds ranking and quick filtering to type-based search, improving performance and having it order results based on their type signatures.
Preview
-------
Profiler output: https://notriddle.com/rustdoc-html-demo-6/profile-8/index.html
Preview: https://notriddle.com/rustdoc-html-demo-6/ranking-and-filtering-v2/std/index.html
Motivation
----------
If I write a query like `str -> String`, a lot of functions come up. That's to be expected, but `String::from` should come up on top, and it doesn't right now. This is because the sorting algorithm is based on the functions name, and doesn't consider the type signature at all. `slice::join` even comes up above it!
To fix this, the sorting should take into account the function's signature, and the closer match should come up on top.
Guide-level description
-----------------------
When searching by type signature, types with a "closer" match will show up above types that match less precisely.
Reference-level explanation
---------------------------
Functions signature search works in three major phases:
* A compact "fingerprint," based on the [bloom filter] technique, is used to check for matches and to estimate the distance. It sometimes has false positive matches, but it also operates on 128 bit contiguous memory and requires no backtracking, so it performs a lot better than real unification.
The fingerprint represents the set of items in the type signature, but it does not represent nesting, and it ignores when the same item appears more than once.
The result is rejected if any query bits are absent in the function, or if the distance is higher than the current maximum and 200 results have already been found.
* The second step performs unification. This is where nesting and true bag semantics are taken into account, and it has no false positives. It uses a recursive, backtracking algorithm.
The result is rejected if any query elements are absent in the function.
[bloom filter]: https://en.wikipedia.org/wiki/Bloom_filter
Drawbacks
---------
This makes the code bigger.
More than that, this design is a subtle trade-off. It makes the cases I've tested against measurably faster, but it's not clear how well this extends to other crates with potentially more functions and fewer types.
The more complex things get, the more important it is to gather a good set of data to test with (this is arguably more important than the actual benchmarking ifrastructure right now).
Rationale and alternatives
--------------------------
Throwing a bloom filter in front makes it faster.
More than that, it tries to take a tactic where the system can not only check for potential matches, but also gets an accurate distance function without needing to do unification. That way it can skip unification even on items that have the needed elems, as long as they have more items than the currently found maximum.
If I didn't want to be able to cheaply do set operations on the fingerprint, a [cuckoo filter] is supposed to have better performance. But the nice bit-banging set intersection doesn't work AFAIK.
I also looked into [minhashing], but since it's actually an unbiased estimate of the similarity coefficient, I'm not sure how it could be used to skip unification (I wouldn't know if the estimate was too low or too high).
This function actually uses the number of distinct items as its "distance function." This should give the same results that it would have gotten from a Jaccard Distance $1-\frac{|F\cap{}Q|}{|F\cup{}Q|}$, while being cheaper to compute. This is because:
* The function $F$ must be a superset of the query $Q$, so their union is just $F$ and the intersection is $Q$ and it can be reduced to $1-\frac{|Q|}{|F|}.
* There are no magic thresholds. These values are only being used to compare against each other while sorting (and, if 200 results are found, to compare with the maximum match). This means we only care if one value is bigger than the other, not what it's actual value is, and since $Q$ is the same for everything, it can be safely left out, reducing the formula to $1-\frac{1}{|F|} = \frac{|F|}{|F|}-\frac{1}{|F|} = |F|-1$. And, since the values are only being compared with each other, $|F|$ is fine.
Prior art
---------
This is significantly different from how Hoogle does it.
It doesn't account for order, and it has no special account for nesting, though `Box<t>` is still two items, while `t` is only one.
This should give the same results that it would have gotten from a Jaccard Distance $1-\frac{|A\cap{}B|}{|A\cup{}B|}$, while being cheaper to compute.
Unresolved questions
--------------------
`[]` and `()`, the slice/array and tuple/union operators, are ignored while building the signature for the query. This is because they match more than one thing, making them ambiguous. Unfortunately, this also makes them a performance cliff. Is this likely to be a problem?
Right now, the system just stashes the type distance into the same field that levenshtein distance normally goes in. This means exact query matches show up on top (for example, if you have a function like `fn nothing(a: Nothing, b: i32)`, then searching for `nothing` will show it on top even if there's another function with `fn bar(x: Nothing)` that's technically a closer match in type signature.
Future possibilities
--------------------
It should be possible to adopt more sorting criteria to act as a tie breaker, which could be determined during unification.
[cuckoo filter]: https://en.wikipedia.org/wiki/Cuckoo_filter
[minhashing]: https://en.wikipedia.org/wiki/MinHash
Add more suggestions to unexpected cfg names and values
This pull request adds more suggestion to unexpected cfg names and values diagnostics:
- it first adds a links to the [rustc unstable book](https://doc.rust-lang.org/nightly/unstable-book/compiler-flags/check-cfg.html) or the [Cargo reference](https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#check-cfg), depending if rustc is invoked by Cargo
- it secondly adds a suggestion on how to expect the cfg name or value:
*excluding well known names and values*
- for Cargo: it suggest using a feature or `cargo:rust-check-cfg` in build script
- for rustc: it suggest using `--check-cfg` (with the correct invocation)
Those diagnostics improvements are directed towards enabling users to fix the issue if the previous suggestions weren't good enough.
r? `@petrochenkov`
This commit adds ranking and quick filtering to type-based search,
improving performance and having it order results based on their
type signatures.
Motivation
----------
If I write a query like `str -> String`, a lot of functions come up.
That's to be expected, but `String::from_str` should come up on top, and
it doesn't right now. This is because the sorting algorithm is based
on the functions name, and doesn't consider the type signature at all.
`slice::join` even comes up above it!
To fix this, the sorting should take into account the function's
signature, and the closer match should come up on top.
Guide-level description
-----------------------
When searching by type signature, types with a "closer" match will
show up above types that match less precisely.
Reference-level explanation
---------------------------
Functions signature search works in three major phases:
* A compact "fingerprint," based on the [bloom filter] technique, is used to
check for matches and to estimate the distance. It sometimes has false
positive matches, but it also operates on 128 bit contiguous memory and
requires no backtracking, so it performs a lot better than real
unification.
The fingerprint represents the set of items in the type signature, but it
does not represent nesting, and it ignores when the same item appears more
than once.
The result is rejected if any query bits are absent in the function, or
if the distance is higher than the current maximum and 200
results have already been found.
* The second step performs unification. This is where nesting and true bag
semantics are taken into account, and it has no false positives. It uses a
recursive, backtracking algorithm.
The result is rejected if any query elements are absent in the function.
[bloom filter]: https://en.wikipedia.org/wiki/Bloom_filter
Drawbacks
---------
This makes the code bigger.
More than that, this design is a subtle trade-off. It makes the cases I've
tested against measurably faster, but it's not clear how well this extends
to other crates with potentially more functions and fewer types.
The more complex things get, the more important it is to gather a good set
of data to test with (this is arguably more important than the actual
benchmarking ifrastructure right now).
Rationale and alternatives
--------------------------
Throwing a bloom filter in front makes it faster.
More than that, it tries to take a tactic where the system can not only check
for potential matches, but also gets an accurate distance function without
needing to do unification. That way it can skip unification even on items
that have the needed elems, as long as they have more items than the
currently found maximum.
If I didn't want to be able to cheaply do set operations on the fingerprint,
a [cuckoo filter] is supposed to have better performance.
But the nice bit-banging set intersection doesn't work AFAIK.
I also looked into [minhashing], but since it's actually an unbiased
estimate of the similarity coefficient, I'm not sure how it could be used
to skip unification (I wouldn't know if the estimate was too low or
too high).
This function actually uses the number of distinct items as its
"distance function."
This should give the same results that it would have gotten from a Jaccard
Distance $1-\frac{|F\cap{}Q|}{|F\cup{}Q|}$, while being cheaper to compute.
This is because:
* The function $F$ must be a superset of the query $Q$, so their union is
just $F$ and the intersection is $Q$ and it can be reduced to
$1-\frac{|Q|}{|F|}.
* There are no magic thresholds. These values are only being used to
compare against each other while sorting (and, if 200 results are found,
to compare with the maximum match). This means we only care if one value
is bigger than the other, not what it's actual value is, and since $Q$ is
the same for everything, it can be safely left out, reducing the formula
to $1-\frac{1}{|F|} = \frac{|F|}{|F|}-\frac{1}{|F|} = |F|-1$. And, since
the values are only being compared with each other, $|F|$ is fine.
Prior art
---------
This is significantly different from how Hoogle does it.
It doesn't account for order, and it has no special account for nesting,
though `Box<t>` is still two items, while `t` is only one.
This should give the same results that it would have gotten from a Jaccard
Distance $1-\frac{|A\cap{}B|}{|A\cup{}B|}$, while being cheaper to compute.
Unresolved questions
--------------------
`[]` and `()`, the slice/array and tuple/union operators, are ignored while
building the signature for the query. This is because they match more than
one thing, making them ambiguous. Unfortunately, this also makes them
a performance cliff. Is this likely to be a problem?
Right now, the system just stashes the type distance into the
same field that levenshtein distance normally goes in. This means exact
query matches show up on top (for example, if you have a function like
`fn nothing(a: Nothing, b: i32)`, then searching for `nothing` will show it
on top even if there's another function with `fn bar(x: Nothing)` that's
technically a closer match in type signature.
Future possibilities
--------------------
It should be possible to adopt more sorting criteria to act as a tie breaker,
which could be determined during unification.
[cuckoo filter]: https://en.wikipedia.org/wiki/Cuckoo_filter
[minhashing]: https://en.wikipedia.org/wiki/MinHash
Coroutine variant fields can be uninitialized
Wrap coroutine variant fields in MaybeUninit to indicate that they might be uninitialized. Otherwise an uninhabited field will make the entire variant uninhabited and introduce undefined behaviour.
The analogous issue in the prefix of coroutine layout was addressed by 6fae7f8071.
Support bare unit structs in destructuring assignments
We should be allowed to use destructuring assignments on *bare* unit structs, not just unit structs that are located within other pattern constructors.
Fixes#118753
r? petrochenkov since you reviewed #95380, reassign if you're busy or don't want to review this.
codegen: panic when trying to compute size/align of extern type
The alignment is also computed when accessing a field of extern type at non-zero offset, so we also panic in that case.
Previously `size_of_val` worked because the code path there assumed that "thin pointer" means "sized". But that's not true any more with extern types. The returned size and align are just blatantly wrong, so it seems better to panic than returning wrong results. We use a non-unwinding panic since code probably does not expect size_of_val to panic.
[`RFC 3086`] Attempt to try to resolve blocking concerns
Implements what is described at https://github.com/rust-lang/rust/issues/83527#issuecomment-1744822345 to hopefully make some progress.
It is unknown if such approach is or isn't desired due to the lack of further feedback, as such, it is probably best to nominate this PR to the official entities.
`@rustbot` labels +I-compiler-nominated
Actually parse async gen blocks correctly
1. I got the control flow in `parse_expr_bottom` messed up, and obviously forgot a test for `async gen`, so we weren't actually ever parsing it correctly.
2. I forgot to gate the span for `async gen {}`, so even if we did parse it, we wouldn't have correctly denied it in `cfg(FALSE)`.
r? eholk
Add rustX check to codeblock attributes lint
We discovered this issue [here](https://github.com/rust-lang/rust/pull/118802#discussion_r1421815943).
I assume that the issue will be present in other places outside of the compiler so it's worth adding a check for it.
First commit is just a small cleanup about variables creation which was a bit strange (at least more than necessary).
r? ```@notriddle```
Fix alignment passed down to LLVM for simd_masked_load
Follow up to #117953
The alignment for a masked load operation should be that of the element/lane, not the vector as a whole
It can produce miscompilations after the LLVM optimizer notices the higher alignment and promotes this to an unmasked, aligned load followed up by blend/select - https://rust.godbolt.org/z/KEeGbevbb
tests: CGU tests require build-pass, not check-pass (remove FIXME)
CGU tests require CGU code to be exercised. We can't merely do "cargo check" on these tests.
Part of #62277
Correctly gate the parsing of match arms without body
https://github.com/rust-lang/rust/pull/118527 accidentally allowed the following to parse on stable:
```rust
match Some(0) {
None => { foo(); }
#[cfg(FALSE)]
Some(_)
}
```
This fixes that oversight. The way I choose which error to emit is the best I could think of, I'm open if you know a better way.
r? `@petrochenkov` since you're the one who noticed
Improve an error involving attribute values.
Attribute values must be literals. The error you get when that doesn't hold is pretty bad, e.g.:
```
unexpected expression: 1 + 1
```
You also get the same error if the attribute value is a literal, but an invalid literal, e.g.:
```
unexpected expression: "foo"suffix
```
This commit does two things.
- Changes the error message to "attribute value must be a literal", which gives a better idea of what the problem is and how to fix it. It also no longer prints the invalid expression, because the carets below highlight it anyway.
- Separates the "not a literal" case from the "invalid literal" case. Which means invalid literals now get the specific error at the literal level, rather than at the attribute level.
r? `@compiler-errors`
On borrow return type, suggest borrowing from arg or owned return type
When we encounter a function with a return type that has an anonymous lifetime with no argument to borrow from, besides suggesting the `'static` lifetime we now also suggest changing the arguments to be borrows or changing the return type to be an owned type.
```
error[E0106]: missing lifetime specifier
--> $DIR/variadic-ffi-6.rs:7:6
|
LL | ) -> &usize {
| ^ expected named lifetime parameter
|
= help: this function's return type contains a borrowed value, but there is no value for it to be borrowed from
help: consider using the `'static` lifetime, but this is uncommon unless you're returning a borrowed value from a `const` or a `static`
|
LL | ) -> &'static usize {
| +++++++
help: instead, you are more likely to want to change one of the arguments to be borrowed...
|
LL | x: &usize,
| +
help: ...or alternatively, to want to return an owned value
|
LL - ) -> &usize {
LL + ) -> usize {
|
```
Fix#85843.
dont ICE when ConstKind::Expr for is_const_evaluatable
The problem is that we are not handling ConstKind::Expr inside report_not_const_evaluatable_error
Fixes [#114151]
End locals' live range before suspending coroutine
State transforms retains storage statements for locals that are not
stored inside a coroutine. It ensures those locals are live when
resuming by inserting StorageLive as appropriate. It forgot to end the
storage of those locals when suspending, which is fixed here.
While the end of live range is implicit when executing return, it is
nevertheless useful for inliner which would otherwise extend the live
range beyond return.
Fixes#117733
Attribute values must be literals. The error you get when that doesn't
hold is pretty bad, e.g.:
```
unexpected expression: 1 + 1
```
You also get the same error if the attribute value is a literal, but an
invalid literal, e.g.:
```
unexpected expression: "foo"suffix
```
This commit does two things.
- Changes the error message to "attribute value must be a literal",
which gives a better idea of what the problem is and how to fix it. It
also no longer prints the invalid expression, because the carets below
highlight it anyway.
- Separates the "not a literal" case from the "invalid literal" case.
Which means invalid literals now get the specific error at the literal
level, rather than at the attribute level.
Wrap coroutine variant fields in MaybeUninit to indicate that they
might be uninitialized. Otherwise an uninhabited field will make
the entire variant uninhabited and introduce undefined behaviour.
The analogous issue in the prefix of coroutine layout was addressed by
6fae7f8071.
State transforms retains storage statements for locals that are not
stored inside a coroutine. It ensures those locals are live when
resuming by inserting StorageLive as appropriate. It forgot to end the
storage of those locals when suspending, which is fixed here.
While the end of live range is implicit when executing return, it is
nevertheless useful for inliner which would otherwise extend the live
range beyond return.
`.debug_pubnames` and `.debug_pubtypes` are poorly designed and people
seldom use them. However, they take a considerable portion of size in
the final binary. This tells LLVM stop emitting those sections on
DWARFv4 or lower. DWARFv5 use `.debug_names` which is more concise
in size and performant for name lookup.
Do not parenthesize exterior struct lit inside match guards
Before this PR, the AST pretty-printer injects parentheses around expressions any time parens _could_ be needed depending on what else is in the code that surrounds that expression. But the pretty-printer did not pass around enough context to understand whether parentheses really _are_ needed on any particular expression. As a consequence, there are false positives where unneeded parentheses are being inserted.
Example:
```rust
#![feature(if_let_guard)]
macro_rules! pp {
($e:expr) => {
stringify!($e)
};
}
fn main() {
println!("{}", pp!(match () { () if let _ = Struct {} => {} }));
}
```
**Before:**
```console
match () { () if let _ = (Struct {}) => {} }
```
**After:**
```console
match () { () if let _ = Struct {} => {} }
```
This PR introduces a bit of state that is passed across various expression printing methods to help understand accurately whether particular situations require parentheses injected by the pretty printer, and it fixes one such false positive involving match guards as shown above.
There are other parenthesization false positive cases not fixed by this PR. I intend to address these in follow-up PRs. For example here is one: the expression `{ let _ = match x {} + 1; }` is pretty-printed as `{ let _ = (match x {}) + 1; }` despite there being no reason for parentheses to appear there.
Add lint against ambiguous wide pointer comparisons
This PR is the resolution of https://github.com/rust-lang/rust/issues/106447 decided in https://github.com/rust-lang/rust/issues/117717 by T-lang.
## `ambiguous_wide_pointer_comparisons`
*warn-by-default*
The `ambiguous_wide_pointer_comparisons` lint checks comparison of `*const/*mut ?Sized` as the operands.
### Example
```rust
let ab = (A, B);
let a = &ab.0 as *const dyn T;
let b = &ab.1 as *const dyn T;
let _ = a == b;
```
### Explanation
The comparison includes metadata which may not be expected.
-------
This PR also drops `clippy::vtable_address_comparisons` which is superseded by this one.
~~One thing: is the current naming right? `invalid` seems a bit too much.~~
Fixes https://github.com/rust-lang/rust/issues/117717
Remove edition umbrella features.
In the 2018 edition, there was an "umbrella" feature `#[feature(rust_2018_preview)]` which was used to enable several other features at once. This umbrella mechanism was not used in the 2021 edition and likely will not be used in 2024 either. During 2018 users reported that setting the feature was awkward, especially since they already needed to opt-in via the edition mechanism.
This PR removes this mechanism because I believe it will not be used (and will clean up and simplify the code). I believe that there are better ways to handle features and editions. In short:
- For highly experimental features, that may or may not be involved in an edition, they can implement regular feature gates like `tcx.features().my_feature`.
- For experimental features that *might* be involved in an edition, they should implement gates with `tcx.features().my_feature && span.at_least_rust_20xx()`. This requires the user to still specify `#![feature(my_feature)]`, to avoid disrupting testing of other edition features which are ready and have been accepted within the edition.
- For experimental features that have graduated to definitely be part of an edition, they should implement gates with `tcx.features().my_feature || span.at_least_rust_20xx()`, or just remove the feature check altogether and just check `span.at_least_rust_20xx()`.
- For relatively simple changes, they can skip the whole feature gating thing and just check `span.at_least_rust_20xx()`, and rely on the instability of the edition itself (which requires `-Zunstable-options`) to gate it.
I am working on documenting all of this in the rustc-dev-guide.
Implement repr(packed) for repr(simd)
This allows creating vectors with non-power-of-2 lengths that do not have padding. See rust-lang/portable-simd#319
Rearrange `default_configuration` and `CheckCfg::fill_well_known`.
There are comments saying these two functions should be kept in sync, but they have very different structures, process symbols in different orders, and there are some inconsistencies.
This commit reorders them so they're both mostly processing symbols in alphabetical order, which makes cross-checking them a lot easier. The commit also adds some macros to factor out repetitive code patterns.
The commit also moves the handling of `sym::test` out of `build_configuration` into `default_configuration`, where all the other symbols are handled.
r? `@bjorn3`
guarantee that char and u32 are ABI-compatible
In https://github.com/rust-lang/rust/pull/116894 we added a guarantee that `char` has the same alignment as `u32`, but there is still one axis where these types could differ: function call ABI. So let's nail that down as well: in a function signature, `char` and `u32` are completely equivalent.
This is a new stable guarantee, so it will need t-lang approval.
