nested aux-build in tests/rustdoc/ tests
* Fixes bug that prevented using nested aux-build in `tests/rustdoc/` tests. Before, `fn document` and the auxiliary builder disagreed about where to find the nested aux-build source file (`auxiliary/auxiliary/aux.rs` vs `auxiliary/aux.rs`), preventing them from building. Picked the latter in line with other builders in compiletest.
* Adds `//@ doc-flags` header, which forwards flags to rustdoc and not rustc.
* Adds `//@ unique-doc-out-dir` header, which sets the --out-dir for the rustdoc invocation to a unique directory: `<root out dir>/docs/<test name>/doc`
* Changes working directory of the rustdoc invocation to the root out directory (common among all aux-builds). Prior art: exec_compiled_test in runtest.rs
* Adds tests that use nested aux builds and new headers
These changes provide useful capabilities for writing rustdoc tests on their own. They are also needed to test the implementation for the [mergable-rustdoc-cross-crate-info](https://github.com/rust-lang/rfcs/pull/3662) RFC.
try-job: x86_64-msvc
This is an alternative to ee6459d652.
That is, it fixes the issue that affects the very long type names
in https://docs.rs/async-stripe/0.31.0/stripe/index.html#structs.
This is, necessarily, a pile of nasty heuristics.
We need to balance a few issues:
- Sometimes, there's no real word break.
For example, `BTreeMap` should be `BTree<wbr>Map`,
not `B<wbr>Tree<wbr>Map`.
- Sometimes, there's a legit word break,
but the name is tiny and the HTML overhead isn't worth it.
For example, if we're typesetting `TyCtx`,
writing `Ty<wbr>Ctx` would have an HTML overhead of 50%.
Line breaking inside it makes no sense.
Remove unnecessary impl sorting in queries and metadata
Removes unnecessary impl sorting because queries already return their keys in HIR definition order: https://github.com/rust-lang/rust/issues/120371#issuecomment-1926422838
r? `@cjgillot` or `@lcnr` -- unless I totally misunderstood what was being asked for here? 😆fixes#120371
Forbid borrows and unsized types from being used as the type of a const generic under `adt_const_params`
Fixes#112219Fixes#112124Fixes#112125
### Motivation
Currently the `adt_const_params` feature allows writing `Foo<const N: [u8]>` this is entirely useless as it is not possible to write an expression which evaluates to a type that is not `Sized`. In order to actually use unsized types in const generics they are typically written as `const N: &[u8]` which *is* possible to provide a value of.
Unfortunately allowing the types of const parameters to contain references is non trivial (#120961) as it introduces a number of difficult questions about how equality of references in the type system should behave. References in the types of const generics is largely only useful for using unsized types in const generics.
This PR introduces a new feature gate `unsized_const_parameters` and moves support for `const N: [u8]` and `const N: &...` from `adt_const_params` into it. The goal here hopefully is to experiment with allowing `const N: [u8]` to work without references and then eventually completely forbid references in const generics.
Splitting this out into a new feature gate means that stabilization of `adt_const_params` does not have to resolve#120961 which is the only remaining "big" blocker for the feature. Remaining issues after this are a few ICEs and naming bikeshed for `ConstParamTy`.
### Implementation
The implementation is slightly subtle here as we would like to ensure that a stabilization of `adt_const_params` is forwards compatible with any outcome of `unsized_const_parameters`. This is inherently tricky as we do not support unstable trait implementations and we determine whether a type is valid as the type of a const parameter via a trait bound.
There are a few constraints here:
- We would like to *allow for the possibility* of adding a `Sized` supertrait to `ConstParamTy` in the event that we wind up opting to not support unsized types and instead requiring people to write the 'sized version', e.g. `const N: [u8; M]` instead of `const N: [u8]`.
- Crates should be able to enable `unsized_const_parameters` and write trait implementations of `ConstParamTy` for `!Sized` types without downstream crates that only enable `adt_const_params` being able to observe this (required for std to be able to `impl<T> ConstParamTy for [T]`
Ultimately the way this is accomplished is via having two traits (sad), `ConstParamTy` and `UnsizedConstParamTy`. Depending on whether `unsized_const_parameters` is enabled or not we change which trait is used to check whether a type is allowed to be a const parameter.
Long term (when stabilizing `UnsizedConstParamTy`) it should be possible to completely merge these traits (and derive macros), only having a single `trait ConstParamTy` and `macro ConstParamTy`.
Under `adt_const_params` it is now illegal to directly refer to `ConstParamTy` it is only used as an internal impl detail by `derive(ConstParamTy)` and checking const parameters are well formed. This is necessary in order to ensure forwards compatibility with all possible future directions for `feature(unsized_const_parameters)`.
Generally the intuition here should be that `ConstParamTy` is the stable trait that everything uses, and `UnsizedConstParamTy` is that plus unstable implementations (well, I suppose `ConstParamTy` isn't stable yet :P).
rustdoc: Add support for `missing_unsafe_on_extern` feature
Follow-up of https://github.com/rust-lang/rust/pull/124482.
Not sure if the `safe` keyword is supposed to be displayed or not though? For now I didn't add it in the generated doc, only `unsafe` as usual.
cc `@spastorino`
r? `@fmease`
Stabilize `custom_code_classes_in_docs` feature
Fixes#79483.
This feature has been around for quite some time now, I think it's fine to stabilize it now.
## Summary
## What is the feature about?
In short, this PR changes two things, both related to codeblocks in doc comments in Rust documentation:
* Allow to disable generation of `language-*` CSS classes with the `custom` attribute.
* Add your own CSS classes to a code block so that you can use other tools to highlight them.
#### The `custom` attribute
Let's start with the new `custom` attribute: it will disable the generation of the `language-*` CSS class on the generated HTML code block. For example:
```rust
/// ```custom,c
/// int main(void) {
/// return 0;
/// }
/// ```
```
The generated HTML code block will not have `class="language-c"` because the `custom` attribute has been set. The `custom` attribute becomes especially useful with the other thing added by this feature: adding your own CSS classes.
#### Adding your own CSS classes
The second part of this feature is to allow users to add CSS classes themselves so that they can then add a JS library which will do it (like `highlight.js` or `prism.js`), allowing to support highlighting for other languages than Rust without increasing burden on rustdoc. To disable the automatic `language-*` CSS class generation, you need to use the `custom` attribute as well.
This allow users to write the following:
```rust
/// Some code block with `{class=language-c}` as the language string.
///
/// ```custom,{class=language-c}
/// int main(void) {
/// return 0;
/// }
/// ```
fn main() {}
```
This will notably produce the following HTML:
```html
<pre class="language-c">
int main(void) {
return 0;
}</pre>
```
Instead of:
```html
<pre class="rust rust-example-rendered">
<span class="ident">int</span> <span class="ident">main</span>(<span class="ident">void</span>) {
<span class="kw">return</span> <span class="number">0</span>;
}
</pre>
```
To be noted, we could have written `{.language-c}` to achieve the same result. `.` and `class=` have the same effect.
One last syntax point: content between parens (`(like this)`) is now considered as comment and is not taken into account at all.
In addition to this, I added an `unknown` field into `LangString` (the parsed code block "attribute") because of cases like this:
```rust
/// ```custom,class:language-c
/// main;
/// ```
pub fn foo() {}
```
Without this `unknown` field, it would generate in the DOM: `<pre class="language-class:language-c language-c">`, which is quite bad. So instead, it now stores all unknown tags into the `unknown` field and use the first one as "language". So in this case, since there is no unknown tag, it'll simply generate `<pre class="language-c">`. I added tests to cover this.
EDIT(camelid): This description is out-of-date. Using `custom,class:language-c` will generate the output `<pre class="language-class:language-c">` as would be expected; it treats `class:language-c` as just the name of a language (similar to the langstring `c` or `js` or what have you) since it does not use the designed class syntax.
Finally, I added a parser for the codeblock attributes to make it much easier to maintain. It'll be pretty easy to extend.
As to why this syntax for adding attributes was picked: it's [Pandoc's syntax](https://pandoc.org/MANUAL.html#extension-fenced_code_attributes). Even if it seems clunkier in some cases, it's extensible, and most third-party Markdown renderers are smart enough to ignore Pandoc's brace-delimited attributes (from [this comment](https://github.com/rust-lang/rust/pull/110800#issuecomment-1522044456)).
r? `@notriddle`
rustdoc: Clarify const-stability with regard to normal stability
Fixes#125511.
- Elide const-unstable if also unstable overall
- Show "const" for const-unstable if also overall unstable
If a const function is unstable overall (and thus, in all circumstances
I know of, also const-unstable), we should show the option to use it as
const. You need to enable a feature to use the function at all anyway.
If the function is stabilized without also being const-stabilized, then
we do not show the const keyword and instead show "const: unstable" in
the version info.
It's confusing because if a function is unstable overall, there's no
need to highlight the constness is also unstable. Technically, these
attributes (overall stability and const-stability) are separate, but in
practice, we don't even show the const-unstable's feature flag (it's
normally the same as the overall function).
To decide if internal items should be inlined in a doc page,
check if the crate is itself internal, rather than if it has
the rustc_private feature flag. The standard library uses
internal items, but is not itself internal and should not show
internal items on its docs pages.
Correctly handle inlining of doc hidden foreign items
Fixes#123435.
In case a foreign item has doc(hidden) attribute, we simply merged its attributes with the re-export's, making it being removed once in the `strip_hidden` pass.
The solution was to use the same as for local reexported items: merge attributes, but not some of them (like `doc(hidden)`).
I originally checked if we could simply update `Item::is_doc_hidden` method to use `self.inline_stmt_id.is_some_and(|def_id| tcx.is_doc_hidden(def_id))` but unfortunately, it added (local) items that shouldn't be inlined. At least it unifies local and foreign items inlining, which I think is the best course of action here.
r? `@notriddle`
rustdoc-search: shard the search result descriptions
## Preview
This makes no visual changes to rustdoc search. It's a pure perf improvement.
<details><summary>old</summary>
Preview: <http://notriddle.com/rustdoc-html-demo-10/doc/std/index.html?search=vec>
WebPageTest Comparison with before branch on a sort of worst case (searching `vec`, winds up downloading most of the shards anyway): <https://www.webpagetest.org/video/compare.php?tests=240317_AiDc61_2EM,240317_AiDcM0_2EN>
Waterfall diagram:
</details>
Preview: <http://notriddle.com/rustdoc-html-demo-10/doc2/std/index.html?search=vec>
WebPageTest Comparison with before branch on a sort of worst case (searching `vec`, winds up downloading most of the shards anyway): <https://www.webpagetest.org/video/compare.php?tests=240322_BiDcCH_13R,240322_AiDcJY_104>
## Description
r? `@GuillaumeGomez`
The descriptions are, on almost all crates[^1], the majority of the size of the search index, even though they aren't really used for searching. This makes it relatively easy to separate them into their own files.
Additionally, this PR pulls out information about whether there's a description into a bitmap. This allows us to sort, truncate, *then* download.
This PR also bumps us to ES8. Out of the browsers we support, all of them support async functions according to caniuse.
https://caniuse.com/async-functions
[^1]:
<https://microsoft.github.io/windows-docs-rs/>, a crate with
44MiB of pure names and no descriptions for them, is an outlier
and should not be counted. But this PR should improve it, by replacing a long line of empty strings with a compressed bitmap with a single Run section. Just not very much.
## Detailed sizes
```console
$ cat test.sh
set -ex
cp ../search-index*.js search-index.js
awk 'FNR==NR {a++;next} FNR<a-3' search-index.js{,} | awk 'NR>1 {gsub(/\],\\$/,""); gsub(/^\["[^"]+",/,""); print} {next}' | sed -E "s:\\\\':':g" > search-index.json
jq -c '.t' search-index.json > t.json
jq -c '.n' search-index.json > n.json
jq -c '.q' search-index.json > q.json
jq -c '.D' search-index.json > D.json
jq -c '.e' search-index.json > e.json
jq -c '.i' search-index.json > i.json
jq -c '.f' search-index.json > f.json
jq -c '.c' search-index.json > c.json
jq -c '.p' search-index.json > p.json
jq -c '.a' search-index.json > a.json
du -hs t.json n.json q.json D.json e.json i.json f.json c.json p.json a.json
$ bash test.sh
+ cp ../search-index1.78.0.js search-index.js
+ awk 'FNR==NR {a++;next} FNR<a-3' search-index.js search-index.js
+ awk 'NR>1 {gsub(/\],\\$/,""); gsub(/^\["[^"]+",/,""); print} {next}'
+ sed -E 's:\\'\'':'\'':g'
+ jq -c .t search-index.json
+ jq -c .n search-index.json
+ jq -c .q search-index.json
+ jq -c .D search-index.json
+ jq -c .e search-index.json
+ jq -c .i search-index.json
+ jq -c .f search-index.json
+ jq -c .c search-index.json
+ jq -c .p search-index.json
+ jq -c .a search-index.json
+ du -hs t.json n.json q.json D.json e.json i.json f.json c.json p.json a.json
64K t.json
800K n.json
8.0K q.json
4.0K D.json
16K e.json
192K i.json
544K f.json
4.0K c.json
36K p.json
20K a.json
```
These are, roughly, the size of each section in the standard library (this tool actually excludes libtest, for parsing-json-with-awk reasons, but libtest is tiny so it's probably not important).
t = item type, like "struct", "free fn", or "type alias". Since one byte is used for every item, this implies that there are approximately 64 thousand items in the standard library.
n = name, and that's now the largest section of the search index with the descriptions removed from it
q = parent *module* path, stored parallel to the items within
D = the size of each description shard, stored as vlq hex numbers
e = empty description bit flags, stored as a roaring bitmap
i = parent *type* index as a link into `p`, stored as decimal json numbers; used only for associated types; might want to switch to vlq hex, since that's shorter, but that would be a separate pr
f = function signature, stored as lists of lists that index into `p`
c = deprecation flag, stored as a roaring bitmap
p = parent *type*, stored separately and linked into from `i` and `f`
a = alias, as [[key, value]] pairs
## Search performance
http://notriddle.com/rustdoc-html-demo-11/perf-shard/index.html
For example, in stm32f4:
<table><thead><tr><th>before<th>after</tr></thead>
<tbody><tr><td>
```
Testing T -> U ... in_args = 0, returned = 0, others = 200
wall time = 617
Testing T, U ... in_args = 0, returned = 0, others = 200
wall time = 198
Testing T -> T ... in_args = 0, returned = 0, others = 200
wall time = 282
Testing crc32 ... in_args = 0, returned = 0, others = 0
wall time = 426
Testing spi::pac ... in_args = 0, returned = 0, others = 0
wall time = 673
```
</td><td>
```
Testing T -> U ... in_args = 0, returned = 0, others = 200
wall time = 716
Testing T, U ... in_args = 0, returned = 0, others = 200
wall time = 207
Testing T -> T ... in_args = 0, returned = 0, others = 200
wall time = 289
Testing crc32 ... in_args = 0, returned = 0, others = 0
wall time = 418
Testing spi::pac ... in_args = 0, returned = 0, others = 0
wall time = 687
```
</td></tr><tr><td>
```
user: 005.345 s
sys: 002.955 s
wall: 006.899 s
child_RSS_high: 583664 KiB
group_mem_high: 557876 KiB
```
</td><td>
```
user: 004.652 s
sys: 000.565 s
wall: 003.865 s
child_RSS_high: 538696 KiB
group_mem_high: 511724 KiB
```
</td></tr>
</table>
This perf tester is janky and unscientific enough that the apparent differences might just be noise. If it's not an order of magnitude, it's probably not real.
## Future possibilities
* Currently, results are not shown until the descriptions are downloaded. Theoretically, the description-less results could be shown. But actually doing that, and making sure it works properly, would require extra work (we have to be careful to avoid layout jumps).
* More than just descriptions can be sharded this way. But we have to be careful to make sure the size wins are worth the round trips. Ideally, data that’s needed only for display should be sharded while data needed for search isn’t.
* [Full text search](https://internals.rust-lang.org/t/full-text-search-for-rustdoc-and-doc-rs/20427) also needs this kind of infrastructure. A good implementation might store a compressed bloom filter in the search index, then download the full keyword in shards. But, we have to be careful not just of the amount readers have to download, but also of the amount that [publishers](https://gist.github.com/notriddle/c289e77f3ed469d1c0238d1d135d49e1) have to store.
