Autodiff batching2
~I will rebase it once my first PR landed.~ done.
This autodiff batch mode is more similar to scalar autodiff, since it still only takes one shadow argument.
However, that argument is supposed to be `width` times larger.
r? `@oli-obk`
Tracking:
- https://github.com/rust-lang/rust/issues/124509
This will allow us to eagerly translate messages on a top-level
diagnostic, such as a `LintDiagnostic`. As a bonus, we can remove the
awkward closure passed into Subdiagnostic and make better use of
`Into`.
I'm removing empty identifiers everywhere, because in practice they
always mean "no identifier" rather than "empty identifier". (An empty
identifier is impossible.) It's better to use `Option` to mean "no
identifier" because you then can't forget about the "no identifier"
possibility.
Some specifics:
- When testing an attribute for a single name, the commit uses the
`has_name` method.
- When testing an attribute for multiple names, the commit uses the new
`has_any_name` method.
- When using `match` on an attribute, the match arms now have `Some` on
them.
In the tests, we now avoid printing empty identifiers by not printing
the identifier in the `error:` line at all, instead letting the carets
point out the problem.
Rollup of 9 pull requests
Successful merges:
- #138336 (Improve `-Z crate-attr` diagnostics)
- #139636 (Encode dep node edge count as u32 instead of usize)
- #139666 (cleanup `mir_borrowck`)
- #139695 (compiletest: consistently use `camino::{Utf8Path,Utf8PathBuf}` throughout)
- #139699 (Proactively update coroutine drop shim's phase to account for later passes applied during shim query)
- #139718 (enforce unsafe attributes in pre-2024 editions by default)
- #139722 (Move some things to rustc_type_ir)
- #139760 (UI tests: migrate remaining compile time `error-pattern`s to line annotations when possible)
- #139776 (Switch attrs to `diagnostic::on_unimplemented`)
r? `@ghost`
`@rustbot` modify labels: rollup
- Show the `#![ ... ]` in the span (to make it clear that it should not
be included in the CLI argument)
- Show more detailed errors when the crate has valid token trees but
invalid syntax.
Previously, `crate-attr=feature(foo),feature(bar)` would just say
"invalid crate attribute" and point at the comma. Now, it explicitly
says that the comma was unexpected, which is useful when using
`--error-format=short`. It also fixes the column to show the correct
span.
- Recover from parse errors. Previously we would abort immediately on
syntax errors; now we go on to try and type-check the rest of the
crate.
The new diagnostic code also happens to be slightly shorter.
Rename some `name` variables as `ident`.
It bugs me when variables of type `Ident` are called `name`. It leads to silly things like `name.name`. `Ident` variables should be called `ident`, and `name` should be used for variables of type `Symbol`.
This commit improves things by by doing `s/name/ident/` on a bunch of `Ident` variables. Not all of them, but a decent chunk.
r? `@fee1-dead`
It bugs me when variables of type `Ident` are called `name`. It leads to
silly things like `name.name`. `Ident` variables should be called
`ident`, and `name` should be used for variables of type `Symbol`.
This commit improves things by by doing `s/name/ident/` on a bunch of
`Ident` variables. Not all of them, but a decent chunk.
- fix errors caused by the move of `ast::Item::ident` (see #138740)
- move the logic of getting `sig`, `vis`, and `ident` from two seperate
`match` statements into one (less repetition especially with the
nested `match`)
Autodiff batching
Enzyme supports batching, which is especially known from the ML side when training neural networks.
There we would normally have a training loop, where in each iteration we would pass in some data (e.g. an image), and a target vector. Based on how close we are with our prediction we compute our loss, and then use backpropagation to compute the gradients and update our weights.
That's quite inefficient, so what you normally do is passing in a batch of 8/16/.. images and targets, and compute the gradients for those all at once, allowing better optimizations.
Enzyme supports batching in two ways, the first one (which I implemented here) just accepts a Batch size,
and then each Dual/Duplicated argument has not one, but N shadow arguments. So instead of
```rs
for i in 0..100 {
df(x[i], y[i], 1234);
}
```
You can now do
```rs
for i in 0..100.step_by(4) {
df(x[i+0],x[i+1],x[i+2],x[i+3], y[i+0], y[i+1], y[i+2], y[i+3], 1234);
}
```
which will give the same results, but allows better compiler optimizations. See the testcase for details.
There is a second variant, where we can mark certain arguments and instead of having to pass in N shadow arguments, Enzyme assumes that the argument is N times longer. I.e. instead of accepting 4 slices with 12 floats each, we would accept one slice with 48 floats. I'll implement this over the next days.
I will also add more tests for both modes.
For any one preferring some more interactive explanation, here's a video of Tim's llvm dev talk, where he presents his work. https://www.youtube.com/watch?v=edvaLAL5RqU
I'll also add some other docs to the dev guide and user docs in another PR.
r? ghost
Tracking:
- https://github.com/rust-lang/rust/issues/124509
- https://github.com/rust-lang/rust/issues/135283
They are no longer needed.
This does slightly worsen the error message for a single test, but that
test contains code that is so badly broken that I'm not worried about
it.
`ast::Item` has an `ident` field.
- It's always non-empty for these item kinds: `ExternCrate`, `Static`,
`Const`, `Fn`, `Mod`, `TyAlias`, `Enum`, `Struct`, `Union`,
`Trait`, `TraitAlias`, `MacroDef`, `Delegation`.
- It's always empty for these item kinds: `Use`, `ForeignMod`,
`GlobalAsm`, `Impl`, `MacCall`, `DelegationMac`.
There is a similar story for `AssocItemKind` and `ForeignItemKind`.
Some sites that handle items check for an empty ident, some don't. This
is a very C-like way of doing things, but this is Rust, we have sum
types, we can do this properly and never forget to check for the
exceptional case and never YOLO possibly empty identifiers (or possibly
dummy spans) around and hope that things will work out.
The commit is large but it's mostly obvious plumbing work. Some notable
things.
- `ast::Item` got 8 bytes bigger. This could be avoided by boxing the
fields within some of the `ast::ItemKind` variants (specifically:
`Struct`, `Union`, `Enum`). I might do that in a follow-up; this
commit is big enough already.
- For the visitors: `FnKind` no longer needs an `ident` field because
the `Fn` within how has one.
- In the parser, the `ItemInfo` typedef is no longer needed. It was used
in various places to return an `Ident` alongside an `ItemKind`, but
now the `Ident` (if present) is within the `ItemKind`.
- In a few places I renamed identifier variables called `name` (or
`foo_name`) as `ident` (or `foo_ident`), to better match the type, and
because `name` is normally used for `Symbol`s. It's confusing to see
something like `foo_name.name`.
`expand_test_case` looks for any item with a `#[test_case]` attribute
and adds a `test_path_symbol` attribute to it while also fiddling with
the item's ident's span.
This is pretty weird, because `#[test_case]` is only valid on
`fn`/`const`/`static` items, as far as I can tell. But you don't
currently get an error or warning if you use it on other kinds of items.
This commit changes things so that a `#[test_case]` item is modified
only if it is `fn`/`const`/`static`. This is relevant for moving idents
from `Item` to `ItemKind`, because some item kinds don't have an ident,
e.g. `impl` blocks.
The commit also does the following.
- Renames a local variable `test_id` as `test_ident`.
- Changes a `const` to `static` in
`tests/ui/custom_test_frameworks/full.rs` to give the `static` case
some test coverage.
- Adds a `struct` and `impl` to the same test to give some test coverage
to the non-affected item kinds. These have a `FIXME` comment
identifying the weirdness here. Hopefully this will be useful
breadcrumbs for somebody else in the future.
Mostly parser: Eliminate code that's been dead / semi-dead since the removal of type ascription syntax
**Disclaimer**: This PR is intended to mostly clean up code as opposed to bringing about behavioral changes. Therefore it doesn't aim to address any of the 'FIXME: remove after a month [dated: 2023-05-02]: "type ascription syntax has been removed, see issue [#]101728"'.
---
By commit:
1. Removes truly dead code:
* Since 1.71 (#109128) `let _ = { f: x };` is a syntax error as opposed to a semantic error which allows the parse-time diagnostic (suggestion) "*struct literal body without path // you might have forgotten […]*" to kick in.
* The analysis-time diagnostic (suggestion) from <=1.70 "*cannot find value \`f\` in this scope // you might have forgotten […]*" is therefore no longer reachable.
2. Updates `is_certainly_not_a_block` to be in line with the current grammar:
* The seq. `{ ident:` is definitely not the start of a block. Before the removal of ty ascr, `{ ident: ty_start` would begin a block expr.
* This shouldn't make more code compile IINM, it should *ultimately* only affect diagnostics.
* For example, `if T { f: () } {}` will now be interpreted as an `if` with struct lit `T { f: () }` as its *condition* (which is banned in the parser anyway) as opposed to just `T` (with the *consequent* being `f : ()` which is also invalid (since 1.71)). The diagnostics are almost the same because we have two separate parse recovery procedures + diagnostics: `StructLiteralNeedingParens` (*invalid struct lit*) before and `StructLiteralNotAllowedHere` (*struct lits aren't allowed here*) now, as you can see from the diff.
* (As an aside, even before this PR, fn `maybe_suggest_struct_literal` should've just used the much older & clearer `StructLiteralNotAllowedHere`)
* NB: This does sadly regress the compiler output for `tests/ui/parser/type-ascription-in-pattern.rs` but that can be fixed in follow-up PRs. It's not super important IMO and a natural consequence.
3. Removes code that's become dead due to the prior commit.
* Basically reverts #106620 + #112475 (without regressing rustc's output!).
* Now the older & more robust parse recovery procedure (cc `StructLiteralNotAllowedHere`) takes care of the cases the removed code used to handle.
* This automatically fixes the suggestions for \[[playground](https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=7e2030163b11ee96d17adc3325b01780)\]:
* `if Ty::<i32> { f: K }.m() {}`: `if Ty::<i32> { SomeStruct { f: K } }.m() {}` (broken) → ` if (Ty::<i32> { f: K }).m() {}`
* `if <T as Trait>::Out { f: K::<> }.m() {}`: `if <T as Trait>(::Out { f: K::<> }).m() {}` (broken) → `if (<T as Trait>::Out { f: K::<> }).m() {}`
4. Merge and simplify UI tests pertaining to this issue, so it's easier to add more regression tests like for the two cases mentioned above.
5. Merge UI tests and add the two regression tests.
Best reviewed commit by commit (on request I'll partially squash after approval).
Visitors track whether an assoc item is in a trait impl or an inherent impl
`AssocCtxt::Impl` now contains an `of_trait` field. This allows ast lowering and nameres to not have to track whether we're in a trait impl or an inherent impl.
Rollup of 10 pull requests
Successful merges:
- #137715 (Allow int literals for pattern types with int base types)
- #138002 (Disable CFI for weakly linked syscalls)
- #138051 (Add support for downloading GCC from CI)
- #138231 (Prevent ICE in autodiff validation by emitting user-friendly errors)
- #138245 (stabilize `ci_rustc_if_unchanged_logic` test for local environments)
- #138256 (Do not feed anon const a type that references generics that it does not have)
- #138284 (Do not write user type annotation for const param value path)
- #138296 (Remove `AdtFlags::IS_ANONYMOUS` and `Copy`/`Clone` condition for anonymous ADT)
- #138352 (miri native_calls: ensure we actually expose *mutable* provenance to the memory FFI can access)
- #138354 (remove redundant `body` arguments)
r? `@ghost`
`@rustbot` modify labels: rollup
Prevent ICE in autodiff validation by emitting user-friendly errors
This PR moves `valid_ret_activity` and `valid_input_activity` checks to the macro expansion phase in compiler/rustc_builtin_macros/src/autodiff.rs, replacing the following internal compiler error (ICE):
```
error: internal compiler error:
compiler/rustc_codegen_ssa/src/codegen_attrs.rs:935:13:
Invalid input activity Dual for Reverse mode
```
with a more user-friendly message.
The issue specifically affected the test file `tests/ui/autodiff/autodiff_illegal.rs`, impacting the functions `f5` and `f6`.
The ICE can be reproduced by following [Enzyme's Rustbook](https://enzymead.github.io/rustbook/installation.html) installation guide.
Additionally, this PR adds tests for invalid return activity in `autodiff_illegal.rs`, which previously triggered an unnoticed ICE before these fixes.
r? ``@oli-obk``
Add `#[define_opaques]` attribute and require it for all type-alias-impl-trait sites that register a hidden type
Instead of relying on the signature of items to decide whether they are constraining an opaque type, the opaque types that the item constrains must be explicitly listed.
A previous version of this PR used an actual attribute, but had to keep the resolved `DefId`s in a side table.
Now we just lower to fields in the AST that have no surface syntax, instead a builtin attribute macro fills in those fields where applicable.
Note that for convenience referencing opaque types in associated types from associated methods on the same impl will not require an attribute. If that causes problems `#[defines()]` can be used to overwrite the default of searching for opaques in the signature.
One wart of this design is that closures and static items do not have generics. So since I stored the opaques in the generics of functions, consts and methods, I would need to add a custom field to closures and statics to track this information. During a T-types discussion we decided to just not do this for now.
fixes#131298
Revert <https://github.com/rust-lang/rust/pull/138084> to buy time to
consider options that avoids breaking downstream usages of cargo on
distributed `rustc-src` artifacts, where such cargo invocations fail due
to inability to inherit `lints` from workspace root manifest's
`workspace.lints` (this is only valid for the source rust-lang/rust
workspace, but not really the distributed `rustc-src` artifacts).
This breakage was reported in
<https://github.com/rust-lang/rust/issues/138304>.
This reverts commit 48caf81484, reversing
changes made to c6662879b2.
add a "future" edition
This idea has been discussed previously [on Zulip](https://rust-lang.zulipchat.com/#narrow/channel/213817-t-lang/topic/Continuous.20edition-like.20changes.3F/near/432559262) (though what I've implemented isn't exactly the "next"/"future" editions proposed in that message, just the "future" edition). I've found myself prototyping changes that involve edition migrations and wanting to target an upcoming edition for those migrations, but none exists. This should be permanently unstable and not removed.
This involves replacing `nt_pretty_printing_compatibility_hack` with
`stream_pretty_printing_compatibility_hack`.
The handling of statements in `transcribe` is slightly different to
other nonterminal kinds, due to the lack of `from_ast` implementation
for empty statements.
Notable test changes:
- `tests/ui/proc-macro/expand-to-derive.rs`: the diff looks large but
the only difference is the insertion of a single invisible-delimited
group around a metavar.
When `#![feature(min_generic_const_args)]` is enabled, we now lower all
const paths in generic arg position to `hir::ConstArgKind::Path`. We
then lower assoc const paths to `ty::ConstKind::Unevaluated` since we
can no longer use the anon const expression lowering machinery. In the
process of implementing this, I factored out `hir_ty_lowering` code that
is now shared between lowering assoc types and assoc consts.
This PR also introduces a `#[type_const]` attribute for trait assoc
consts that are allowed as const args. However, we still need to
implement code to check that assoc const definitions satisfy
`#[type_const]` if present (basically is it a const path or a
monomorphic anon const).
Inject `compiler_builtins` during postprocessing and ensure it is made private
Follow up of https://github.com/rust-lang/rust/pull/135278
Do the following:
* Inject `compiler_builtins` during postprocessing, rather than injecting `extern crate compiler_builtins as _` into the AST
* Do not make dependencies of `std` private by default (this was added in #135278)
* Make sure sysroot crates correctly mark their dependencies private/public
* Ensure that marking a dependency private makes its dependents private by default as well, unless otherwise specified
* Do the `compiler_builtins` update that has been blocked on this
There is more detail in the commit messages. This includes the changes I was working on in https://github.com/rust-lang/rust/pull/136226.
try-job: test-various
try-job: x86_64-msvc-1
try-job: x86_64-msvc-2
try-job: i686-mingw-1
try-job: i686-mingw-2
`compiler_builtins` is currently injected as `extern crate
compiler_builtins as _`. This has made gating via diagnostics difficult
because it appears in the crate graph as a non-private dependency, and
there isn't an easy way to differentiate between the injected AST and
user-specified `extern crate compiler_builtins`.
Resolve this by injecting `compiler_builtins` during postprocessing
rather than early in the AST. Most of the time this isn't even needed
because it shows up in `std` or `core`'s crate graph, but injection is
still needed to ensure `#![no_core]` works correctly.
A similar change was attempted at [1] but this encountered errors
building `proc_macro` and `rustc-std-workspace-std`. Similar failures
showed up while working on this patch, which were traced back to
`compiler_builtins` showing up in the graph twice (once via dependency
and once via injection). This is resolved by not injecting if a
`#![compiler_builtins]` crate already exists.
[1]: https://github.com/rust-lang/rust/pull/113634
Add a TyPat in the AST to reuse the generic arg lowering logic
This simplifies ast lowering significantly with little cost to the pattern types parser.
Also fixes any problems we've had with generic args (well, pushes any problems onto the `generic_const_exprs` feature gate)
follow-up to https://github.com/rust-lang/rust/pull/136284#discussion_r1939292367
r? ``@BoxyUwU``
Introduce CoercePointeeWellformed for coherence checks at typeck stage
Fix#135206
This is the first PR to introduce the "wellformedness" check for `derive(CoercePointee)`.
This patch introduces a new error code to cover all the prerequisites of the said macro. The checks that is enforced with this patch is whether the data is indeed `struct` and whether the layout is set to `repr(transparent)`.
A following series of patch will arrive later to address the following concern.
1. #135217 so that we would only admit one single coercion on one type parameter, and leave the rest for future consideration in tandem of development of other coercion rules.
1. Enforcement of data field requirements.
**An open question** is whether there is a good schema to encode the `#[pointee]` as well, so that we could also check if the `#[pointee]` type parameter is indeed `?Sized`.
``@rustbot`` label F-derive_coerce_pointee
tree-wide: parallel: Fully removed all `Lrc`, replaced with `Arc`
tree-wide: parallel: Fully removed all `Lrc`, replaced with `Arc`
This is continuation of https://github.com/rust-lang/rust/pull/132282 .
I'm pretty sure I did everything right. In particular, I searched all occurrences of `Lrc` in submodules and made sure that they don't need replacement.
There are other possibilities, through.
We can define `enum Lrc<T> { Rc(Rc<T>), Arc(Arc<T>) }`. Or we can make `Lrc` a union and on every clone we can read from special thread-local variable. Or we can add a generic parameter to `Lrc` and, yes, this parameter will be everywhere across all codebase.
So, if you think we should take some alternative approach, then don't merge this PR. But if it is decided to stick with `Arc`, then, please, merge.
cc "Parallel Rustc Front-end" ( https://github.com/rust-lang/rust/issues/113349 )
r? SparrowLii
`@rustbot` label WG-compiler-parallel
This has now been approved as a language feature and no longer needs
a `rustc_` prefix.
Also change the `contracts` feature to be marked as incomplete and
`contracts_internals` as internal.
Instead of parsing the different components of a function signature,
eagerly look for either the `where` keyword or the function body.
- Also address feedback to use `From` instead of `TryFrom` in cranelift
contract and ubcheck codegen.
The extended syntax for function signature that includes contract clauses
should never be user exposed versus the interface we want to ship
externally eventually.
includes post-developed commit: do not suggest internal-only keywords as corrections to parse failures.
includes post-developed commit: removed tabs that creeped in into rustfmt tool source code.
includes post-developed commit, placating rustfmt self dogfooding.
includes post-developed commit: add backquotes to prevent markdown checking from trying to treat an attr as a markdown hyperlink/
includes post-developed commit: fix lowering to keep contracts from being erroneously inherited by nested bodies (like closures).
Rebase Conflicts:
- compiler/rustc_parse/src/parser/diagnostics.rs
- compiler/rustc_parse/src/parser/item.rs
- compiler/rustc_span/src/hygiene.rs
Remove contracts keywords from diagnostic messages
Provide a new function `listify`, meant to be used in cases similar to `pluralize!`. When you have a slice of arbitrary elements that need to be presented to the user, `listify` allows you to turn that into a list of comma separated strings.
This reduces a lot of redundant logic that happens often in diagnostics.
Point at invalid utf-8 span on user's source code
```
error: couldn't read `$DIR/not-utf8-bin-file.rs`: stream did not contain valid UTF-8
--> $DIR/not-utf8-2.rs:6:5
|
LL | include!("not-utf8-bin-file.rs");
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
note: byte `193` is not valid utf-8
--> $DIR/not-utf8-bin-file.rs:2:14
|
LL | let _ = "�|�␂!5�cc␕␂��";
| ^
= note: this error originates in the macro `include` (in Nightly builds, run with -Z macro-backtrace for more info)
```
When we attempt to load a Rust source code file, if there is a OS file failure we try reading the file as bytes. If that succeeds we try to turn it into UTF-8. If *that* fails, we provide additional context about *where* the file has the first invalid UTF-8 character.
Fix#76869.
```
error: couldn't read `$DIR/not-utf8-bin-file.rs`: stream did not contain valid UTF-8
--> $DIR/not-utf8-2.rs:6:5
|
LL | include!("not-utf8-bin-file.rs");
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
note: `[193]` is not valid utf-8
--> $DIR/not-utf8-bin-file.rs:2:14
|
LL | let _ = "�|�␂!5�cc␕␂��";
| ^
= note: this error originates in the macro `include` (in Nightly builds, run with -Z macro-backtrace for more info)
```
When we attempt to load a Rust source code file, if there is a OS file failure we try reading the file as bytes. If that succeeds we try to turn it into UTF-8. If *that* fails, we provide additional context about *where* the file has the first invalid UTF-8 character.
Fix#76869.
Use orphaned error code for the same error it belonged to before.
```
error[E0665]: `#[derive(Default)]` on enum with no `#[default]`
--> $DIR/macros-nonfatal-errors.rs:42:10
|
LL | #[derive(Default)]
| ^^^^^^^
LL | / enum NoDeclaredDefault {
LL | | Foo,
LL | | Bar,
LL | | }
| |_- this enum needs a unit variant marked with `#[default]`
|
= note: this error originates in the derive macro `Default` (in Nightly builds, run with -Z macro-backtrace for more info)
help: make this unit variant default by placing `#[default]` on it
|
LL | #[default] Foo,
| ~~~~~~~~~~~~~~
help: make this unit variant default by placing `#[default]` on it
|
LL | #[default] Bar,
| ~~~~~~~~~~~~~~
```
