offset_from: always allow pointers to point to the same address
This PR implements the last remaining part of the t-opsem consensus in https://github.com/rust-lang/unsafe-code-guidelines/issues/472: always permits offset_from when both pointers have the same address, no matter how they are computed. This is required to achieve *provenance monotonicity*.
Tracking issue: https://github.com/rust-lang/rust/issues/117945
### What is provenance monotonicity and why does it matter?
Provenance monotonicity is the property that adding arbitrary provenance to any no-provenance pointer must never make the program UB. More specifically, in the program state, data in memory is stored as a sequence of [abstract bytes](https://rust-lang.github.io/unsafe-code-guidelines/glossary.html#abstract-byte), where each byte can optionally carry provenance. When a pointer is stored in memory, all of the bytes it is stored in carry that provenance. Provenance monotonicity means: if we take some byte that does not have provenance, and give it some arbitrary provenance, then that cannot change program behavior or introduce UB into a UB-free program.
We care about provenance monotonicity because we want to allow the optimizer to remove provenance-stripping operations. Removing a provenance-stripping operation effectively means the program after the optimization has provenance where the program before the optimization did not -- since the provenance removal does not happen in the optimized program. IOW, the compiler transformation added provenance to previously provenance-free bytes. This is exactly what provenance monotonicity lets us do.
We care about removing provenance-stripping operations because `*ptr = *ptr` is, in general, (likely) a provenance-stripping operation. Specifically, consider `ptr: *mut usize` (or any integer type), and imagine the data at `*ptr` is actually a pointer (i.e., we are type-punning between pointers and integers). Then `*ptr` on the right-hand side evaluates to the data in memory *without* any provenance (because [integers do not have provenance](https://rust-lang.github.io/rfcs/3559-rust-has-provenance.html#integers-do-not-have-provenance)). Storing that back to `*ptr` means that the abstract bytes `ptr` points to are the same as before, except their provenance is now gone. This makes `*ptr = *ptr` a provenance-stripping operation (Here we assume `*ptr` is fully initialized. If it is not initialized, evaluating `*ptr` to a value is UB, so removing `*ptr = *ptr` is trivially correct.)
### What does `offset_from` have to do with provenance monotonicity?
With `ptr = without_provenance(N)`, `ptr.offset_from(ptr)` is always well-defined and returns 0. By provenance monotonicity, I can now add provenance to the two arguments of `offset_from` and it must still be well-defined. Crucially, I can add *different* provenance to the two arguments, and it must still be well-defined. In other words, this must always be allowed: `ptr1.with_addr(N).offset_from(ptr2.with_addr(N))` (and it returns 0). But the current spec for `offset_from` says that the two pointers must either both be derived from an integer or both be derived from the same allocation, which is not in general true for arbitrary `ptr1`, `ptr2`.
To obtain provenance monotonicity, this PR hence changes the spec for offset_from to say that if both pointers have the same address, the function is always well-defined.
### What further consequences does this have?
It means the compiler can no longer transform `end2 = begin.offset(end.offset_from(begin))` into `end2 = end`. However, it can still be transformed into `end2 = begin.with_addr(end.addr())`, which later parts of the backend (when provenance has been erased) can trivially turn into `end2 = end`.
The only alternative I am aware of is a fundamentally different handling of zero-sized accesses, where a "no provenance" pointer is not allowed to do zero-sized accesses and instead we have a special provenance that indicates "may be used for zero-sized accesses (and nothing else)". `offset` and `offset_from` would then always be UB on a "no provenance" pointer, and permit zero-sized offsets on a "zero-sized provenance" pointer. This achieves provenance monotonicity. That is, however, a breaking change as it contradicts what we landed in https://github.com/rust-lang/rust/pull/117329. It's also a whole bunch of extra UB, which doesn't seem worth it just to achieve that transformation.
### What about the backend?
LLVM currently doesn't have an intrinsic for pointer difference, so we anyway cast to integer and subtract there. That's never UB so it is compatible with any relaxation we may want to apply.
If LLVM gets a `ptrsub` in the future, then plausibly it will be consistent with `ptradd` and [consider two equal pointers to be inbounds](https://github.com/rust-lang/rust/pull/124921#issuecomment-2205795829).
Make os/windows and pal/windows default to `#![deny(unsafe_op_in_unsafe_fn)]`
This is to prevent regressions in modules that currently pass. I did also fix up a few trivial places where the module contained only one or two simple wrappers. In more complex cases we should try to ensure the `unsafe` blocks are appropriately scoped and have any appropriate safety comments.
This does not fix the windows bits of #127747 but it should help prevent regressions until that is done and also make it more obvious specifically which modules need attention.
std: `#![deny(unsafe_op_in_unsafe_fn)]` in platform-independent code
This applies the `unsafe_op_in_unsafe_fn` lint in all places in std that _do not have platform-specific cfg in their code_. For all such places, the lint remains allowed, because they need further work to address the relevant concerns. This list includes:
- `std::backtrace_rs` (internal-only)
- `std::sys` (internal-only)
- `std::os`
Notably this eliminates all "unwrapped" unsafe operations in `std::io` and `std::sync`, which will make them much more auditable in the future. Such has *also* been left for future work. While I made a few safety comments along the way on interfaces I have grown sufficiently familiar with, in most cases I had no context, nor particular confidence the unsafety was correct.
In the cases where I was able to determine the unsafety was correct without having prior context, it was obviously redundant. For example, an unsafe function calling another unsafe function that has the exact same contract, forwarding its caller's requirements just as it forwards its actual call.
Windows: Remove some unnecessary type aliases
Back in the olden days, C did not have fixed-width types so these type aliases were at least potentially useful. Nowadays, and especially in Rust, we don't need the aliases and they don't help with anything. Notably the windows bindings we use also don't bother with the aliases. And even when we have used aliases they're often only used once then forgotten about.
The only one that gives me pause is `DWORD` because it's used a fair bit. But it's still used inconsistently and we implicitly assume it's a `u32` anyway (e.g. `as` casting from an `i32`).
std: removes logarithms family function edge cases handling for solaris.
Issue had been fixed over time with solaris, 11.x behaves correctly
(and we support it as minimum), illumos works correctly too.
Rollup of 5 pull requests
Successful merges:
- #127273 (Fix `DebugParser`.)
- #127587 (Report usage of lib features in ast validation)
- #127592 (doc: Suggest `str::repeat` over `iter::repeat().take().collect()`)
- #127630 (Remove lang feature for type ascription (since it's a lib feature now))
- #127711 (Add regression test for a gce + effects ICE)
r? `@ghost`
`@rustbot` modify labels: rollup
Merge Apple `std::os` extensions modules into `std::os::darwin`
The functionality available on Apple platforms are very similar, and were (basically) duplicated for each platform.
This PR rectifies that by merging the code into one module.
Ultimately, I've done this to fix `./x build library --target=aarch64-apple-tvos,aarch64-apple-watchos,aarch64-apple-visionos`, as that currently fails because of dead code warnings.
Publically exposing these to tvOS/watchOS/visionOS targets is considered in https://github.com/rust-lang/rust/pull/123723, but that seems to be dragging out, and in any case I think it makes sense to do the refactor separately from stabilization.
r? libs
Fixes https://github.com/rust-lang/rust/issues/121640 and https://github.com/rust-lang/rust/issues/124825.
The functionality available on Apple platforms are very similar, and
were duplicated for each platform.
Additionally, this fixes a warning when compiling the standard library
for tvOS, watchOS and visionOS by marking the corresponding code as
dead code.
Use ManuallyDrop in BufWriter::into_parts
The fact that `mem::forget` takes by value means that it interacts very poorly with Stacked Borrows; generally users think of calling it as a no-op, but in Stacked Borrows, the field retagging tends to cause surprise tag invalidation.
Remove memory leaks in doctests in `core`, `alloc`, and `std`
cc `@RalfJung` https://github.com/rust-lang/rust/issues/126067https://github.com/rust-lang/miri/issues/3670
Should be no actual *documentation* changes[^1], all added/modified lines in the doctests are hidden with `#`,
This PR splits the existing memory leaks in doctests in `core`, `alloc`, and `std` into two general categories:
1. "Non-focused" memory leaks that are incidental to the thing being documented, and/or are easy to remove, i.e. they are only there because preventing the leak would make the doctest less clear and/or concise.
- These doctests simply have a comment like `# // Prevent leaks for Miri.` above the added line that removes the memory leak.
- [^2]Some of these would perhaps be better as part of the public documentation part of the doctest, to clarify that a memory leak can happen if it is not otherwise mentioned explicitly in the documentation (specifically the ones in `(A)Rc::increment_strong_count(_in)`).
2. "Focused" memory leaks that are intentional and documented, and/or are possibly fragile to remove.
- These doctests have a `# // FIXME` comment above the line that removes the memory leak, with a note that once `-Zmiri-disable-leak-check` can be applied at test granularity, these tests should be "un-unleakified" and have `-Zmiri-disable-leak-check` enabled.
- Some of these are possibly fragile (e.g. unleaking the result of `Vec::leak`) and thus should definitely not be made part of the documentation.
This should be all of the leaks currently in `core` and `alloc`. I only found one leak in `std`, and it was in the first category (excluding the modules `@RalfJung` mentioned in https://github.com/rust-lang/rust/issues/126067 , and reducing the number of iterations of [one test](https://github.com/rust-lang/rust/blob/master/library/std/src/sync/once_lock.rs#L49-L94) from 1000 to 10)
[^1]: assuming [^2] is not added
[^2]: backlink
Windows: Add experimental support for linking std-required system DLLs using raw-dylib
For Windows, this allows std to define system imports without needing the user to have import libraries. It's intended for this to become the default.
For now it's an experimental feature so it can be tested using build-std.
Stabilize const unchecked conversion from u32 to char
Closes https://github.com/rust-lang/rust/issues/89259.
The functions in this PR were left out of the initial set of `feature(const_char_convert)` stabilizations in https://github.com/rust-lang/rust/pull/102470, but have since been unblocked by https://github.com/rust-lang/rust/pull/118979.
If `unsafe { from_u32_unchecked(u) }` is called in const with a value for which `from_u32(u)` returns None, we get the following compile error.
```rust
fn main() {
let _ = const { unsafe { char::from_u32_unchecked(0xd800) } };
}
```
```console
error[E0080]: it is undefined behavior to use this value
--> src/main.rs:2:19
|
2 | let _ = const { unsafe { char::from_u32_unchecked(0xd800) } };
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ constructing invalid value: encountered 0x0000d800, but expected a valid unicode scalar value (in `0..=0x10FFFF` but not in `0xD800..=0xDFFF`)
|
= note: The rules on what exactly is undefined behavior aren't clear, so this check might be overzealous. Please open an issue on the rustc repository if you believe it should not be considered undefined behavior.
= note: the raw bytes of the constant (size: 4, align: 4) {
00 d8 00 00 │ ....
}
note: erroneous constant encountered
--> src/main.rs:2:13
|
2 | let _ = const { unsafe { char::from_u32_unchecked(0xd800) } };
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
```
Improved slice documentation
Improve slice documentation to include assert_eq checks for all the cases where there were existing examples. I think it makes things more clear when the documentation explicitly checks against values and shows the reader what it does.
I also started a rust internals discussion about it here: https://internals.rust-lang.org/t/improve-slice-documentaion/21168
