Optimized vec::IntoIter::next_chunk impl
```
x86_64v1, default
test vec::bench_next_chunk ... bench: 696 ns/iter (+/- 22)
x86_64v1, pr
test vec::bench_next_chunk ... bench: 309 ns/iter (+/- 4)
znver2, default
test vec::bench_next_chunk ... bench: 17,272 ns/iter (+/- 117)
znver2, pr
test vec::bench_next_chunk ... bench: 211 ns/iter (+/- 3)
```
On znver2 the default impl seems to be slow due to different inlining decisions. It goes through `core::array::iter_next_chunk`
which has a deep call tree.
codegen: use new {re,de,}allocator annotations in llvm
This obviates the patch that teaches LLVM internals about
_rust_{re,de}alloc functions by putting annotations directly in the IR
for the optimizer.
The sole test change is required to anchor FileCheck to the body of the
`box_uninitialized` method, so it doesn't see the `allocalign` on
`__rust_alloc` and get mad about the string `alloca` showing up. Since I
was there anyway, I added some checks on the attributes to prove the
right attributes got set.
r? `@nikic`
```
test vec::bench_next_chunk ... bench: 696 ns/iter (+/- 22)
x86_64v1, pr
test vec::bench_next_chunk ... bench: 309 ns/iter (+/- 4)
znver2, default
test vec::bench_next_chunk ... bench: 17,272 ns/iter (+/- 117)
znver2, pr
test vec::bench_next_chunk ... bench: 211 ns/iter (+/- 3)
```
The znver2 default impl seems to be slow due to inlining decisions. It goes through `core::array::iter_next_chunk`
which has a deeper call tree.
This obviates the patch that teaches LLVM internals about
_rust_{re,de}alloc functions by putting annotations directly in the IR
for the optimizer.
The sole test change is required to anchor FileCheck to the body of the
`box_uninitialized` method, so it doesn't see the `allocalign` on
`__rust_alloc` and get mad about the string `alloca` showing up. Since I
was there anyway, I added some checks on the attributes to prove the
right attributes got set.
While we're here, we also emit allocator attributes on
__rust_alloc_zeroed. This should allow LLVM to perform more
optimizations for zeroed blocks, and probably fixes#90032. [This
comment](https://github.com/rust-lang/rust/issues/24194#issuecomment-308791157)
mentions "weird UB-like behaviour with bitvec iterators in
rustc_data_structures" so we may need to back this change out if things
go wrong.
The new test cases require LLVM 15, so we copy them into LLVM
14-supporting versions, which we can delete when we drop LLVM 14.
correct the output of a `capacity` method example
The output of this example in std::alloc is different from which shown in the comment. I have tested it on both Linux and Windows.
* Implement IsZero trait for tuples up to 8 IsZero elements;
* Implement IsZero for u8/i8, leading to implementation of it for arrays of them too;
* Add more codegen tests for this optimization.
* Lower size of array for IsZero trait because it fails to inline checks
add missing null ptr check in alloc example
`alloc` can return null on OOM, if I understood correctly. So we should never just deref a pointer we get from `alloc`.
Borrow Vec<T, A> as [T]
Hello all,
When `Vec` was parametrized with `A`, the `Borrow` impls were omitted and currently `Vec<T, A>` can't be borrowed as `[T]`. This PR fixes that.
This was probably missed, because the `Borrow` impls are in a different file - `src/alloc/slice.rs`.
We briefly discussed this here: https://github.com/rust-lang/wg-allocators/issues/96 and I was told to go ahead and make a PR :)
I tested this by building the toolchain and building my code that needed the `Borrow` impl against it, but let me know if I should add any tests to this PR.
Stabilize `core::ffi::CStr`, `alloc::ffi::CString`, and friends
Stabilize the `core_c_str` and `alloc_c_string` feature gates.
Change `std::ffi` to re-export these types rather than creating type
aliases, since they now have matching stability.
Stabilize the `core_c_str` and `alloc_c_string` feature gates.
Change `std::ffi` to re-export these types rather than creating type
aliases, since they now have matching stability.
Stabilize `core::ffi:c_*` and rexport in `std::ffi`
This only stabilizes the base types, not the non-zero variants, since
those have their own separate tracking issue and have not gone through
FCP to stabilize.
This only stabilizes the base types, not the non-zero variants, since
those have their own separate tracking issue and have not gone through
FCP to stabilize.
Enforce that layout size fits in isize in Layout
As it turns out, enforcing this _in APIs that already enforce `usize` overflow_ is fairly trivial. `Layout::from_size_align_unchecked` continues to "allow" sizes which (when rounded up) would overflow `isize`, but these are now declared as library UB for `Layout`, meaning that consumers of `Layout` no longer have to check this before making an allocation.
(Note that this is "immediate library UB;" IOW it is valid for a future release to make this immediate "language UB," and there is an extant patch to do so, to allow Miri to catch this misuse.)
See also #95252, [Zulip discussion](https://rust-lang.zulipchat.com/#narrow/stream/219381-t-libs/topic/Layout.20Isn't.20Enforcing.20The.20isize.3A.3AMAX.20Rule).
Fixes https://github.com/rust-lang/rust/issues/95334
Some relevant quotes:
`@eddyb,` https://github.com/rust-lang/rust/pull/95252#issuecomment-1078513769
> [B]ecause of the non-trivial presence of both of these among code published on e.g. crates.io:
>
> 1. **`Layout` "producers" / `GlobalAlloc` "users"**: smart pointers (including `alloc::rc` copies with small tweaks), collections, etc.
> 2. **`Layout` "consumers" / `GlobalAlloc` "providers"**: perhaps fewer of these, but anything built on top of OS APIs like `mmap` will expose `> isize::MAX` allocations (on 32-bit hosts) if they lack extra checks
>
> IMO the only responsible option is to enforce the `isize::MAX` limit in `Layout`, which:
>
> * makes `Layout` _sound_ in terms of only ever allowing allocations where `(alloc_base_ptr: *mut u8).offset(size)` is never UB
> * frees both "producers" and "consumers" of `Layout` from manually reimplementing the checks
> * manual checks can be risky, e.g. if the final size passed to the allocator isn't the one being checked
> * this applies retroactively, fixing the overall soundness of existing code with zero transition period or _any_ changes required from users (as long as going through `Layout` is mandatory, making a "choke point")
>
>
> Feel free to quote this comment onto any relevant issue, I might not be able to keep track of developments.
`@Gankra,` https://github.com/rust-lang/rust/pull/95252#issuecomment-1078556371
> As someone who spent way too much time optimizing libcollections checks for this stuff and tried to splatter docs about it everywhere on the belief that it was a reasonable thing for people to manually take care of: I concede the point, it is not reasonable. I am wholy spiritually defeated by the fact that _liballoc_ of all places is getting this stuff wrong. This isn't throwing shade at the folks who implemented these Rc features, but rather a statement of how impractical it is to expect anyone out in the wider ecosystem to enforce them if _some of the most audited rust code in the library that defines the very notion of allocating memory_ can't even reliably do it.
>
> We need the nuclear option of Layout enforcing this rule. Code that breaks this rule is _deeply_ broken and any "regressions" from changing Layout's contract is a _correctness_ fix. Anyone who disagrees and is sufficiently motivated can go around our backs but the standard library should 100% refuse to enable them.
cc also `@RalfJung` `@rust-lang/wg-allocators.` Even though this technically supersedes #95252, those potential failure points should almost certainly still get nicer panics than just "unwrap failed" (which they would get by this PR).
It might additionally be worth recommending to users of the `Layout` API that they should ideally use `.and_then`/`?` to complete the entire layout calculation, and then `panic!` from a single location at the end of `Layout` manipulation, to reduce the overhead of the checks and optimizations preserving the exact location of each `panic` which are conceptually just one failure: allocation too big.
Probably deserves a T-lang and/or T-libs-api FCP (this technically solidifies the [objects must be no larger than `isize::MAX`](https://rust-lang.github.io/unsafe-code-guidelines/layout/scalars.html#isize-and-usize) rule further, and the UCG document says this hasn't been RFCd) and a crater run. Ideally, no code exists that will start failing with this addition; if it does, it was _likely_ (but not certainly) causing UB.
Changes the raw_vec allocation path, thus deserves a perf run as well.
I suggest hiding whitespace-only changes in the diff view.
Optimize `Vec::insert` for the case where `index == len`.
By skipping the call to `copy` with a zero length. This makes it closer
to `push`.
I did this recently for `SmallVec`
(https://github.com/servo/rust-smallvec/pull/282) and it was a big perf win in
one case. Although I don't have a specific use case in mind, it seems
worth doing it for `Vec` as well.
Things to note:
- In the `index < len` case, the number of conditions checked is
unchanged.
- In the `index == len` case, the number of conditions checked increases
by one, but the more expensive zero-length copy is avoided.
- In the `index > len` case the code now reserves space for the extra
element before panicking. This seems like an unimportant change.
r? `@cuviper`
Make `ThinBox<T>` covariant in `T`
Just like `Box<T>`, we want `ThinBox<T>` to be covariant in `T`, but the
projection in `WithHeader<<T as Pointee>::Metadata>` was making it
invariant. This is now hidden as `WithOpaqueHeader`, which we type-cast
whenever the real `WithHeader<H>` type is needed.
Fixes the problem noted in <https://github.com/rust-lang/rust/issues/92791#issuecomment-1104636249>.
By skipping the call to `copy` with a zero length. This makes it closer
to `push`.
I did this recently for `SmallVec`
(https://github.com/servo/rust-smallvec/pull/282) and it was a big perf win in
one case. Although I don't have a specific use case in mind, it seems
worth doing it for `Vec` as well.
Things to note:
- In the `index < len` case, the number of conditions checked is
unchanged.
- In the `index == len` case, the number of conditions checked increases
by one, but the more expensive zero-length copy is avoided.
- In the `index > len` case the code now reserves space for the extra
element before panicking. This seems like an unimportant change.
Rust 1.62.0 introduced a couple new `unused_imports` warnings
in `no_global_oom_handling` builds, making a total of 5 warnings:
```txt
warning: unused import: `Unsize`
--> library/alloc/src/boxed/thin.rs:6:33
|
6 | use core::marker::{PhantomData, Unsize};
| ^^^^^^
|
= note: `#[warn(unused_imports)]` on by default
warning: unused import: `from_fn`
--> library/alloc/src/string.rs:51:18
|
51 | use core::iter::{from_fn, FusedIterator};
| ^^^^^^^
warning: unused import: `core::ops::Deref`
--> library/alloc/src/vec/into_iter.rs:12:5
|
12 | use core::ops::Deref;
| ^^^^^^^^^^^^^^^^
warning: associated function `shrink` is never used
--> library/alloc/src/raw_vec.rs:424:8
|
424 | fn shrink(&mut self, cap: usize) -> Result<(), TryReserveError> {
| ^^^^^^
|
= note: `#[warn(dead_code)]` on by default
warning: associated function `forget_remaining_elements` is never used
--> library/alloc/src/vec/into_iter.rs:126:19
|
126 | pub(crate) fn forget_remaining_elements(&mut self) {
| ^^^^^^^^^^^^^^^^^^^^^^^^^
```
This patch cleans them so that projects compiling `alloc` without
infallible allocations do not see the warnings. It also enables
the use of `-Dwarnings`.
The couple `dead_code` ones may be reverted when some fallible
allocation support starts using them.
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
Implement `Send` and `Sync` for `ThinBox<T>`
Just like `Box<T>`, `ThinBox<T>` owns its data on the heap, so it should
implement `Send` and `Sync` when `T` does.
This extends tracking issue #92791.
Just like `Box<T>`, we want `ThinBox<T>` to be covariant in `T`, but the
projection in `WithHeader<<T as Pointee>::Metadata>` was making it
invariant. This is now hidden as `WithOpaqueHeader`, which we type-cast
whenever the real `WithHeader<H>` type is needed.
Fix `panic` message for `BTreeSet`'s `range` API and document `panic` cases
Currently, the `panic` cases for [`BTreeSet`'s `range` API](https://doc.rust-lang.org/std/collections/struct.BTreeSet.html#method.range) are undocumented and produce a slightly wrong `panic` message (says `BTreeMap` instead of `BTreeSet`).
Panic case 1 code:
```rust
use std::collections::BTreeSet;
use std::ops::Bound::Excluded;
fn main() {
let mut set = BTreeSet::new();
set.insert(3);
set.insert(5);
set.insert(8);
for &elem in set.range((Excluded(&3), Excluded(&3))) {
println!("{elem}");
}
}
```
Panic case 1 message:
```
thread 'main' panicked at 'range start and end are equal and excluded in BTreeMap', /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/alloc/src/collections/btree/search.rs:105:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
Panic case 2 code:
```rust
use std::collections::BTreeSet;
use std::ops::Bound::Included;
fn main() {
let mut set = BTreeSet::new();
set.insert(3);
set.insert(5);
set.insert(8);
for &elem in set.range((Included(&8), Included(&3))) {
println!("{elem}");
}
}
```
Panic case 2:
```
thread 'main' panicked at 'range start is greater than range end in BTreeMap', /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/alloc/src/collections/btree/search.rs:110:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
This PR fixes the output messages to say `BTreeSet`, adds the relevant unit tests, and updates the documentation for the API.
clarify Arc::clone overflow check comment
I had to read this twice to realize that this is explaining that the code is technically unsound, so move that into a dedicated paragraph and make the wording a bit more explicit.
Fix documentation for `with_capacity` and `reserve` families of methods
Fixes#95614
Documentation for the following methods
- `with_capacity`
- `with_capacity_in`
- `with_capacity_and_hasher`
- `reserve`
- `reserve_exact`
- `try_reserve`
- `try_reserve_exact`
was inconsistent and often not entirely correct where they existed on the following types
- `Vec`
- `VecDeque`
- `String`
- `OsString`
- `PathBuf`
- `BinaryHeap`
- `HashSet`
- `HashMap`
- `BufWriter`
- `LineWriter`
since the allocator is allowed to allocate more than the requested capacity in all such cases, and will frequently "allocate" much more in the case of zero-sized types (I also checked `BufReader`, but there the docs appear to be accurate as it appears to actually allocate the exact capacity).
Some effort was made to make the documentation more consistent between types as well.
Documentation for the following methods
with_capacity
with_capacity_in
with_capacity_and_hasher
reserve
reserve_exact
try_reserve
try_reserve_exact
was inconsistent and often not entirely correct where they existed on the following types
Vec
VecDeque
String
OsString
PathBuf
BinaryHeap
HashSet
HashMap
BufWriter
LineWriter
since the allocator is allowed to allocate more than the requested capacity in all such cases, and will frequently "allocate" much more in the case of zero-sized types (I also checked BufReader, but there the docs appear to be accurate as it appears to actually allocate the exact capacity).
Some effort was made to make the documentation more consistent between types as well.
Fix with_capacity* methods for Vec
Fix *reserve* methods for Vec
Fix docs for *reserve* methods of VecDeque
Fix docs for String::with_capacity
Fix docs for *reserve* methods of String
Fix docs for OsString::with_capacity
Fix docs for *reserve* methods on OsString
Fix docs for with_capacity* methods on HashSet
Fix docs for *reserve methods of HashSet
Fix docs for with_capacity* methods of HashMap
Fix docs for *reserve methods on HashMap
Fix expect messages about OOM in doctests
Fix docs for BinaryHeap::with_capacity
Fix docs for *reserve* methods of BinaryHeap
Fix typos
Fix docs for with_capacity on BufWriter and LineWriter
Fix consistent use of `hasher` between `HashMap` and `HashSet`
Fix warning in doc test
Add test for capacity of vec with ZST
Fix doc test error
Add VecDeque::extend from TrustedLen specialization
Continuation of #95904
Inspired by how [`VecDeque::copy_slice` works](c08b235a5c/library/alloc/src/collections/vec_deque/mod.rs (L437-L454)).
## Benchmarks
Before
```
test vec_deque::bench_extend_chained_bytes ... bench: 1,026 ns/iter (+/- 17)
test vec_deque::bench_extend_chained_trustedlen ... bench: 1,024 ns/iter (+/- 40)
test vec_deque::bench_extend_trustedlen ... bench: 637 ns/iter (+/- 693)
```
After
```
test vec_deque::bench_extend_chained_bytes ... bench: 828 ns/iter (+/- 24)
test vec_deque::bench_extend_chained_trustedlen ... bench: 25 ns/iter (+/- 1)
test vec_deque::bench_extend_trustedlen ... bench: 21 ns/iter (+/- 0)
```
## Why do it this way
https://rust.godbolt.org/z/15qY1fMYh
The Compiler Explorer example shows how "just" removing the capacity check, like the [`Vec` `TrustedLen` specialization](c08b235a5c/library/alloc/src/vec/spec_extend.rs (L22-L58)) does, wouldn't have been enough for `VecDeque`. `wrap_add` would still have greatly limited what LLVM could do while optimizing.
---
r? `@the8472`
btree: avoid forcing the allocator to be a reference
The previous code forces the actual allocator used to be some `&A`. This generalizes the code to allow any `A: Copy`. If people truly want to use a reference, they can use `&A` themselves.
Fixes https://github.com/rust-lang/rust/issues/98176
