UTF-8 validation: Compute block end upfront
Simplify the conditional used for ensuring that the whole word loop is
only used if there are at least two whole words left to read.
This makes the function slightly smaller and simpler, a 0-5% reduction
in runtime for various test cases.
Use more specific panic message for &str slicing errors
Separate out of bounds errors from character boundary errors, and print
more details for character boundary errors.
It reports the first error it finds in:
1. begin out of bounds
2. end out of bounds
3. begin <= end violated
3. begin not char boundary
5. end not char boundary.
Example:
&"abcαβγ"[..4]
thread 'str::test_slice_fail_boundary_1' panicked at 'byte index 4 is not
a char boundary; it is inside 'α' (bytes 3..5) of `abcαβγ`'
Fixes#38052
Separate out of bounds errors from character boundary errors, and print
more details for character boundary errors.
Example:
&"abcαβγ"[..4]
thread 'str::test_slice_fail_boundary_1' panicked at 'byte index 4 is not
a char boundary; it is inside `α` (bytes 3..5) of `abcαβγ`'
Simplify the conditional used for ensuring that the whole word loop is
only used if there are at least two whole words left to read.
This makes the function slightly smaller and simpler, a 0-5% reduction
in runtime for various test cases.
Improve .chars().count()
Use a simpler loop to count the `char` of a string: count the
number of non-continuation bytes. Use `count += <conditional>` which the
compiler understands well and can apply loop optimizations to.
benchmark descriptions and results for two configurations:
- ascii: ascii text
- cy: cyrillic text
- jp: japanese text
- words ascii: counting each split_whitespace item from the ascii text
- words jp: counting each split_whitespace item from the jp text
```
x86-64 rustc -Copt-level=3
name orig_ ns/iter cmov_ ns/iter diff ns/iter diff %
count_ascii 1,453 (1755 MB/s) 1,398 (1824 MB/s) -55 -3.79%
count_cy 5,990 (856 MB/s) 2,545 (2016 MB/s) -3,445 -57.51%
count_jp 3,075 (1169 MB/s) 1,772 (2029 MB/s) -1,303 -42.37%
count_words_ascii 4,157 (521 MB/s) 1,797 (1205 MB/s) -2,360 -56.77%
count_words_jp 3,337 (1071 MB/s) 1,772 (2018 MB/s) -1,565 -46.90%
x86-64 rustc -Ctarget-feature=+avx -Copt-level=3
name orig_ ns/iter cmov_ ns/iter diff ns/iter diff %
count_ascii 1,444 (1766 MB/s) 763 (3343 MB/s) -681 -47.16%
count_cy 5,871 (874 MB/s) 1,527 (3360 MB/s) -4,344 -73.99%
count_jp 2,874 (1251 MB/s) 1,073 (3351 MB/s) -1,801 -62.67%
count_words_ascii 4,131 (524 MB/s) 1,871 (1157 MB/s) -2,260 -54.71%
count_words_jp 3,253 (1099 MB/s) 1,331 (2686 MB/s) -1,922 -59.08%
```
I briefly explored a more involved blocked algorithm (looking at 8 or more bytes at a time),
but the code in this PR was always winning `count_words_ascii` in particular (counting
many small strings); this solution is an improvement without tradeoffs.
Use a simpler loop to count the `char` of a string: count the
number of non-continuation bytes. Use `count += <conditional>` which the
compiler understands well and can apply loop optimizations to.
Use `len` instead of `size_hint` where appropiate
This makes it clearer that we're not just looking for a lower bound but
rather know that the iterator is an `ExactSizeIterator`.
* Remove the deprecated `CharRange` type which was forgotten to be removed
awhile back.
* Stabilize the `os::$platform::raw::pthread_t` type which was intended to be
stabilized as part of #32804
std: Stabilize APIs for the 1.9 release
This commit applies all stabilizations, renamings, and deprecations that the
library team has decided on for the upcoming 1.9 release. All tracking issues
have gone through a cycle-long "final comment period" and the specific APIs
stabilized/deprecated are:
Stable
* `std::panic`
* `std::panic::catch_unwind` (renamed from `recover`)
* `std::panic::resume_unwind` (renamed from `propagate`)
* `std::panic::AssertUnwindSafe` (renamed from `AssertRecoverSafe`)
* `std::panic::UnwindSafe` (renamed from `RecoverSafe`)
* `str::is_char_boundary`
* `<*const T>::as_ref`
* `<*mut T>::as_ref`
* `<*mut T>::as_mut`
* `AsciiExt::make_ascii_uppercase`
* `AsciiExt::make_ascii_lowercase`
* `char::decode_utf16`
* `char::DecodeUtf16`
* `char::DecodeUtf16Error`
* `char::DecodeUtf16Error::unpaired_surrogate`
* `BTreeSet::take`
* `BTreeSet::replace`
* `BTreeSet::get`
* `HashSet::take`
* `HashSet::replace`
* `HashSet::get`
* `OsString::with_capacity`
* `OsString::clear`
* `OsString::capacity`
* `OsString::reserve`
* `OsString::reserve_exact`
* `OsStr::is_empty`
* `OsStr::len`
* `std::os::unix::thread`
* `RawPthread`
* `JoinHandleExt`
* `JoinHandleExt::as_pthread_t`
* `JoinHandleExt::into_pthread_t`
* `HashSet::hasher`
* `HashMap::hasher`
* `CommandExt::exec`
* `File::try_clone`
* `SocketAddr::set_ip`
* `SocketAddr::set_port`
* `SocketAddrV4::set_ip`
* `SocketAddrV4::set_port`
* `SocketAddrV6::set_ip`
* `SocketAddrV6::set_port`
* `SocketAddrV6::set_flowinfo`
* `SocketAddrV6::set_scope_id`
* `<[T]>::copy_from_slice`
* `ptr::read_volatile`
* `ptr::write_volatile`
* The `#[deprecated]` attribute
* `OpenOptions::create_new`
Deprecated
* `std::raw::Slice` - use raw parts of `slice` module instead
* `std::raw::Repr` - use raw parts of `slice` module instead
* `str::char_range_at` - use slicing plus `chars()` plus `len_utf8`
* `str::char_range_at_reverse` - use slicing plus `chars().rev()` plus `len_utf8`
* `str::char_at` - use slicing plus `chars()`
* `str::char_at_reverse` - use slicing plus `chars().rev()`
* `str::slice_shift_char` - use `chars()` plus `Chars::as_str`
* `CommandExt::session_leader` - use `before_exec` instead.
Closes#27719
cc #27751 (deprecating the `Slice` bits)
Closes#27754Closes#27780Closes#27809Closes#27811Closes#27830Closes#28050Closes#29453Closes#29791Closes#29935Closes#30014Closes#30752Closes#31262
cc #31398 (still need to deal with `before_exec`)
Closes#31405Closes#31572Closes#31755Closes#31756
This commit applies all stabilizations, renamings, and deprecations that the
library team has decided on for the upcoming 1.9 release. All tracking issues
have gone through a cycle-long "final comment period" and the specific APIs
stabilized/deprecated are:
Stable
* `std::panic`
* `std::panic::catch_unwind` (renamed from `recover`)
* `std::panic::resume_unwind` (renamed from `propagate`)
* `std::panic::AssertUnwindSafe` (renamed from `AssertRecoverSafe`)
* `std::panic::UnwindSafe` (renamed from `RecoverSafe`)
* `str::is_char_boundary`
* `<*const T>::as_ref`
* `<*mut T>::as_ref`
* `<*mut T>::as_mut`
* `AsciiExt::make_ascii_uppercase`
* `AsciiExt::make_ascii_lowercase`
* `char::decode_utf16`
* `char::DecodeUtf16`
* `char::DecodeUtf16Error`
* `char::DecodeUtf16Error::unpaired_surrogate`
* `BTreeSet::take`
* `BTreeSet::replace`
* `BTreeSet::get`
* `HashSet::take`
* `HashSet::replace`
* `HashSet::get`
* `OsString::with_capacity`
* `OsString::clear`
* `OsString::capacity`
* `OsString::reserve`
* `OsString::reserve_exact`
* `OsStr::is_empty`
* `OsStr::len`
* `std::os::unix::thread`
* `RawPthread`
* `JoinHandleExt`
* `JoinHandleExt::as_pthread_t`
* `JoinHandleExt::into_pthread_t`
* `HashSet::hasher`
* `HashMap::hasher`
* `CommandExt::exec`
* `File::try_clone`
* `SocketAddr::set_ip`
* `SocketAddr::set_port`
* `SocketAddrV4::set_ip`
* `SocketAddrV4::set_port`
* `SocketAddrV6::set_ip`
* `SocketAddrV6::set_port`
* `SocketAddrV6::set_flowinfo`
* `SocketAddrV6::set_scope_id`
* `<[T]>::copy_from_slice`
* `ptr::read_volatile`
* `ptr::write_volatile`
* The `#[deprecated]` attribute
* `OpenOptions::create_new`
Deprecated
* `std::raw::Slice` - use raw parts of `slice` module instead
* `std::raw::Repr` - use raw parts of `slice` module instead
* `str::char_range_at` - use slicing plus `chars()` plus `len_utf8`
* `str::char_range_at_reverse` - use slicing plus `chars().rev()` plus `len_utf8`
* `str::char_at` - use slicing plus `chars()`
* `str::char_at_reverse` - use slicing plus `chars().rev()`
* `str::slice_shift_char` - use `chars()` plus `Chars::as_str`
* `CommandExt::session_leader` - use `before_exec` instead.
Closes#27719
cc #27751 (deprecating the `Slice` bits)
Closes#27754Closes#27780Closes#27809Closes#27811Closes#27830Closes#28050Closes#29453Closes#29791Closes#29935Closes#30014Closes#30752Closes#31262
cc #31398 (still need to deal with `before_exec`)
Closes#31405Closes#31572Closes#31755Closes#31756
Where T is a type that can be compared for equality bytewise, we can use
memcmp. We can also use memcmp for PartialOrd, Ord for [u8] and by
extension &str.
This is an improvement for example for the comparison [u8] == [u8] that
used to emit a loop that compared the slices byte by byte.
One worry here could be that this introduces function calls to memcmp
in contexts where it should really inline the comparison or even
optimize it out, but llvm takes care of recognizing memcmp specifically.
Hardcode accepting 0 as a valid str char boundary
If we check explicitly for index == 0, that removes the need to read the
byte at index 0, so it avoids a trip to the string's memory, and it
optimizes out the slicing index' bounds check whenever it is (a constant) zero.
Index 0 must be a valid char boundary (invariant of str that it contains
valid UTF-8 data).
If we check explicitly for index == 0, that removes the need to read the
byte at index 0, so it avoids a trip to the string's memory, and it
optimizes out the slicing index' bounds check whenever it is zero.
With this change, the following examples all change from having a read of
the byte at 0 and a branch to possibly panicing, to having the bounds
checking optimized away.
```rust
pub fn split(s: &str) -> (&str, &str) {
s.split_at(0)
}
pub fn both(s: &str) -> &str {
&s[0..s.len()]
}
pub fn first(s: &str) -> &str {
&s[..0]
}
pub fn last(s: &str) -> &str {
&s[0..]
}
```
Automated conversion using the untry tool [1] and the following command:
```
$ find -name '*.rs' -type f | xargs untry
```
at the root of the Rust repo.
[1]: https://github.com/japaric/untry
Add a link validator to rustbuild
This commit was originally targeted at just adding a link checking script to the rustbuild system. This ended up snowballing a bit to extend rustbuild to be amenable to various tools we have as part of the build system in general.
There's a new `src/tools` directory which has a number of scripts/programs that are purely intended to be used as part of the build system and CI of this repository. This is currently inhabited by rustbook, the error index generator, and a new linkchecker script added as part of this PR. I suspect that more tools like compiletest, tidy scripts, snapshot scripts, etc will migrate their way into this directory over time.
The commit which adds the error index generator shows the steps necessary to add new tools to the build system, namely:
1. New steps are defined for building the tool and running the tool
2. The dependencies are configured
3. The steps are implemented
In terms of the link checker, these commits do a few things:
* A new `src/tools/linkchecker` script is added. This will read an entire documentation tree looking for broken relative links (HTTP links aren't followed yet).
* A large number of broken links throughout the documentation were fixed. Many of these were just broken when viewed from core as opposed to std, but were easily fixed.
* A few rustdoc bugs here and there were fixed
A few categories:
* Links into compiler docs were just all removed as we're not generating
compiler docs.
* Move up one more level to forcibly go to std docs to fix inlined documentation
across the facade crates.
Clarify documentation for string slicing (Index impls)
Clarify documentation for string slicing (Index impls)
- Mention the sugared syntax for the implementations, since it's not
apparent from the docs that `Index<Range<usize>>` corresponds to
`&self[a..b]`.
- Be specific in that start <= end and end <= len
This is just one fix in response to #32057
- Mention the sugared syntax for the implementations, since it's not
apparent from the docs that `Index<Range<usize>>` corresponds to
`&self[a..b]`.
- Be specific in that start <= end and end <= len
Fix panic on string slicing error to truncate the string
The string may be arbitrarily long, but we want to limit the panic
message to a reasonable length. Truncate the string if it is too long
(simply to char boundary).
Also add details to the start <= end message. I think it's ok to flesh
out the code here, since it's in a cold function.
Fixes#32063
This PR implements [RFC 1192](https://github.com/rust-lang/rfcs/blob/master/text/1192-inclusive-ranges.md), which is triple-dot syntax for inclusive range expressions. The new stuff is behind two feature gates (one for the syntax and one for the std::ops types). This replaces the deprecated functionality in std::iter. Along the way I simplified the desugaring for all ranges.
This is my first contribution to rust which changes more than one character outside of a test or comment, so please review carefully! Some of the individual commit messages have more of my notes. Also thanks for putting up with my dumb questions in #rust-internals.
- For implementing `std::ops::RangeInclusive`, I took @Stebalien's suggestion from https://github.com/rust-lang/rfcs/pull/1192#issuecomment-137864421. It seemed to me to make the implementation easier and increase type safety. If that stands, the RFC should be amended to avoid confusion.
- I also kind of like @glaebhoerl's [idea](https://github.com/rust-lang/rfcs/pull/1254#issuecomment-147815299), which is unified inclusive/exclusive range syntax something like `x>..=y`. We can experiment with this while everything is behind a feature gate.
- There are a couple of FIXMEs left (see the last commit). I didn't know what to do about `RangeArgument` and I haven't added `Index` impls yet. Those should be discussed/finished before merging.
cc @Gankro since you [complained](https://www.reddit.com/r/rust/comments/3xkfro/what_happened_to_inclusive_ranges/cy5j0yq)
cc #27777#30877rust-lang/rust#1192rust-lang/rfcs#1254
relevant to #28237 (tracking issue)
The string may be arbitrarily long, but we want to limit the panic
message to a reasonable length. Truncate the string if it is too long
(simply to char boundary).
Also add details to the start <= end message. I think it's ok to flesh
out the code here, since it's in a cold function.
