Currently, on macOS, Rust never uses the fast posix_spawn path if a
directory change is requested due to a bug in Apple's libc. However, the
bug is only triggered if the program is a relative path.
This PR makes it so that the fast path continues to work if the program
is an absolute path or a lone filename.
This was an alternative proposed in
https://github.com/rust-lang/rust/pull/80537#issue-776674009, and it
makes a measurable performance difference in some of my code that spawns
thousands of processes.
This makes it possible to instruct libstd to never touch the signal
handler for `SIGPIPE`, which makes programs pipeable by default (e.g.
with `./your-program | head -n 1`) without `ErrorKind::BrokenPipe`
errors.
std::io: migrate ReadBuf to BorrowBuf/BorrowCursor
This PR replaces `ReadBuf` (used by the `Read::read_buf` family of methods) with `BorrowBuf` and `BorrowCursor`.
The general idea is to split `ReadBuf` because its API is large and confusing. `BorrowBuf` represents a borrowed buffer which is mostly read-only and (other than for construction) deals only with filled vs unfilled segments. a `BorrowCursor` is a mostly write-only view of the unfilled part of a `BorrowBuf` which distinguishes between initialized and uninitialized segments. For `Read::read_buf`, the caller would create a `BorrowBuf`, then pass a `BorrowCursor` to `read_buf`.
In addition to the major API split, I've made the following smaller changes:
* Removed some methods entirely from the API (mostly the functionality can be replicated with two calls rather than a single one)
* Unified naming, e.g., by replacing initialized with init and assume_init with set_init
* Added an easy way to get the number of bytes written to a cursor (`written` method)
As well as simplifying the API (IMO), this approach has the following advantages:
* Since we pass the cursor by value, we remove the 'unsoundness footgun' where a malicious `read_buf` could swap out the `ReadBuf`.
* Since `read_buf` cannot write into the filled part of the buffer, we prevent the filled part shrinking or changing which could cause underflow for the caller or unexpected behaviour.
## Outline
```rust
pub struct BorrowBuf<'a>
impl Debug for BorrowBuf<'_>
impl<'a> From<&'a mut [u8]> for BorrowBuf<'a>
impl<'a> From<&'a mut [MaybeUninit<u8>]> for BorrowBuf<'a>
impl<'a> BorrowBuf<'a> {
pub fn capacity(&self) -> usize
pub fn len(&self) -> usize
pub fn init_len(&self) -> usize
pub fn filled(&self) -> &[u8]
pub fn unfilled<'this>(&'this mut self) -> BorrowCursor<'this, 'a>
pub fn clear(&mut self) -> &mut Self
pub unsafe fn set_init(&mut self, n: usize) -> &mut Self
}
pub struct BorrowCursor<'buf, 'data>
impl<'buf, 'data> BorrowCursor<'buf, 'data> {
pub fn clone<'this>(&'this mut self) -> BorrowCursor<'this, 'data>
pub fn capacity(&self) -> usize
pub fn written(&self) -> usize
pub fn init_ref(&self) -> &[u8]
pub fn init_mut(&mut self) -> &mut [u8]
pub fn uninit_mut(&mut self) -> &mut [MaybeUninit<u8>]
pub unsafe fn as_mut(&mut self) -> &mut [MaybeUninit<u8>]
pub unsafe fn advance(&mut self, n: usize) -> &mut Self
pub fn ensure_init(&mut self) -> &mut Self
pub unsafe fn set_init(&mut self, n: usize) -> &mut Self
pub fn append(&mut self, buf: &[u8])
}
```
## TODO
* ~~Migrate non-unix libs and tests~~
* ~~Naming~~
* ~~`BorrowBuf` or `BorrowedBuf` or `SliceBuf`? (We might want an owned equivalent for the async IO traits)~~
* ~~Should we rename the `readbuf` module? We might keep the name indicate it includes both the buf and cursor variations and someday the owned version too. Or we could change it. It is not publicly exposed, so it is not that important~~.
* ~~`read_buf` method: we read into the cursor now, so the `_buf` suffix is a bit weird.~~
* ~~Documentation~~
* Tests are incomplete (I adjusted existing tests, but did not add new ones).
cc https://github.com/rust-lang/rust/issues/78485, https://github.com/rust-lang/rust/issues/94741
supersedes: https://github.com/rust-lang/rust/pull/95770, https://github.com/rust-lang/rust/pull/93359fixes#93305
Optimize `Wtf8Buf::into_string` for the case where it contains UTF-8.
Add a `is_known_utf8` flag to `Wtf8Buf`, which tracks whether the
string is known to contain UTF-8. This is efficiently computed in many
common situations, such as when a `Wtf8Buf` is constructed from a `String`
or `&str`, or with `Wtf8Buf::from_wide` which is already doing UTF-16
decoding and already checking for surrogates.
This makes `OsString::into_string` O(1) rather than O(N) on Windows in
common cases.
And, it eliminates the need to scan through the string for surrogates in
`Args::next` and `Vars::next`, because the strings are already being
translated with `Wtf8Buf::from_wide`.
Many things on Windows construct `OsString`s with `Wtf8Buf::from_wide`,
such as `DirEntry::file_name` and `fs::read_link`, so with this patch,
users of those functions can subsequently call `.into_string()` without
paying for an extra scan through the string for surrogates.
r? `@ghost`
Align android `sigaddset` impl with the reference impl from Bionic
In https://github.com/rust-lang/rust/pull/100737 I noticed we were treating the sigset_t as an array of bytes, while referencing code from android (ad8dcd6023/libc/include/android/legacy_signal_inlines.h) which treats it as an array of unsigned long.
That said, the behavior difference is so subtle here that it's not hard to see why nobody noticed. This fixes the implementation to be equivalent to the one in bionic.
Use pointer `is_aligned*` methods
This PR replaces some manual alignment checks with calls to `pointer::{is_aligned, is_aligned_to}` and removes a useless pointer cast.
r? `@scottmcm`
_split off from #100746_
Replace most uses of `pointer::offset` with `add` and `sub`
As PR title says, it replaces `pointer::offset` in compiler and standard library with `pointer::add` and `pointer::sub`. This generally makes code cleaner, easier to grasp and removes (or, well, hides) integer casts.
This is generally trivially correct, `.offset(-constant)` is just `.sub(constant)`, `.offset(usized as isize)` is just `.add(usized)`, etc. However in some cases we need to be careful with signs of things.
r? ````@scottmcm````
_split off from #100746_
add miri-test-libstd support to libstd
- The first commit mirrors what we already have in liballoc.
- The second commit adds some regression tests that only really make sense to be run in Miri, since they rely on Miri's extra checks to detect anything.
- The third commit makes the MPSC tests work in reasonable time in Miri by reducing iteration counts.
- The fourth commit silences some warnings due to code being disabled with `cfg(miri)`
Windows: Load synch functions together
Attempt to load all the required sync functions and fail if any one of them fails.
This fixes a FIXME by going back to optional loading of `WakeByAddressSingle`.
Also reintroduces a macro for optional loading of functions but keeps it separate from the fallback macro rather than having that do two different jobs.
r? `@thomcc`
Expose `Utf8Lossy` as `Utf8Chunks`
This PR changes the feature for `Utf8Lossy` from `str_internals` to `utf8_lossy` and improves the API. This is done to eventually expose the API as stable.
Proposal: rust-lang/libs-team#54
Tracking Issue: #99543
Avoid zeroing a 1kb stack buffer on every call to `std::sys::windows::fill_utf16_buf`
I've also tried to be slightly more careful about integer overflows, although in practice this is likely still not handled ideally.
r? `@ChrisDenton`
Attempt to load all the required sync functions and fail if any one of them fails.
This reintroduces a macro for optional loading of functions but keeps it separate from the fallback macro rather than having that do two different jobs.
Fix HorizonOS regression in FileTimes
The changes in #98246 caused a regression for multiple Newlib-based systems. This is just a fix including HorizonOS to the list of targets which require a workaround.
``@AzureMarker`` ``@ian-h-chamberlain``
r? ``@nagisa``
promote debug_assert to assert when possible and useful
This PR fixed a very old issue https://github.com/rust-lang/rust/issues/94705 to clarify and improve the POSIX error checking, and some of the checks are skipped because can have no benefit, but I'm sure that this can open some interesting discussion.
Fixes https://github.com/rust-lang/rust/issues/94705
cc: `@tavianator`
cc: `@cuviper`
linux: Use `pthread_setname_np` instead of `prctl`
This function is available on Linux since glibc 2.12, musl 1.1.16, and
uClibc 1.0.20. The main advantage over `prctl` is that it properly
represents the pointer argument, rather than a multi-purpose `long`,
so we're better representing strict provenance (#95496).
This function is available on Linux since glibc 2.12, musl 1.1.16, and
uClibc 1.0.20. The main advantage over `prctl` is that it properly
represents the pointer argument, rather than a multi-purpose `long`,
so we're better representing strict provenance (#95496).
Remove Windows function preloading
After `@Mark-Simulacrum` asked me to provide guidance for when optionally imported functions should be preloaded, I realised my justifications were now quite weak. I think the strongest argument that can be made is that it avoids some degree of nondeterminism when calling these functions (in as far as system API calls can be said to be deterministic). However, I don't think that's particularly convincing unless there's a real world use case where it matters. Further discussion with `@thomcc` has strengthened my feeling that preloading isn't really needed.
Note that `WaitOnAddress` needed some adjustment to work without preloading. I opted not to use a macro for this special case as it seemed silly to do so for just one thing (and I don't like macros tbh).
Fix futex module imports on wasm+atomics
The futex modules were rearranged a bit in #98707, which meant that wasm+atomics would no longer compile on nightly. I don’t believe any other targets were impacted by this.
Remove synchronization from Windows `hashmap_random_keys`
Unfortunately using synchronization when generating hashmap keys can prevent it being used in `DllMain`.
~~Fixes #99341~~
kmc-solid: Update `Socket::connect_timeout` to be in line with #78802
Fixes the build failure of the [`*-kmc-solid_*`](https://doc.rust-lang.org/nightly/rustc/platform-support/kmc-solid.html) Tier 3 targets after #78802.
```
error[E0308]: mismatched types
--> library\std\src\sys\solid\net.rs:234:45
|
234 | cvt(netc::connect(self.0.raw(), addrp, len))
| ------------- ^^^^^ expected *-ptr, found union `SocketAddrCRepr`
| |
| arguments to this function are incorrect
|
= note: expected raw pointer `*const sockets::sockaddr`
found union `SocketAddrCRepr`
note: function defined here
--> library\std\src\sys\solid\abi\sockets.rs:173:12
|
173 | pub fn connect(s: c_int, name: *const sockaddr, namelen: socklen_t) -> c_int;
| ^^^^^^^
```
Support setting file accessed/modified timestamps
Add `struct FileTimes` to contain the relevant file timestamps, since
most platforms require setting all of them at once. (This also allows
for future platform-specific extensions such as setting creation time.)
Add `File::set_file_time` to set the timestamps for a `File`.
Implement the `sys` backends for UNIX, macOS (which needs to fall back
to `futimes` before macOS 10.13 because it lacks `futimens`), Windows,
and WASI.
Implement network primitives with ideal Rust layout, not C system layout
This PR is the result of this internals forum thread: https://internals.rust-lang.org/t/why-are-socketaddrv4-socketaddrv6-based-on-low-level-sockaddr-in-6/13321.
Instead of basing `std:::net::{Ipv4Addr, Ipv6Addr, SocketAddrV4, SocketAddrV6}` on system (C) structs, they are encoded in a more optimal and idiomatic Rust way.
This changes the public API of std by introducing structural equality impls for all four types here, which means that `match ipv4addr { SOME_CONSTANT => ... }` will now compile, whereas previously this was an error. No other intentional changes are introduced to public API.
It's possible to observe the current layout of these types (e.g., by pointer casting); most but not all libraries which were found by Crater to do this have had updates issued and affected versions yanked. See report below.
### Benefits of this change
- It will become possible to move these fundamental network types from `std` into `core` ([RFC](https://github.com/rust-lang/rfcs/pull/2832)).
- Some methods that can't be made `const fn`s today can be made `const fn`s with this change.
- `SocketAddrV4` only occupies 6 bytes instead of 16 bytes.
- These simple primitives become easier to read and uses less `unsafe`.
- Makes these types support structural equality, which means you can now (for instance) match an `Ipv4Addr` against a constant
### ~Remaining~ Previous problems
This change obviously changes the memory layout of the types. And it turns out some libraries invalidly assumes the memory layout and does very dangerous pointer casts to convert them. These libraries will have undefined behaviour and perform invalid memory access until patched.
- [x] - `mio` - Issue: https://github.com/tokio-rs/mio/issues/1386.
- [x] `0.7` branch https://github.com/tokio-rs/mio/pull/1388
- [x] `0.7.6` published https://github.com/tokio-rs/mio/pull/1398
- [x] Yank all `0.7` versions older than `0.7.6`
- [x] Report `<0.7.6` to RustSec Advisory Database https://rustsec.org/advisories/RUSTSEC-2020-0081.html
- [x] - `socket2` - Issue: https://github.com/rust-lang/socket2-rs/issues/119.
- [x] `0.3.x` branch https://github.com/rust-lang/socket2-rs/pull/120
- [x] `0.3.16` published
- [x] `master` branch https://github.com/rust-lang/socket2-rs/pull/122
- [x] Yank all `0.3` versions older than `0.3.16`
- [x] Report `<0.3.16` to RustSec Advisory Database https://rustsec.org/advisories/RUSTSEC-2020-0079.html
- [x] - `net2` - Issue: https://github.com/deprecrated/net2-rs/issues/105
- [x] https://github.com/deprecrated/net2-rs/pull/106
- [x] `0.2.36` published
- [x] Yank all `0.2` versions older than `0.2.36`
- [x] Report `<0.2.36` to RustSec Advisory Database https://rustsec.org/advisories/RUSTSEC-2020-0078.html
- [x] - `miow` - Issue: https://github.com/yoshuawuyts/miow/issues/38
- [x] `0.3.x` - https://github.com/yoshuawuyts/miow/pull/39
- [x] `0.3.6` published
- [x] `0.2.x` - https://github.com/yoshuawuyts/miow/pull/40
- [x] `0.2.2` published
- [x] Yanked all `0.2` versions older than `0.2.2`
- [x] Yanked all `0.3` versions older than `0.3.6`
- [x] Report `<0.2.2` and `<0.3.6` to RustSec Advisory Database https://rustsec.org/advisories/RUSTSEC-2020-0080.html
- [x] - `quinn master` (aka what became 0.7) - https://github.com/quinn-rs/quinn/issues/968https://github.com/quinn-rs/quinn/pull/987
- [x] - `quinn 0.6` - https://github.com/quinn-rs/quinn/pull/1045
- [x] - `quinn 0.5` - https://github.com/quinn-rs/quinn/pull/1046
- [x] - Release `0.7.0`, `0.6.2` and `0.5.4`
- [x] - `nb-connect` - https://github.com/smol-rs/nb-connect/issues/1
- [x] - Release `1.0.3`
- [x] - Yank all versions older than `1.0.3`
- [x] - `shadowsocks-rust` - https://github.com/shadowsocks/shadowsocks-rust/issues/462
- [ ] - `rio` - https://github.com/spacejam/rio/issues/44
- [ ] - `seaslug` - https://github.com/spacejam/seaslug/issues/1
#### Fixed crate versions
All crates I have found that assumed the memory layout have been fixed and published. The crates and versions that will continue working even as/if this PR is merged is (please upgrade these to help unblock this PR):
* `net2 0.2.36`
* `socket2 0.3.16`
* `miow 0.2.2`
* `miow 0.3.6`
* `mio 0.7.6`
* `mio 0.6.23` - Never had the invalid assumption itself, but has now been bumped to only allow fixed dependencies (`net2` + `miow`)
* `nb-connect 1.0.3`
* `quinn 0.5.4`
* `quinn 0.6.2`
### Release notes draft
This release changes the memory layout of `Ipv4Addr`, `Ipv6Addr`, `SocketAddrV4` and `SocketAddrV6`. The standard library no longer implements these as the corresponding `libc` structs (`sockaddr_in`, `sockaddr_in6` etc.). This internal representation was never exposed, but some crates relied on it anyway by unsafely transmuting. This change will cause those crates to make invalid memory accesses. Notably `net2 <0.2.36`, `socket2 <0.3.16`, `mio <0.7.6`, `miow <0.3.6` and a few other crates are affected. All known affected crates have been patched and have had fixed versions published over a year ago. If any affected crate is still in your dependency tree, you need to upgrade them before using this version of Rust.
Rewrite Windows `compat_fn` macro
This allows using most delay loaded functions before the init code initializes them. It also only preloads a select few functions, rather than all functions.
This is optimized for the common case where a function is used after already being loaded (or failed to load). The only change in codegen at the call site is to use an atomic load instead of a plain load, which should have negligible or no impact.
I've split the old `compat_fn` macro in two so as not to mix two different use cases. If/when Windows 7 support is dropped `compat_fn_optional` can be removed entirely.
r? rust-lang/libs
This allows using most delay loaded functions before the init code initializes them. It also only preloads a select few functions, rather than all functions.
Co-Authored-By: Mark Rousskov <mark.simulacrum@gmail.com>
kmc-solid: Use `libc::abort` to abort a program
This PR updates the target-specific abort subroutine for the [`*-kmc-solid_*`](https://doc.rust-lang.org/nightly/rustc/platform-support/kmc-solid.html) Tier 3 targets.
The current implementation uses a `hlt` instruction, which is the most direct way to notify a connected debugger but is not the most flexible way. This PR changes it to call the `abort` libc function, making it possible for a system designer to override its behavior as they see fit.
Add cgroupv1 support to available_parallelism
Fixes#97549
My dev machine uses cgroup v2 so I was only able to test that code path. So the v1 code path is written only based on documentation. I could use some help testing that it works on a machine with cgroups v1:
```
$ x.py build --stage 1
# quota.rs
fn main() {
println!("{:?}", std:🧵:available_parallelism());
}
# assuming stage1 is linked in rustup
$ rust +stage1 quota.rs
# spawn a new cgroup scope for the current user
$ sudo systemd-run -p CPUQuota="300%" --uid=$(id -u) -tdS
# should print Ok(3)
$ ./quota
```
If it doesn't work as expected an strace, the contents of `/proc/self/cgroups` and the structure of `/sys/fs/cgroups` would help.
Some language settings can result in unreliable UTF-8 being produced.
This can result in failing to emit the error string, panicking instead.
from_lossy_utf8 allows us to assume these strings usually will be fine.
std: use futex-based locks on Fuchsia
This switches `Condvar` and `RwLock` to the futex-based implementation currently used on Linux and some BSDs. Additionally, `Mutex` now has its own, priority-inheriting implementation based on the mutex in Fuchsia's `libsync`. It differs from the original in that it panics instead of aborting when reentrant locking is detected.
````@rustbot```` ping fuchsia
r? ````@m-ou-se````
stdlib support for Apple WatchOS
This is a follow-up to https://github.com/rust-lang/rust/pull/95243 (Add Apple WatchOS compiler targets) that adds stdlib support for Apple WatchOS.
`@deg4uss3r`
`@nagisa`
Windows: Use `FindFirstFileW` for getting the metadata of locked system files
Fixes#96980
Usually opening a file handle with access set to metadata only will always succeed, even if the file is locked. However some special system files, such as `C:\hiberfil.sys`, are locked by the system in a way that denies even that. So as a fallback we try reading the cached metadata from the directory.
Note that the test is a bit iffy. I don't know if `hiberfil.sys` actually exists in the CI.
r? rust-lang/libs
Add `struct FileTimes` to contain the relevant file timestamps, since
most platforms require setting all of them at once. (This also allows
for future platform-specific extensions such as setting creation time.)
Add `File::set_file_time` to set the timestamps for a `File`.
Implement the `sys` backends for UNIX, macOS (which needs to fall back
to `futimes` before macOS 10.13 because it lacks `futimens`), Windows,
and WASI.
Implement ExitCodeExt for Windows
Fixes#97914
### Motivation:
On Windows it is common for applications to return `HRESULT` (`i32`) or `DWORD` (`u32`) values. These stem from COM based components ([HRESULTS](https://docs.microsoft.com/en-us/windows/win32/api/objbase/nf-objbase-coinitialize)), Win32 errors ([GetLastError](https://docs.microsoft.com/en-us/windows/win32/api/errhandlingapi/nf-errhandlingapi-getlasterror)), GUI applications ([WM_QUIT](https://docs.microsoft.com/en-us/windows/win32/winmsg/wm-quit)) and more. The newly stabilized `ExitCode` provides an excellent fit for propagating these values, because `std::process::exit` does not run deconstructors which can result in errors. However, `ExitCode` currently only implements `From<u8> for ExitCode`, which disallows the full range of `i32`/`u32` values. This pull requests attempts to address that shortcoming by providing windows specific extensions that accept a `u32` value (which covers all possible `HRESULTS` and Win32 errors) analog to [ExitStatusExt::from_raw](https://doc.rust-lang.org/std/os/windows/process/trait.ExitStatusExt.html#tymethod.from_raw).
This was also intended by the original Stabilization https://github.com/rust-lang/rust/pull/93840#issue-1129209143= as pointed out by ``@eggyal`` in https://github.com/rust-lang/rust/issues/97914#issuecomment-1151076755:
> Issues around platform specific representations: We resolved this issue by changing the return type of report from i32 to the opaque type ExitCode. __That way we can change the underlying representation without affecting the API, letting us offer full support for platform specific exit code APIs in the future.__
[Emphasis added]
### API
```rust
/// Windows-specific extensions to [`process::ExitCode`].
///
/// This trait is sealed: it cannot be implemented outside the standard library.
/// This is so that future additional methods are not breaking changes.
#[stable(feature = "windows_process_exit_code_from", since = "1.63.0")]
pub trait ExitCodeExt: Sealed {
/// Creates a new `ExitCode` from the raw underlying `u32` return value of
/// a process.
#[stable(feature = "windows_process_exit_code_from", since = "1.63.0")]
fn from_raw(raw: u32) -> Self;
}
#[stable(feature = "windows_process_exit_code_from", since = "1.63.0")]
impl ExitCodeExt for process::ExitCode {
fn from_raw(raw: u32) -> Self {
process::ExitCode::from_inner(From::from(raw))
}
}
```
### Misc
I apologize in advance if I misplaced any attributes regarding stabilzation, as far as I learned traits are insta-stable so I chose to make them stable. If this is an error, please let me know and I'll correct it. I also added some additional machinery to make it work, analog to [ExitStatus](https://doc.rust-lang.org/std/process/struct.ExitStatus.html#).
EDIT: Proposal: https://github.com/rust-lang/libs-team/issues/48
Usually opening a file handle with access set to metadata only will always succeed, even if the file is locked. However some special system files, such as `C:\hiberfil.sys`, are locked by the system in a way that denies even that. So as a fallback we try reading the cached metadata from the directory.
Remove feature `const_option` from std
This is part of the effort to reduce the number of unstable features used by std. This one is easy as it's only used in one place.
std: use an event-flag-based thread parker on SOLID
`Mutex` and `Condvar` are being replaced by more efficient implementations, which need thread parking themselves (see #93740). Therefore, the generic `Parker` needs to be replaced on all platforms where the new lock implementation will be used, which, after #96393, are SOLID, SGX and Hermit (more PRs coming soon).
SOLID, conforming to the [μITRON specification](http://www.ertl.jp/ITRON/SPEC/FILE/mitron-400e.pdf), has event flags, which are a thread parking primitive very similar to `Parker`. However, they do not make any atomic ordering guarantees (even though those can probably be assumed) and necessitate a system call even when the thread token is already available. Hence, this `Parker`, like the Windows parker, uses an extra atomic state variable.
I future-proofed the code by wrapping the event flag in a `WaitFlag` structure, as both SGX and Hermit can share the Parker implementation, they just have slightly different primitives (SGX uses signals and Hermit has a thread blocking API).
`````@kawadakk````` I assume you are the target maintainer? Could you test this for me?
Mitigate MMIO stale data vulnerability
Intel publicly disclosed the MMIO stale data vulnerability on June 14. To mitigate this vulnerability, compiler changes are required for the `x86_64-fortanix-unknown-sgx` target.
cc: ````@jethrogb````
Windows: Iterative `remove_dir_all`
This will allow better strategies for use of memory and File handles. However, fully taking advantage of that is left to future work.
Note to reviewer: It's probably best to view the `remove_dir_all_recursive` as a new function. The diff is not very helpful (imho).
Add a `is_known_utf8` flag to `Wtf8Buf`, which tracks whether the
string is known to contain UTF-8. This is efficiently computed in many
common situations, such as when a `Wtf8Buf` is constructed from a `String`
or `&str`, or with `Wtf8Buf::from_wide` which is already doing UTF-16
decoding and already checking for surrogates.
This makes `OsString::into_string` O(1) rather than O(N) on Windows in
common cases.
And, it eliminates the need to scan through the string for surrogates in
`Args::next` and `Vars::next`, because the strings are already being
translated with `Wtf8Buf::from_wide`.
Many things on Windows construct `OsString`s with `Wtf8Buf::from_wide`,
such as `DirEntry::file_name` and `fs::read_link`, so with this patch,
users of those functions can subsequently call `.into_string()` without
paying for an extra scan through the string for surrogates.
once cell renamings
This PR does the renamings proposed in https://github.com/rust-lang/rust/issues/74465#issuecomment-1153703128
- Move/rename `lazy::{OnceCell, Lazy}` to `cell::{OnceCell, LazyCell}`
- Move/rename `lazy::{SyncOnceCell, SyncLazy}` to `sync::{OnceLock, LazyLock}`
(I used `Lazy...` instead of `...Lazy` as it seems to be more consistent, easier to pronounce, etc)
```@rustbot``` label +T-libs-api -T-libs
Windows: No panic if function not (yet) available
In some situations (e.g. #97814) it is possible for required functions to be called before they've had a chance to be loaded. Therefore, we make it possible to recover from this situation simply by looking at error codes.
`@rustbot` label +O-windows
Add `#[inline]` to small fns of futex `RwLock`
The important methods like `read` and `write` were already inlined,
which can propagate all the way to inlining in user code, but these
small state functions were left behind as normal calls. They should
almost always be inlined as well, as they're just a few instructions.
Add a `BorrowedFd::try_clone_to_owned` and accompanying documentation
Add a `BorrowedFd::try_clone_to_owned`, which returns a new `OwnedFd` sharing the underlying file description. And similar for `BorrowedHandle` and `BorrowedSocket` on WIndows.
This is similar to the existing `OwnedFd::try_clone`, but it's named differently to reflect that it doesn't return `Result<Self, ...>`. I'm open to suggestions for better names.
Also, extend the `unix::io` documentation to mention that `dup` is permitted on `BorrowedFd`.
This was originally requsted [here](https://github.com/rust-lang/rust/issues/88564#issuecomment-910786081). At the time I wasn't sure whether it was desirable, but it does have uses and it helps clarify the API. The documentation previously didn't rule out using `dup` on a `BorrowedFd`, but the API only offered convenient ways to do it from an `OwnedFd`. With this patch, the API allows one to do `try_clone` on any type where it's permitted.
The important methods like `read` and `write` were already inlined,
which can propagate all the way to inlining in user code, but these
small state functions were left behind as normal calls. They should
almost always be inlined as well, as they're just a few instructions.
STD support for the Nintendo 3DS
Rustc already supports compiling for the Nintendo 3DS using the `armv6k-nintendo-3ds` target (Tier 3). Until now though, only `core` and `alloc` were supported. This PR adds standard library support for the Nintendo 3DS. A notable exclusion is `std::thread` support, which will come in a follow-up PR as it requires more complicated changes.
This has been a joint effort by `@Meziu,` `@ian-h-chamberlain,` myself, and prior work by `@rust3ds` members.
### Background
The Nintendo 3DS (Horizon OS) is a mostly-UNIX looking system, with the caveat that it does not come with a full libc implementation out of the box. On the homebrew side (I'm not under NDA), the libc interface is partially implemented by the [devkitPro](https://devkitpro.org/wiki/devkitPro_pacman) toolchain and a user library like [`libctru`](https://github.com/devkitPro/libctru). This is important because there are [some possible legal barriers](https://github.com/rust-lang/rust/pull/88529#issuecomment-919938396) to linking directly to a library that uses the underlying platform APIs, since they might be considered a trade secret or under NDA.
To get around this, the standard library impl for the 3DS does not directly depend on any platform-level APIs. Instead, it expects standard libc functions to be linked in. The implementation of these libc functions is left to the user. Some functions are provided by the devkitPro toolchain, but in our testing, we used the following to fill in the other functions:
- [`libctru`] - provides more basic APIs, such as `nanosleep`. Linked in by way of [`ctru-sys`](https://github.com/Meziu/ctru-rs/tree/master/ctru-sys).
- [`pthread-3ds`](https://github.com/Meziu/pthread-3ds) - provides pthread APIs for `std::thread`. Implemented using [`libctru`].
- [`linker-fix-3ds`](https://github.com/Meziu/rust-linker-fix-3ds) - fulfills some other missing libc APIs. Implemented using [`libctru`].
For more details, see the `src/doc/rustc/src/platform-support/armv6k-nintendo-3ds.md` file added in this PR.
### Notes
We've already upstreamed changes to the [`libc`] crate to support this PR, as well as the upcoming threading PR. These changes have all been released as of 0.2.121, so we bump the crate version in this PR.
Edit: After some rebases, the version bump has already been merged so it doesn't appear in this PR.
A lot of the changes in this PR are straightforward, and follow in the footsteps of the ESP-IDF target: https://github.com/rust-lang/rust/pull/87666.
The 3DS does not support user space process spawning, so these APIs are unimplemented (similar to ESP-IDF).
[`libctru`]: https://github.com/devkitPro/libctru
[`libc`]: https://github.com/rust-lang/libc
Our condvar doesn't support setting attributes, like
pthread_condattr_setclock, which the current wait_timeout expects to
have configured.
Switch to a different implementation, following espidf.
Use `fcntl(fd, F_GETFD)` to detect if standard streams are open
In the previous implementation, if the standard streams were open,
but the RLIMIT_NOFILE value was below three, the poll would fail
with EINVAL:
> ERRORS: EINVAL The nfds value exceeds the RLIMIT_NOFILE value.
Switch to the existing fcntl based implementation to avoid the issue.
Fixes#96621.
In some situations it is possible for required functions to be called before they've had a chance to be loaded. Therefore, we make it possible to recover from this situation simply by looking at error codes.
Call the OS function to set the main thread's name on program init
Normally, `Thread::spawn` takes care of setting the thread's name, if
one was provided, but since the main thread wasn't created by calling
`Thread::spawn`, we need to call that function in `std::rt::init`.
This is mainly useful for system tools like debuggers and profilers
which might show the thread name to a user. Prior to these changes, gdb
and WinDbg would show all thread names except the main thread's name to
a user. I've validated that this patch resolves the issue for both
debuggers.
Lazily allocate and initialize pthread locks.
Lazily allocate and initialize pthread locks.
This allows {Mutex, Condvar, RwLock}::new() to be const, while still using the platform's native locks for features like priority inheritance and debug tooling. E.g. on macOS, we cannot directly use the (private) APIs that pthread's locks are implemented with, making it impossible for us to use anything other than pthread while still preserving priority inheritance, etc.
This PR doesn't yet make the public APIs const. That's for a separate PR with an FCP.
Tracking issue: https://github.com/rust-lang/rust/issues/93740
The oldest occurence is from 9e224c2bf1,
which is from the pre-1.0 days. In the years since then, std::sys still
hasn't been exported, and the last attempt was met with strong criticism:
https://github.com/rust-lang/rust/pull/97151
Thus, removing the "yet" part makes a lot of sense.
Normally, `Thread::spawn` takes care of setting the thread's name, if
one was provided, but since the main thread wasn't created by calling
`Thread::spawn`, we need to call that function in `std::rt::init`.
This is mainly useful for system tools like debuggers and profilers
which might show the thread name to a user. Prior to these changes, gdb
and WinDbg would show all thread names except the main thread's name to
a user. I've validated that this patch resolves the issue for both
debuggers.
I wrote `state` where I should've used `s`.
This removes the unnecessary `s` variable to prevent that mistake.
Fortunately, this typo didn't affect the correctness of the lock, as the
second half of the condition (!has_writers_waiting) is enough for
correctness, which explains why this mistake didn't show up during
testing.
From reading the source code, it appears like the desired semantic of
std::unix::rand is to always provide some bytes and never block. For
that reason GRND_NONBLOCK is checked before calling getrandom(0), so
that getrandom(0) won't block. If it would block, then the function
falls back to using /dev/urandom, which for the time being doesn't
block. There are some drawbacks to using /dev/urandom, however, and so
getrandom(GRND_INSECURE) was created as a replacement for this exact
circumstance.
getrandom(GRND_INSECURE) is the same as /dev/urandom, except:
- It won't leave a warning in dmesg if used at early boot time, which is
a common occurance (and the reason why I found this issue);
- It won't introduce a tiny delay at early boot on newer kernels when
/dev/urandom tries to opportunistically create jitter entropy;
- It only requires 1 syscall, rather than 3.
Other than that, it returns the same "quality" of randomness as
/dev/urandom, and never blocks.
It's only available on kernels ≥5.6, so we try to use it, cache the
result of that attempt, and fall back to to the previous code if it
didn't work.
Make HashMap fall back to RtlGenRandom if BCryptGenRandom fails
With PR #84096, Rust `std::collections::hash_map::RandomState` changed from using `RtlGenRandom()` ([msdn](https://docs.microsoft.com/en-us/windows/win32/api/ntsecapi/nf-ntsecapi-rtlgenrandom)) to `BCryptGenRandom()` ([msdn](https://docs.microsoft.com/en-us/windows/win32/api/bcrypt/nf-bcrypt-bcryptgenrandom)) as its source of secure randomness after much discussion ([here](https://github.com/rust-random/getrandom/issues/65#issuecomment-753634074), among other places).
Unfortunately, after that PR landed, Mozilla Firefox started experiencing fairly-rare crashes during startup while attempting to initialize the `env_logger` crate. ([docs for env_logger](https://docs.rs/env_logger/latest/env_logger/)) The root issue is that on some machines, `BCryptGenRandom()` will fail with an `Access is denied. (os error 5)` error message. ([Bugzilla issue 1754490](https://bugzilla.mozilla.org/show_bug.cgi?id=1754490)) (Discussion in issue #94098)
Note that this is happening upon startup of Firefox's unsandboxed Main Process, so this behavior is different and separate from previous issues ([like this](https://bugzilla.mozilla.org/show_bug.cgi?id=1746254)) where BCrypt DLLs were blocked by process sandboxing. In the case of sandboxing, we knew we were doing something abnormal and expected that we'd have to resort to abnormal measures to make it work.
However, in this case we are in a regular unsandboxed process just trying to initialize `env_logger` and getting a panic. We suspect that this may be caused by a virus scanner or some other security software blocking the loading of the BCrypt DLLs, but we're not completely sure as we haven't been able to replicate locally.
It is also possible that Firefox is not the only software affected by this; we just may be one of the pieces of Rust software that has the telemetry and crash reporting necessary to catch it.
I have read some of the historical discussion around using `BCryptGenRandom()` in Rust code, and I respect the decision that was made and agree that it was a good course of action, so I'm not trying to open a discussion about a return to `RtlGenRandom()`. Instead, I'd like to suggest that perhaps we use `RtlGenRandom()` as a "fallback RNG" in the case that BCrypt doesn't work.
This pull request implements this fallback behavior. I believe this would improve the robustness of this essential data structure within the standard library, and I see only 2 potential drawbacks:
1. Slight added overhead: It should be quite minimal though. The first call to `sys::rand::hashmap_random_keys()` will incur a bit of initialization overhead, and every call after will incur roughly 2 non-atomic global reads and 2 easily predictable branches. Both should be negligible compared to the actual cost of generating secure random numbers
2. `RtlGenRandom()` is deprecated by Microsoft: Technically true, but as mentioned in [this comment on GoLang](https://github.com/golang/go/issues/33542#issuecomment-626124873), this API is ubiquitous in Windows software and actually removing it would break lots of things. Also, Firefox uses it already in [our C++ code](https://searchfox.org/mozilla-central/rev/5f88c1d6977e03e22d3420d0cdf8ad0113c2eb31/mfbt/RandomNum.cpp#25), and [Chromium uses it in their code as well](https://source.chromium.org/chromium/chromium/src/+/main:base/rand_util_win.cc) (which transitively means that Microsoft uses it in their own web browser, Edge). If there did come a time when Microsoft truly removes this API, it should be easy enough for Rust to simply remove the fallback in the code I've added here
In the previous implementation, if the standard streams were open,
but the RLIMIT_NOFILE value was below three, the poll would fail
with EINVAL:
> ERRORS: EINVAL The nfds value exceeds the RLIMIT_NOFILE value.
Switch to the existing fcntl based implementation to avoid the issue.
Issue #84096 changed the hashmap RNG to use BCryptGenRandom instead of
RtlGenRandom on Windows.
Mozilla Firefox started experiencing random failures in
env_logger::Builder::new() (Issue #94098) during initialization of their
unsandboxed main process with an "Access Denied" error message from
BCryptGenRandom(), which is used by the HashMap contained in
env_logger::Builder
The root cause appears to be a virus scanner or other software interfering
with BCrypt DLLs loading.
This change adds a fallback option if BCryptGenRandom is unusable for
whatever reason. It will fallback to RtlGenRandom in this case.
Fixes#94098
Windows: Make stdin pipes synchronous
Stdin pipes do not need to be used asynchronously within the standard library. This is a first step in making pipes mostly synchronous.
r? `@m-ou-se`
std: directly use pthread in UNIX parker implementation
`Mutex` and `Condvar` are being replaced by more efficient implementations, which need thread parking themselves (see #93740). Therefore we should use the `pthread` synchronization primitives directly. Also, we can avoid allocating the mutex and condition variable because the `Parker` struct is being placed in an `Arc` anyways.
This basically is just a copy of the current `Mutex` and `Condvar` code, which will however be removed (again, see #93740). An alternative implementation could be to use dedicated private `OsMutex` and `OsCondvar` types, but all the other platforms supported by std actually have their own thread parking primitives.
I used `Pin` to guarantee a stable address for the `Parker` struct, while the current implementation does not, rather using extra unsafe declaration. Since the thread struct is shared anyways, I assumed this would not add too much clutter while being clearer.
Mutex and Condvar are being replaced by more efficient implementations, which need thread parking themselves (see #93740). Therefore use the pthread synchronization primitives directly. Also, avoid allocating because the Parker struct is being placed in an Arc anyways.
Windows Command: Don't run batch files using verbatim paths
Fixes#95178
Note that the first commit does some minor refactoring (moving command line argument building to args.rs). The actual changes are in the second.
Reduce allocations for path conversions on Windows
Previously, UTF-8 to UTF-16 Path conversions on Windows unnecessarily allocate twice, as described in #96297. This commit fixes that issue.
Improve Windows path prefix parsing
This PR fixes improves parsing of Windows path prefixes. `parse_prefix` now supports both types of separators on Windows (`/` and `\`).
[fuchsia] Add implementation for `current_exe`
This implementation returns a best attempt at the current exe path. On
fuchsia, fdio will always use `argv[0]` as the process name and if it is
not set then an error will be returned. Because this is not guaranteed
to be the case, this implementation returns an error if `argv` does not
contain any elements.
remove_dir_all_recursive: treat ELOOP the same as ENOTDIR
On older Linux kernels (I tested on 4.4, corresponding to Ubuntu 16.04), opening a symlink using `O_DIRECTORY | O_NOFOLLOW` returns `ELOOP` instead of `ENOTDIR`. We should handle it the same, since a symlink is still not a directory and needs to be `unlink`ed.
Use sys::unix::locks::futex* on wasm+atomics.
This removes the wasm-specific lock implementations and instead re-uses the implementations from sys::unix.
Tracking issue: https://github.com/rust-lang/rust/issues/93740
cc ``@alexcrichton``
This implementation returns a best attempt at the current exe path. On
fuchsia, fdio will always use `argv[0]` as the process name and if it is
not set then an error will be returned. Because this is not guaranteed
to be the case, this implementation returns an error if `argv` does not
contain any elements.
Use a single ReentrantMutex implementation on all platforms.
This replaces all platform specific ReentrantMutex implementations by the one I added in #95727 for Linux, since that one does not depend on any platform specific details.
r? `@Amanieu`
fix error handling for pthread_sigmask(3)
Errors from `pthread_sigmask(3)` were handled using `cvt()`, which expects a return value of `-1` on error and uses `errno`.
However, `pthread_sigmask(3)` returns `0` on success and an error number otherwise.
Fix it by replacing `cvt()` with `cvt_nz()`.
Use u32 instead of i32 for futexes.
This changes futexes from i32 to u32. The [Linux man page](https://man7.org/linux/man-pages/man2/futex.2.html) uses `uint32_t` for them, so I'm not sure why I used i32 for them. Maybe because I first used them for thread parkers, where I used -1, 0, and 1 as the states.
(Wasm's `memory.atomic.wait32` does use `i32`, because wasm doesn't support `u32`.)
It doesn't matter much, but using the unsigned type probably results in fewer surprises when shifting bits around or using comparison operators.
r? ```@Amanieu```
Windows: Use a pipe relay for chaining pipes
Fixes#95759
This fixes the issue by chaining pipes synchronously and manually pumping messages between them. It's not ideal but it has the advantage of not costing anything if pipes are not chained ("don't pay for what you don't use") and it also avoids breaking existing code that rely on our end of the pipe being asynchronous (which includes rustc's own testing framework).
Libraries can avoid needing this by using their own pipes to chain commands.
Replace RwLock by a futex based one on Linux
This replaces the pthread-based RwLock on Linux by a futex based one.
This implementation is similar to [the algorithm](https://gist.github.com/kprotty/3042436aa55620d8ebcddf2bf25668bc) suggested by `@kprotty,` but modified to prefer writers and spin before sleeping. It uses two futexes: One for the readers to wait on, and one for the writers to wait on. The readers futex contains the state of the RwLock: The number of readers, a bit indicating whether writers are waiting, and a bit indicating whether readers are waiting. The writers futex is used as a simple condition variable and its contents are meaningless; it just needs to be changed on every notification.
Using two futexes rather than one has the obvious advantage of allowing a separate queue for readers and writers, but it also means we avoid the problem a single-futex RwLock would have of making it hard for a writer to go to sleep while the number of readers is rapidly changing up and down, as the writers futex is only changed when we actually want to wake up a writer.
It always prefers writers, as we decided [here](https://github.com/rust-lang/rust/issues/93740#issuecomment-1070696128).
To be able to prefer writers, it relies on futex_wake to return the number of awoken threads to be able to handle write-unlocking while both the readers-waiting and writers-waiting bits are set. Instead of waking both and letting them race, it first wakes writers and only continues to wake the readers too if futex_wake reported there were no writers to wake up.
r? `@Amanieu`
