Use associated items of `char` instead of freestanding items in `core::char`
The associated functions and constants on `char` have been stable since 1.52 and the freestanding items have soft-deprecated since 1.62 (https://github.com/rust-lang/rust/pull/95566). This PR ~~marks them as "deprecated in future", similar to the integer and floating point modules (`core::{i32, f32}` etc)~~ replaces all uses of `core::char::*` with `char::*` to prepare for future deprecation of `core::char::*`.
Stop at the first `NULL` argument when iterating `argv`
Some C commandline parsers (e.g. GLib and Qt) are replacing already handled arguments in `argv` with `NULL` and move them to the end. That means that `argc` might be bigger than the actual number of non-`NULL` pointers in `argv` at this point.
To handle this we simply stop iterating at the first `NULL` argument.
`argv` is also guaranteed to be `NULL`-terminated so any non-`NULL` arguments after the first `NULL` can safely be ignored.
Fixes https://github.com/rust-lang/rust/issues/105999
Use `__wasilibc_get_environ()` to read the environment variable list
from wasi-libc instead of using `environ`. `environ` is a global
variable which effectively requires wasi-libc to initialize the
environment variables eagerly, and `__wasilibc_get_environ()` is
specifically designed to be an alternative that lets wasi-libc
intiailize its environment variables lazily.
This should have the side effect of fixing at least some of the cases
of #107635.
library/std/sys_common: Define MIN_ALIGN for m68k-unknown-linux-gnu
This PR adds the missing definition of MIN_ALIGN for the m68k-unknown-linux target.
Avoid __cxa_thread_atexit_impl on Emscripten
- Fixes https://github.com/rust-lang/rust/issues/91628.
- Fixes https://github.com/emscripten-core/emscripten/issues/15722.
See discussion in both issues.
The TL;DR is that weak linkage causes LLVM to produce broken Wasm, presumably due to pointer mismatch. The code is casting a void pointer to a function pointer with specific signature, but Wasm is very strict about function pointer compatibility, so the resulting code is invalid.
Ideally LLVM should catch this earlier in the process rather than emit invalid Wasm, but it currently doesn't and this is an easy and valid fix, given that Emcripten doesn't have `__cxa_thread_atexit_impl` these days anyway.
Unfortunately, I can't add a regression test as even after looking into this issue for a long time, I couldn't reproduce it with any minimal Rust example, only with extracted LLVM IR or on a large project involving Rust + C++.
Do not use box syntax in `std`
See #94970 and #49733. About half of the `box` instances in `std` do not even need to allocate, the other half can simply be replaced with `Box::new`.
`@rustbot` label +T-libs
r? rust-lang/libs
Remove various double spaces in the libraries.
I was just pretty bothered by this when reading the source for a function, and was suggested to check if this happened elsewhere.
Stop probing for statx unless necessary
As is the current toy program:
fn main() -> std::io::Result<()> {
use std::fs;
let metadata = fs::metadata("foo.txt")?;
assert!(!metadata.is_dir());
Ok(())
}
... observed under strace will issue:
[snip]
statx(0, NULL, AT_STATX_SYNC_AS_STAT, STATX_ALL, NULL) = -1 EFAULT (Bad address) statx(AT_FDCWD, "foo.txt", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=0, ...}) = 0
While statx is not necessarily always present, checking for it can be delayed to the first error condition. Said condition may very well never happen, in which case the check got avoided altogether.
Note this is still suboptimal as there still will be programs issuing it, but bulk of the problem is removed.
Tested by forbidding the syscall for the binary and observing it correctly falls back to newfstatat.
While here tidy up the commentary, in particular by denoting some problems with the current approach.
- Fixes https://github.com/rust-lang/rust/issues/91628.
- Fixes https://github.com/emscripten-core/emscripten/issues/15722.
See discussion in both issues.
The TL;DR is that weak linkage causes LLVM to produce broken Wasm, presumably due to pointer mismatch. The code is casting a void pointer to a function pointer with specific signature, but Wasm is very strict about function pointer compatibility, so the resulting code is invalid.
Ideally LLVM should catch this earlier in the process rather than emit invalid Wasm, but it currently doesn't and this is an easy and valid fix, given that Emcripten doesn't have `__cxa_thread_atexit_impl` these days anyway.
Unfortunately, I can't add a regression test as even after looking into this issue for a long time, I couldn't reproduce it with any minimal Rust example, only with extracted LLVM IR or on a large project involving Rust + C++.
r? @alexcrichton
As is the current toy program:
fn main() -> std::io::Result<()> {
use std::fs;
let metadata = fs::metadata("foo.txt")?;
assert!(!metadata.is_dir());
Ok(())
}
... observed under strace will issue:
[snip]
statx(0, NULL, AT_STATX_SYNC_AS_STAT, STATX_ALL, NULL) = -1 EFAULT (Bad address)
statx(AT_FDCWD, "foo.txt", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=0, ...}) = 0
While statx is not necessarily always present, checking for it can be
delayed to the first error condition. Said condition may very well never
happen, in which case the check got avoided altogether.
Note this is still suboptimal as there still will be programs issuing
it, but bulk of the problem is removed.
Tested by forbidding the syscall for the binary and observing it
correctly falls back to newfstatat.
While here tidy up the commentary, in particular by denoting some
problems with the current approach.
Even where actually running processes is not supported.
Needed for the next commit.
The manual trait implementations now belong on ExitStatusError,
which still can't exist.
Unify id-based thread parking implementations
Multiple platforms currently use thread-id-based parking implementations (NetBSD and SGX[^1]). Even though the strategy does not differ, these are duplicated for each platform, as the id is encoded into an atomic thread variable in different ways for each platform.
Since `park` is only called by one thread, it is possible to move the thread id into a separate field. By ensuring that the field is only written to once, before any other threads access it, these accesses can be unsynchronized, removing any restrictions on the size and niches of the thread id.
This PR also renames the internal `thread_parker` modules to `thread_parking`, as that name now better reflects their contents. I hope this does not add too much reviewing noise.
r? `@m-ou-se`
`@rustbot` label +T-libs
[^1]: SOLID supports this as well, I will switch it over in a follow-up PR.
kmc-solid: Fix memory ordering in thread operations
Fixes two memory ordering issues in the thread state machine (`ThreadInner::lifecycle`) of the [`*-kmc-solid_*`](https://doc.rust-lang.org/nightly/rustc/platform-support/kmc-solid.html) Tier 3 targets.
1. When detaching a thread that is still running (i.e., the owner updates `lifecycle` first, and the child updates it next), the first update did not synchronize-with the second update, resulting in a data race between the first update and the deallocation of `ThreadInner` by the child thread.
2. When joining on a thread, the joiner has to pass its own task ID to the joinee in order to be woken up later, but in doing so, it did not synchronize-with the read operation, creating possible sequences of execution where the joinee wakes up an incorrect or non-existent task.
Both issue are theoretical and most likely have never manifested in practice because of the stronger guarantees provided by the Arm memory model (particularly due to its barrier-based definition). Compiler optimizations could have subverted this, but the inspection of compiled code did not reveal such optimizations taking place.
Bump master bootstrap compiler
This PR bumps the bootstrap compiler to the beta created earlier this week, cherry-picks the stabilization version number updates, and updates the `cfg(bootstrap)`s.
r? `@Mark-Simulacrum`
available_parallelism: Gracefully handle zero value cfs_period_us
There seem to be some scenarios where the cgroup cpu quota field `cpu.cfs_period_us` can contain `0`. This field is used to determine the "amount" of parallelism suggested by the function `std:🧵:available_parallelism`
A zero value of this field cause a panic when `available_parallelism()` is invoked. This issue was detected by the call from binaries built by `cargo test`. I really don't feel like `0` is a good value for `cpu.cfs_period_us`, but I also don't think applications should panic if this value is seen.
This panic started happening with rust 1.64.0.
This case is gracefully handled by other projects which read this information: [num_cpus](e437b9d908/src/linux.rs (L207-L210)), [ninja](https://github.com/ninja-build/ninja/pull/2174/files), [dotnet](c4341d45ac/src/coreclr/pal/src/misc/cgroup.cpp (L481-L483))
Before this change, running `cargo test` in environments configured as described above would trigger this panic:
```
$ RUST_BACKTRACE=1 cargo test
Finished test [unoptimized + debuginfo] target(s) in 3.55s
Running unittests src/main.rs (target/debug/deps/x-9a42e145aca2934d)
thread 'main' panicked at 'attempt to divide by zero', library/std/src/sys/unix/thread.rs:546:70
stack backtrace:
0: rust_begin_unwind
1: core::panicking::panic_fmt
2: core::panicking::panic
3: std::sys::unix:🧵:cgroups::quota
4: std::sys::unix:🧵:available_parallelism
5: std:🧵:available_parallelism
6: test::helpers::concurrency::get_concurrency
7: test::console::run_tests_console
8: test::test_main
9: test::test_main_static
10: x::main
at ./src/main.rs:1:1
11: core::ops::function::FnOnce::call_once
at /tmp/rust-1.64-1.64.0-1/library/core/src/ops/function.rs:248:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
error: test failed, to rerun pass '--bin x'
```
I've tested this change in an environment which has the bad (questionable?) setup and rebuilding the test executable against a fixed std library fixes the panic.
Use correct clock in `park_timeout` on Horizon
Horizon does not support using `CLOCK_MONOTONIC` with condition variables, so use the system time instead.
Some C commandline parsers (e.g. GLib and Qt) are replacing already
handled arguments in `argv` with `NULL` and move them to the end. That
means that `argc` might be bigger than the actual number of non-`NULL`
pointers in `argv` at this point.
To handle this we simply stop iterating at the first `NULL` argument.
`argv` is also guaranteed to be `NULL`-terminated so any non-`NULL`
arguments after the first `NULL` can safely be ignored.
Fixes https://github.com/rust-lang/rust/issues/105999
Use a more efficient `Once` on platforms without threads
The current implementation uses an atomic queue and spins rather than panicking when calling `call_once` recursively. Since concurrency is not supported on platforms like WASM, `Once` can be implemented much more efficiently using just a single non-atomic state variable.
fs: Fix#50619 (again) and add a regression test
Bug #50619 was fixed by adding an end_of_stream flag in #50630.
Unfortunately, that fix only applied to the readdir_r() path. When I
switched Linux to use readdir() in #92778, I inadvertently reintroduced
the bug on that platform. Other platforms that had always used
readdir() were presumably never fixed.
This patch enables end_of_stream for all platforms, and adds a
Linux-specific regression test that should hopefully prevent the bug
from being reintroduced again.
Allow blocking `Command::output`
### Problem
Currently, `Command::output` is internally implemented using `Command::spawn`. This is problematic because some targets (like UEFI) do not actually support multitasking and thus block while the program is executing. This coupling does not make much sense as `Command::output` is supposed to block until the execution is complete anyway and thus does not need to rely on a non-blocking `Child` or any other intermediate.
### Solution
This PR moves the implementation of `Command::output` to `std::sys`. This means targets can choose to implement only `Command::output` without having to implement `Command::spawn`.
### Additional Information
This was originally conceived when working on https://github.com/rust-lang/rust/pull/100316. Currently, the only target I know about that will benefit from this change is UEFI.
This PR can also be used to implement more efficient `Command::output` since the intermediate `Process` is not actually needed anymore, but that is outside the scope of this PR.
Since this is not a public API change, I'm not sure if an RFC is needed or not.
Use more LFS functions.
On Linux, use mmap64, open64, openat64, and sendfile64 in place of their non-LFS counterparts.
This is relevant to #94173.
With these changes (together with rust-lang/backtrace-rs#501), the simple binaries I produce with rustc seem to have no non-LFS functions, so maybe #94173 is fixed. But I can't be sure if I've missed something and maybe some non-LFS functions could sneak in somehow.
Bug #50619 was fixed by adding an end_of_stream flag in #50630.
Unfortunately, that fix only applied to the readdir_r() path. When I
switched Linux to use readdir() in #92778, I inadvertently reintroduced
the bug on that platform. Other platforms that had always used
readdir() were presumably never fixed.
This patch enables end_of_stream for all platforms, and adds a
Linux-specific regression test that should hopefully prevent the bug
from being reintroduced again.
This allows decoupling `Command::spawn` and `Command::output`. This is
useful for targets which do support launching programs in blocking mode
but do not support multitasking (Eg: UEFI).
This was originally conceived when working on https://github.com/rust-lang/rust/pull/100316
Signed-off-by: Ayush Singh <ayushsingh1325@gmail.com>
Avoid heap allocation when truncating thread names
Ensure that heap allocation does not occur in a thread until `std::thread` is ready. This fixes issues with custom allocators that call `std:🧵:current()`, since doing so prematurely initializes `THREAD_INFO` and causes the following `thread_info::set()` to fail.
Reimplement std's thread parker on top of events on SGX
Mutex and Condvar are being replaced by more efficient implementations, which need thread parking themselves (see #93740). Therefore, the generic `Parker` needs to be replaced on all platforms where the new lock implementation will be used.
SGX enclaves have a per-thread event state, which allows waiting for and setting specific bits. This is already used by the current mutex implementation. The thread parker can however be much more efficient, as it only needs to store the `TCS` address of one thread. This address is stored in a state variable, which can also be set to indicate the thread was already notified.
`park_timeout` does not guard against spurious wakeups like the current condition variable does. This is allowed by the API of `Parker`, and I think it is better to let users handle these wakeups themselves as the guarding is quite expensive and might not be necessary.
`@jethrogb` as you wrote the initial SGX support for `std`, I assume you are the target maintainer? Could you help me test this, please? Lacking a x86_64 chip, I can't run SGX.
Add `read_to_end` method for `sys::{target}::pipe::AnonPipe`. This allows
having a more optimized version of `read_to_end` for ChildStdout.
Signed-off-by: Ayush Singh <ayushsingh1325@gmail.com>
kmc-solid: `std::sys` code maintenance
Includes a set of changes to fix the [`*-kmc-solid_*`](https://doc.rust-lang.org/nightly/rustc/platform-support/kmc-solid.html) Tier 3 targets and make some other improvements.
- Address `fuzzy_provenance_casts` by using `expose_addr` and `from_exposed_addr` for pointer-integer casts
- Add a stub implementation of `is_terminal` (#98070)
- Address `unused_imports` and `unused_unsafe`
- Stop doing `Box::from_raw(&*(x: Box<T>) as *const T as *mut T)`
Ensure that heap allocation does not occur in a thread until std::thread
is ready. This fixes issues with custom allocators that call
std:🧵:current(), since doing so prematurely initializes
THREAD_INFO and causes the following thread_info::set() to fail.
On Linux, use mmap64, open64, openat64, and sendfile64 in place of their
non-LFS counterparts.
This is relevant to #94173.
With these changes (together with rust-lang/backtrace-rs#501), the
simple binaries I produce with rustc seem to have no non-LFS functions,
so maybe #94173 is fixed. But I can't be sure if I've missed something
and maybe some non-LFS functions could sneak in somehow.
Pass on null handle values to child process
Fixes#101645
In Windows, stdio handles are (semantically speaking) `Option<Handle>` where `Handle` is a non-zero value. When spawning a process with `Stdio::Inherit`, Rust currently turns zero values into `-1` values. This has the unfortunate effect of breaking console subprocesses (which typically need stdio) that are spawned from gui applications (that lack stdio by default) because the console process won't be assigned handles from the newly created console (as they usually would in that situation). Worse, `-1` is actually [a valid handle](https://doc.rust-lang.org/std/os/windows/io/struct.OwnedHandle.html) which means "the current process". So if a console process, for example, waits on stdin and it has a `-1` value then the process will end up waiting on itself.
This PR fixes it by propagating the nulls instead of converting them to `-1`.
While I think the current behaviour is a mistake, changing it (however justified) is an API change so I think this PR should at least have some input from t-libs-api. So choosing at random...
r? `@joshtriplett`
Commit 77005950f0 implemented masking of
FileType to fix an issue[^1] in the semantic of FileType comparison.
This commit introduces masking to Hash to maintain the invariant that
x == y => hash(x) == hash(y).
[^1]: https://github.com/rust-lang/rust/issues/104900
When spawning Commands, the path we use can end up being queried using `env::current_exe` (or the equivalent in other languages). Not all applications handle these paths properly therefore we should have a stronger preference for non-verbatim paths when spawning processes.
Pointer-integer casts are required for conversion between `EXINF` (ITRON
task entry point parameter) and `*const ThreadInner`. Addresses the
deny-level lint `fuzzy_provenance_casts`.
Extract WStrUnits to sys_common::wstr
This commit extracts WStrUnits from sys::windows::args to sys_common::wstr. This allows using the same structure for other targets which use wtf8 (example UEFI).
This was originally a part of https://github.com/rust-lang/rust/pull/100316
Signed-off-by: Ayush Singh <ayushsingh1325@gmail.com>
This commit extracts WStrUnits from sys::windows::args to sys_common::wstr. This
allows using the same structure for other targets which use wtf8 (example UEFI).
This was originally a part of https://github.com/rust-lang/rust/pull/100316
Signed-off-by: Ayush Singh <ayushsingh1325@gmail.com>
There seem to be some scenarios where `cpu.cfs_period_us` can contain `0`
This causes a panic when calling `std:🧵:available_parallelism()` as is done so
from binaries built by `cargo test`, which was how the issue was
discovered. I don't feel like `0` is a good value for `cpu.cfs_period_us`, but I
also don't think applications should panic if this value is seen.
This case is handled by other projects which read this information:
- num_cpus: e437b9d908/src/linux.rs (L207-L210)
- ninja: https://github.com/ninja-build/ninja/pull/2174/files
- dotnet: c4341d45ac/src/coreclr/pal/src/misc/cgroup.cpp (L481-L483)
Before this change, this panic could be seen in environments setup as described
above:
```
$ RUST_BACKTRACE=1 cargo test
Finished test [unoptimized + debuginfo] target(s) in 3.55s
Running unittests src/main.rs (target/debug/deps/x-9a42e145aca2934d)
thread 'main' panicked at 'attempt to divide by zero', library/std/src/sys/unix/thread.rs:546:70
stack backtrace:
0: rust_begin_unwind
1: core::panicking::panic_fmt
2: core::panicking::panic
3: std::sys::unix:🧵:cgroups::quota
4: std::sys::unix:🧵:available_parallelism
5: std:🧵:available_parallelism
6: test::helpers::concurrency::get_concurrency
7: test::console::run_tests_console
8: test::test_main
9: test::test_main_static
10: x::main
at ./src/main.rs:1:1
11: core::ops::function::FnOnce::call_once
at /tmp/rust-1.64-1.64.0-1/library/core/src/ops/function.rs:248:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
error: test failed, to rerun pass '--bin local-rabmq-amqpprox'
```
I've tested this change in an environment which has the bad setup and
rebuilding the test executable against a fixed std library fixes the
panic.
Truncate thread names on Linux and Apple targets
These targets have system limits on the thread names, 16 and 64 bytes
respectively, and `pthread_setname_np` returns an error if the name is
longer. However, we're not in a context that can propagate errors when
we call this, and we used to implicitly truncate on Linux with `prctl`,
so now we manually truncate these names ahead of time.
r? ``````@thomcc``````
Don't link to `libresolv` in libstd on Darwin
Currently we link `libresolv` into every Rust program on apple targets despite never using it (as of https://github.com/rust-lang/rust/pull/44965). I had thought we needed this for `getaddrinfo` or something, but we do not / cannot safely use it.
I'd like to fix this for `libiconv` too (the other library we pull in. that's harder since it's coming in through `libc`, which is https://github.com/rust-lang/libc/pull/2944)).
---
This may warrant release notes. I'm not sure but I've added the flag regardless -- It's a change to the list of dylibs every Rust program pulls in, so it's worth mentioning.
It's pretty unlikely anybody was relying on this being pulled in, and `std` does not guarantee that it will link (and thus transitively provide access to) any particular system library -- anybody relying on that behavior would already be broken when dynamically linking std. That is, there's an outside chance something will fail to link on macOS and iOS because it was accidentally relying on our unnecessary dependency.
(If that *does* happen, that project could be easily fixed by linking libresolv explicitly on those platforms, probably via `#[link(name = "resolv")] extern {}`,` -Crustc-link-lib=resolv`, `println!("cargo:rustc-link-lib=resolv")`, or one of several places in `.config/cargo.toml`)
---
I'm also going to preemptively add the nomination for discussing this in the libs meeting. Basically: Do we care about programs that assume we will bring libraries in that we do not use. `libresolv` and `libiconv` on macOS/iOS are in this camp (`libresolv` because we used to use it, and `libiconv` because the `libc` crate was unintentionally(?) pulling it in to every Rust program).
I'd like to remove them both, but this may cause link issues programs that are relying on `std` to depend on them transitively. (Relying on std for this does not work in all build configurations, so this seems very fragile, and like a use case we should not support).
More generally, IMO we should not guarantee the specific set of system-provided libraries we use (beyond what is implied by an OS version requirement), which means we'd be free to remove this cruft.
kmc-solid: Handle errors returned by `SOLID_FS_ReadDir`
Fixes the issue where the `std::fs::ReadDir` implementaton of the [`*-kmc-solid_*`](https://doc.rust-lang.org/nightly/rustc/platform-support/kmc-solid.html) Tier 3 targets silently suppressed errors returned by the underlying `SOLID_FS_ReadDir` system function. The new implementation correctly handles all cases:
- `SOLID_ERR_NOTFOUND` indicates the end of directory stream.
- `SOLID_ERR_OK` + non-empty `d_name` indicates success.
- Some old filesystem drivers may return `SOLID_ERR_OK` + empty `d_name` to indicate the end of directory stream.
- Any other negative values (per ITRON convention) represent an error.
Eliminate 280-byte memset from ReadDir iterator
This guy:
1536ab1b38/library/std/src/sys/unix/fs.rs (L589)
It turns out `libc::dirent64` is quite big—https://docs.rs/libc/0.2.135/libc/struct.dirent64.html. In #103135 this memset accounted for 0.9% of the runtime of iterating a big directory.
Almost none of the big zeroed value is ever used. We memcpy a tiny prefix (19 bytes) into it, and then read just 9 bytes (`d_ino` and `d_type`) back out. We can read exactly those 9 bytes we need directly from the original entry_ptr instead.
## History
This code got added in #93459 and tweaked in #94272 and #94750.
Prior to #93459, there was no memset but a full 280 bytes were being copied from the entry_ptr.
<table><tr><td>copy 280 bytes</td></tr></table>
This was not legal because not all of those bytes might be initialized, or even allocated, depending on the length of the directory entry's name, leading to a segfault. That PR fixed the segfault by creating a new zeroed dirent64 and copying just the guaranteed initialized prefix into it.
<table><tr><td>memset 280 bytes</td><td>copy 19 bytes</td></tr></table>
However this was still buggy because it used `addr_of!((*entry_ptr).d_name)`, which is considered UB by Miri in the case that the full extent of entry_ptr is not in bounds of the same allocation. (Arguably this shouldn't be a requirement, but here we are.)
The UB got fixed by #94272 by replacing `addr_of` with some pointer manipulation based on `offset_from`, but still fundamentally the same operation.
<table><tr><td>memset 280 bytes</td><td>copy 19 bytes</td></tr></table>
Then #94750 noticed that only 9 of those 19 bytes were even being used, so we could pick out only those 9 to put in the ReadDir value.
<table><tr><td>memset 280 bytes</td><td>copy 19 bytes</td><td>copy 9 bytes</td></tr></table>
After my PR we just grab the 9 needed bytes directly from entry_ptr.
<table><tr><td>copy 9 bytes</td></tr></table>
The resulting code is more complex but I believe still worthwhile to land for the following reason. This is an extremely straightforward thing to accomplish in C and clearly libc assumes that; literally just `entry_ptr->d_name`. The extra work in comparison to accomplish it in Rust is not an example of any actual safety being provided by Rust. I believe it's useful to have uncovered that and think about what could be done in the standard library or language to support this obvious operation better.
## References
- https://man7.org/linux/man-pages/man3/readdir.3.html
Reduce false positives in msys2 detection
Currently msys2 will be detected by getting the file path and looking to see if it contains the substrings "msys-" and "-ptr" (or "cygwin-" and "-pty"). This risks false positives, especially with filesystem files and if `GetFileInformationByHandleEx` returns a [full path](https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/nf-ntifs-ntqueryinformationfile#remarks).
This PR adds a check to see if the handle is a pipe before doing the substring search. Additionally, for "msys2-" or "cygwin-" it only checks if the file name starts with the substring rather than looking at the whole path.
These targets have system limits on the thread names, 16 and 64 bytes
respectively, and `pthread_setname_np` returns an error if the name is
longer. However, we're not in a context that can propagate errors when
we call this, and we used to implicitly truncate on Linux with `prctl`,
so now we manually truncate these names ahead of time.
Previously, the signal mask is always reset when a child process is
started. This breaks tools like `nohup` which expect `SIGHUP` to be
blocked.
With this change, the default behavior changes to inherit the signal mask.
This also changes the signal disposition for `SIGPIPE` to only be
changed if the `#[unix_sigpipe]` attribute isn't set.
std: use `sync::Mutex` for internal statics
Since `sync::Mutex` is now `const`-constructible, it can be used for internal statics, removing the need for `sys_common::StaticMutex`. This adds some extra allocations on platforms which need to box their mutexes (currently SGX and some UNIX), but these will become unnecessary with the lock improvements tracked in #93740.
I changed the program argument implementation on Hermit, it does not need `Mutex` but can use atomics like some UNIX systems (ping `@mkroening` `@stlankes).`
Use semaphores for thread parking on Apple platforms
Currently we use a mutex-condvar pair for thread parking on Apple systems. Unfortunately, `pthread_cond_timedwait` uses the real-time clock for measuring time, which causes problems when the system time changes. The parking implementation in this PR uses a semaphore instead, which measures monotonic time by default, avoiding these issues. As a further benefit, this has the potential to improve performance a bit, since `unpark` does not need to wait for a lock to be released.
Since the Mach semaphores are poorly documented (I could not find availability or stability guarantees for instance), this uses a [dispatch semaphore](https://developer.apple.com/documentation/dispatch/dispatch_semaphore?language=objc) instead. While it adds a layer of indirection (it uses Mach semaphores internally), the overhead is probably negligible.
Tested on macOS 12.5.
r? ``````@thomcc``````
Add `IsTerminal` trait to determine if a descriptor or handle is a terminal
The UNIX implementation uses `isatty`. The Windows implementation uses
the same logic the `atty` crate uses, including the hack needed to
detect msys terminals.
Implement this trait for `Stdin`/`Stdout`/`Stderr`/`File` on all
platforms. On Unix, implement it for `BorrowedFd`/`OwnedFd`. On Windows,
implement it for `BorrowedHandle`/`OwnedHandle`.
Based on https://github.com/rust-lang/rust/pull/91121
Co-authored-by: Matt Wilkinson <mattwilki17@gmail.com>
Rather than referencing a slice's pointer and then creating a new slice
with a longer length, offset from the base structure pointer instead.
This makes some choices of Rust semantics happier.
The UNIX and WASI implementations use `isatty`. The Windows
implementation uses the same logic the `atty` crate uses, including the
hack needed to detect msys terminals.
Implement this trait for `File` and for `Stdin`/`Stdout`/`Stderr` and
their locked counterparts on all platforms. On UNIX and WASI, implement
it for `BorrowedFd`/`OwnedFd`. On Windows, implement it for
`BorrowedHandle`/`OwnedHandle`.
Based on https://github.com/rust-lang/rust/pull/91121
Co-authored-by: Matt Wilkinson <mattwilki17@gmail.com>
sync thread_local key conditions exactly with what the macro uses
This makes the `cfg` in `mod.rs` syntactically the same as those in `local.rs`.
I don't think this should actually change anything, but seems better to be consistent?
I looked into this due to https://github.com/rust-lang/rust/issues/102549, but this PR would make it *less* likely that `__OsLocalKeyInner` is going to get provided, so this cannot help with that issue.
r? `@thomcc`
More dupe word typos
I only picked those changes (from the regex search) that I am pretty certain doesn't change meaning and is just a typo fix. Do correct me if any fix is undesirable and I can revert those. Thanks.
impl AsFd and AsRawFd for io::{Stdin, Stdout, Stderr}, not the sys versions
https://github.com/rust-lang/rust/pull/100892 implemented AsFd for the
sys versions, rather than for the public types. Change the
implementations to apply to the public types.
openbsd: don't reallocate a guard page on the stack.
the kernel currently enforce that a stack is immutable. calling mmap(2) or mprotect(2) to change it will result in EPERM, which generate a panic!().
so just do like for Linux, and trust the kernel to do the right thing.
Optimize TLS on Windows
This implements the suggestion in the current TLS code to embed the linked list of destructors in the `StaticKey` structure to save allocations. Additionally, locking is avoided when no destructor needs to be run. By using one Windows-provided `Once` per key instead of a global lock, locking is more finely-grained (this unblocks #100579).
Allow compiling the `wasm32-wasi` std library with atomics
The issue #102157 demonstrates how currently the `-Z build-std` option will fail when re-compiling the standard library with `RUSTFLAGS` like `RUSTFLAGS="-C target-feature=+atomics,+bulk-memory -C link-args=--shared-memory"`. This change attempts to resolve those build issues by depending on the the WebAssembly `futex` module and providing an implementation for `env_lock`. Fixes#102157.
fs::get_path solarish version.
similar to linux, albeit there is no /proc/self notion on solaris
based system thus flattening the difference for simplification sake.
Copying the approach of the Unix target, this change uses the standard
`RwLock` to protect against concurrent access of libc's environment.
This locking is only enabled when WebAssembly's `atomics` feature is
also enabled.
The issue #102157 demonstrates how currently the `-Z build-std` option
will fail when re-compiling the standard library with `RUSTFLAGS` like
`RUSTFLAGS="-C target-feature=+atomics,+bulk-memory -C
link-args=--shared-memory"`. This change attempts to resolve those build
issues by depending on the the WebAssembly `futex` module and providing
an implementation for `env_lock`. Fixes#102157.
Rollup of 8 pull requests
Successful merges:
- #101118 (fs::get_mode enable getting the data via fcntl/F_GETFL on major BSD)
- #102072 (Add `ptr::Alignment` type)
- #102799 (rustdoc: remove hover gap in file picker)
- #102820 (Show let-else suggestion on stable.)
- #102829 (rename `ImplItemKind::TyAlias` to `ImplItemKind::Type`)
- #102831 (Don't use unnormalized type in `Ty::fn_sig` call in rustdoc `clean_middle_ty`)
- #102834 (Remove unnecessary `lift`/`lift_to_tcx` calls from rustdoc)
- #102838 (remove cfg(bootstrap) from Miri)
Failed merges:
r? `@ghost`
`@rustbot` modify labels: rollup
the kernel currently enforce that a stack is immutable. calling mmap(2) or
mprotect(2) to change it will result in EPERM, which generate a panic!().
so just do like for Linux, and trust the kernel to do the right thing.
Reduce CString allocations in std as much as possible
Currently, every operation involving paths in `fs` allocates memory to hold the path before sending it through the syscall. This PR instead uses a stack allocation (chosen size is somewhat arbitrary) when the path is short before falling back to heap allocations for long paths.
Benchmarks show that the stack allocation is ~2x faster for short paths:
```
test sys::unix::fd::tests::bench_heap_path_alloc ... bench: 34 ns/iter (+/- 2)
test sys::unix::fd::tests::bench_stack_path_alloc ... bench: 15 ns/iter (+/- 1)
```
For long paths, I couldn't find any measurable difference.
---
I'd be surprised if I was the first to think of this, so I didn't fully flush out the PR. If this change is desirable, I'll make use of `run_with_cstr` across all platforms in every fs method (currently just unix open for testing). I also added an `impl From<FromBytesWithNulError>` which is presumably a no-no (or at least needs to be done in another PR).
---
Also see https://github.com/nix-rust/nix/pull/1655 with a bunch of discussion where I'm doing something similar.
Add `AsFd` implementations for stdio lock types on WASI.
This mirrors the implementations on Unix platforms, and also mirrors the existing `AsRawFd` impls.
This is similar to #100892, but is for the `*Lock` types.
Recover error strings on Unix from_lossy_utf8
Some language settings can result in unreliable UTF-8 being produced.
This can result in failing to emit the error string, panicking instead.
from_lossy_utf8 allows us to assume these strings usually will be fine.
This fixes rust-lang#99535.
make Condvar, Mutex, RwLock const constructors work with the `unsupported` impl
applying this patch locally to the `rust-src` component fixes#98378
however, the solution seems wrong to me because PR #97791 didn't add any `rustc_const_stable` attribute to underlying implementations like `std::sys::unix::futex`, so I must be missing something about how const-stability is checked ... maybe the `restricted_std` feature (gate?) has an effect?
fixes#98378fixes#98293 (probably)
Refactor some `std` code that works with pointer offstes
This PR replaces `pointer::offset` in standard library with `pointer::add` and `pointer::sub`, [re]moving some casts and using `.addr()` while we are at it.
This is a more complicated refactor than all other sibling PRs, so take a closer look when reviewing, please 😃 (though I've checked this multiple times and it looks fine).
r? ````@scottmcm````
_split off from #100746, continuation of #100822_
Remove `RtlGenRandom` (take two)
First try to use the system preferred RNG but if that fails (e.g. due to a broken system configuration) then fallback to manually opening an algorithm handle.
This mirrors the implementations on Unix platforms, and also mirrors the
existing `AsRawFd` impls.
This is similar to #100892, but is for the `*Lock` types.
Remove use of `io::ErrorKind::Other` in std
The documentation states that this `ErrorKind` is not used by the standard library. Instead, `io::ErrorKind::Uncategorized` should be used.
The two instances are in the unstable API [linux_pidfd](https://github.com/rust-lang/rust/issues/82971).
std: use `sync::RwLock` for internal statics
Since `sync::RwLock` is now `const`-constructible, it can be used for internal statics, removing the need for `sys_common::StaticRwLock`. This adds some extra allocations on platforms which need to box their locks (currently SGX and some UNIX), but these will become unnecessary with the lock improvements tracked in #93740.
First try to use the system preferred RNG but if that fails (e.g. due to a broken system configuration) then fallback to manually opening an algorithm handle.
Check if TCS is a null pointer on SGX
The `EENTER` instruction only checks if the TCS is aligned, not if it zero. Saying the address returned is a `NonNull<u8>` (for which `Tcs` is a type alias) is unsound. As well-behaved runners will not put the TCS at address zero, so the definition of `Tcs` is correct. However, `std` should check the address before casting it to a `NonNull`.
ping `@jethrogb` `@raoulstrackx`
`@rustbot` label I-unsound
Optimize thread parking on NetBSD
As the futex syscall is not present in the latest stable release, NetBSD cannot use the efficient thread parker and locks Linux uses. Currently, it therefore relies on a pthread-based parker, consisting of a mutex and semaphore which protect a state variable. NetBSD however has more efficient syscalls available: [`_lwp_park`](https://man.netbsd.org/_lwp_park.2) and [`_lwp_unpark`](https://man.netbsd.org/_lwp_unpark.2). These already provide the exact semantics of `thread::park` and `Thread::unpark`, but work with thread ids. In `std`, this ID is here stored in an atomic state variable, which is also used to optimize cases were the parking token is already available at the time `thread::park` is called.
r? `@m-ou-se`
Rollup of 5 pull requests
Successful merges:
- #101366 (Restore old behaviour on broken UNC paths)
- #101492 (Suggest adding array lengths to references to arrays if possible)
- #101529 (Fix the example code and doctest for Formatter::sign_plus)
- #101573 (update `ParamKindOrd`)
- #101612 (Fix code generation of `Rvalue::Repeat` with 128 bit values)
Failed merges:
r? `@ghost`
`@rustbot` modify labels: rollup
Restore old behaviour on broken UNC paths
This fixes#101358 by restoring the behaviour from previous stable Rust versions. I'm not convinced this is ultimately right but I think it's less wrong and maybe this should be backported to beta?
r? libs
Open a BCrypt algorithm handle
Fixes#101474, supplants #101456.
Replaces use of a pseduo handle with manually opening a algorithm handle.
Most interesting thing here is the atomics.
r? `@thomcc`
Make `ReentrantMutex` movable and `const`
As `MovableMutex` is now `const`, it can be used to simplify the implementation and interface of the internal reentrant mutex type. Consequently, the standard error stream does not need to be wrapped in `OnceLock` and `OnceLock::get_or_init_pin()` can be removed.
Windows RNG: Use `BCRYPT_RNG_ALG_HANDLE` by default
This only changes a small amount of actual code, the rest is documentation outlining the history of this module as I feel it will be relevant to any future issues that might crop up.
The code change is to use the `BCRYPT_RNG_ALG_HANDLE` [pseudo-handle](https://docs.microsoft.com/en-us/windows/win32/seccng/cng-algorithm-pseudo-handles) by default, which simply uses the default RNG. Previously we used `BCRYPT_USE_SYSTEM_PREFERRED_RNG` which has to load the system configuration and then find and load that RNG. I suspect this was the cause of failures on some systems (e.g. due to corrupted config). However, this is admittedly speculation as I can't reproduce the issue myself (and it does seem quite rare even in the wild). Still, removing a possible point of failure is likely worthwhile in any case.
r? libs
Support `#[unix_sigpipe = "inherit|sig_dfl"]` on `fn main()` to prevent ignoring `SIGPIPE`
When enabled, programs don't have to explicitly handle `ErrorKind::BrokenPipe` any longer. Currently, the program
```rust
fn main() { loop { println!("hello world"); } }
```
will print an error if used with a short-lived pipe, e.g.
% ./main | head -n 1
hello world
thread 'main' panicked at 'failed printing to stdout: Broken pipe (os error 32)', library/std/src/io/stdio.rs:1016:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
by enabling `#[unix_sigpipe = "sig_dfl"]` like this
```rust
#![feature(unix_sigpipe)]
#[unix_sigpipe = "sig_dfl"]
fn main() { loop { println!("hello world"); } }
```
there is no error, because `SIGPIPE` will not be ignored and thus the program will be killed appropriately:
% ./main | head -n 1
hello world
The current libstd behaviour of ignoring `SIGPIPE` before `fn main()` can be explicitly requested by using `#[unix_sigpipe = "sig_ign"]`.
With `#[unix_sigpipe = "inherit"]`, no change at all is made to `SIGPIPE`, which typically means the behaviour will be the same as `#[unix_sigpipe = "sig_dfl"]`.
See https://github.com/rust-lang/rust/issues/62569 and referenced issues for discussions regarding the `SIGPIPE` problem itself
See the [this](https://rust-lang.zulipchat.com/#narrow/stream/219381-t-libs/topic/Proposal.3A.20First.20step.20towards.20solving.20the.20SIGPIPE.20problem) Zulip topic for more discussions, including about this PR.
Tracking issue: https://github.com/rust-lang/rust/issues/97889
Fix a bunch of typo
This PR will fix some typos detected by [typos].
I only picked the ones I was sure were spelling errors to fix, mostly in
the comments.
[typos]: https://github.com/crate-ci/typos
This PR will fix some typos detected by [typos].
I only picked the ones I was sure were spelling errors to fix, mostly in
the comments.
[typos]: https://github.com/crate-ci/typos
Fix UB from misalignment and provenance widening in `std::sys::windows`
This fixes two types of UB:
1. Reading past the end of a reference in types like `&c::REPARSE_DATA_BUFFER` (see https://github.com/rust-lang/unsafe-code-guidelines/issues/256). This is fixed by using `addr_of!`. I think there are probably a couple more cases where we do this for other structures, and will look into it in a bit.
2. Failing to ensure that a `[u8; N]` on the stack is sufficiently aligned to convert to a `REPARSE_DATA_BUFFER`. ~~This was done by introducing a new `AlignedAs` struct that allows aligning one type to the alignment of another type. I expect there are other places where we have this issue too, or I wouldn't introduce this type, but will get to them after this lands.~~
~~Worth noting, it *is* implemented in a way that can cause problems depending on how we fix#81996, but this would be caught by the test I added (and presumably if we decide to fix that in a way that would break this code, we'd also introduce a `#[repr(simple)]` or `#[repr(linear)]` as a replacement for this usage of `#[repr(C)]`).~~
Edit: None of that is still in the code, I just went with a `Align8` since that's all we'll need for almost everything we want to call.
These are more or less "potential UB" since it's likely at the moment everything works fine, although the alignment not causing issues might just be down to luck (and x86 being forgiving).
~~NB: I've only ensured this check builds, but will run tests soon.~~ All tests pass, including stage2 compiler tests.
r? ``@ChrisDenton``
Use getentropy when possible on all Apple platforms
As the current code comments say, `SecRandomCopyBytes` is very heavyweight (regardless of purpose) compared to just asking the kernel directly for bytes from its own CSPRNG. We were not previously making an attempt to use the more efficient `getentropy` call on other Apple targets, instead solely using it on macOS. As the function is available on newer versions of Apple's different OSes, this changes the random filling to always attempt it first everywhere, only falling back to the less ideal alternatives after. This also cleans up the multiple Apple `imp` blocks into one.
It also should give a perf improvement, even if its likely unnoticeably small.
Refed XCode header for `getentropy` in the SDK:
```h
int getentropy(void* buffer, size_t size) __OSX_AVAILABLE(10.12) __IOS_AVAILABLE(10.0) __TVOS_AVAILABLE(10.0) __WATCHOS_AVAILABLE(3.0);
```
r? ``@thomcc``
Reinstate preloading of some dll imports
I've now come around to the conclusion that there is a justification for pre-loading the synchronization functions `WaitOnAddress` and `WakeByAddressSingle`. I've found this to have a particularly impact in testing frameworks that may have short lived processes which immediately spawn lots of threads.
Also, because pre-main initializers imply a single-threaded environment, we can switch back to using relaxed atomics which might be a minor perf improvement on some platforms (though I doubt it's particularly notable).
r? ``@Mark-Simulacrum`` and sorry for the churn here.
For convenience I'll summarise previous issues with preloading and the solutions that are included in this PR (if any):
**Issue:** User pre-main initializers may be run before std's
**Solution:** The std now uses initializers that are guaranteed to run earlier than the old initializers. A note is also added that users should not copy std's behaviour if they want to ensure they run their initializers after std.
**Issue:** Miri does not understand pre-main initializers.
**Solution:** For miri only, run the function loading lazily instead.
**Issue:** We should ideally use `LoadLibrary` to get "api-ms-win-core-synch-l1-2-0". Only "ntdll" and "kernel32" are guaranteed to always be loaded.
**Solution:** None. We can't use `LoadLibrary` pre-main. However, in the past `GetModuleHandle` has always worked in practice so this should hopefully not be a problem.
If/when Windows 7 support is dropped, we can finally remove all this for good and just use normal imports.
Avoid zeroing large stack buffers in stdio on Windows
Does what it says on the tin, using `[MaybeUninit<u16>; N]` instead of `[0u16; N]`. These buffers seem to be around 8kb, which is big enough that this is likely to be a very nice perf boost to stdio-heavy windows code.
r? ``@ChrisDenton``
*(Note: this PR also has a commit that adds windows to CI, but as it mentions I'll revert that after it comes out green -- I can only do a check build on the machine I'm typing this on)*
Use posix_spawn for absolute paths on macOS
Currently, on macOS, Rust never uses the fast posix_spawn path if a
directory change is requested, due to a bug in Apple's libc. However, the
bug is only triggered if the program is a relative path.
This PR makes it so that the fast path continues to work if the program
is an absolute path or a lone filename.
This was an alternative proposed in https://github.com/rust-lang/rust/pull/80537#issue-776674009, and it makes a measurable performance difference in some of my code that spawns thousands of processes.
Currently, on macOS, Rust never uses the fast posix_spawn path if a
directory change is requested due to a bug in Apple's libc. However, the
bug is only triggered if the program is a relative path.
This PR makes it so that the fast path continues to work if the program
is an absolute path or a lone filename.
This was an alternative proposed in
https://github.com/rust-lang/rust/pull/80537#issue-776674009, and it
makes a measurable performance difference in some of my code that spawns
thousands of processes.
This makes it possible to instruct libstd to never touch the signal
handler for `SIGPIPE`, which makes programs pipeable by default (e.g.
with `./your-program | head -n 1`) without `ErrorKind::BrokenPipe`
errors.
std::io: migrate ReadBuf to BorrowBuf/BorrowCursor
This PR replaces `ReadBuf` (used by the `Read::read_buf` family of methods) with `BorrowBuf` and `BorrowCursor`.
The general idea is to split `ReadBuf` because its API is large and confusing. `BorrowBuf` represents a borrowed buffer which is mostly read-only and (other than for construction) deals only with filled vs unfilled segments. a `BorrowCursor` is a mostly write-only view of the unfilled part of a `BorrowBuf` which distinguishes between initialized and uninitialized segments. For `Read::read_buf`, the caller would create a `BorrowBuf`, then pass a `BorrowCursor` to `read_buf`.
In addition to the major API split, I've made the following smaller changes:
* Removed some methods entirely from the API (mostly the functionality can be replicated with two calls rather than a single one)
* Unified naming, e.g., by replacing initialized with init and assume_init with set_init
* Added an easy way to get the number of bytes written to a cursor (`written` method)
As well as simplifying the API (IMO), this approach has the following advantages:
* Since we pass the cursor by value, we remove the 'unsoundness footgun' where a malicious `read_buf` could swap out the `ReadBuf`.
* Since `read_buf` cannot write into the filled part of the buffer, we prevent the filled part shrinking or changing which could cause underflow for the caller or unexpected behaviour.
## Outline
```rust
pub struct BorrowBuf<'a>
impl Debug for BorrowBuf<'_>
impl<'a> From<&'a mut [u8]> for BorrowBuf<'a>
impl<'a> From<&'a mut [MaybeUninit<u8>]> for BorrowBuf<'a>
impl<'a> BorrowBuf<'a> {
pub fn capacity(&self) -> usize
pub fn len(&self) -> usize
pub fn init_len(&self) -> usize
pub fn filled(&self) -> &[u8]
pub fn unfilled<'this>(&'this mut self) -> BorrowCursor<'this, 'a>
pub fn clear(&mut self) -> &mut Self
pub unsafe fn set_init(&mut self, n: usize) -> &mut Self
}
pub struct BorrowCursor<'buf, 'data>
impl<'buf, 'data> BorrowCursor<'buf, 'data> {
pub fn clone<'this>(&'this mut self) -> BorrowCursor<'this, 'data>
pub fn capacity(&self) -> usize
pub fn written(&self) -> usize
pub fn init_ref(&self) -> &[u8]
pub fn init_mut(&mut self) -> &mut [u8]
pub fn uninit_mut(&mut self) -> &mut [MaybeUninit<u8>]
pub unsafe fn as_mut(&mut self) -> &mut [MaybeUninit<u8>]
pub unsafe fn advance(&mut self, n: usize) -> &mut Self
pub fn ensure_init(&mut self) -> &mut Self
pub unsafe fn set_init(&mut self, n: usize) -> &mut Self
pub fn append(&mut self, buf: &[u8])
}
```
## TODO
* ~~Migrate non-unix libs and tests~~
* ~~Naming~~
* ~~`BorrowBuf` or `BorrowedBuf` or `SliceBuf`? (We might want an owned equivalent for the async IO traits)~~
* ~~Should we rename the `readbuf` module? We might keep the name indicate it includes both the buf and cursor variations and someday the owned version too. Or we could change it. It is not publicly exposed, so it is not that important~~.
* ~~`read_buf` method: we read into the cursor now, so the `_buf` suffix is a bit weird.~~
* ~~Documentation~~
* Tests are incomplete (I adjusted existing tests, but did not add new ones).
cc https://github.com/rust-lang/rust/issues/78485, https://github.com/rust-lang/rust/issues/94741
supersedes: https://github.com/rust-lang/rust/pull/95770, https://github.com/rust-lang/rust/pull/93359fixes#93305
Optimize `Wtf8Buf::into_string` for the case where it contains UTF-8.
Add a `is_known_utf8` flag to `Wtf8Buf`, which tracks whether the
string is known to contain UTF-8. This is efficiently computed in many
common situations, such as when a `Wtf8Buf` is constructed from a `String`
or `&str`, or with `Wtf8Buf::from_wide` which is already doing UTF-16
decoding and already checking for surrogates.
This makes `OsString::into_string` O(1) rather than O(N) on Windows in
common cases.
And, it eliminates the need to scan through the string for surrogates in
`Args::next` and `Vars::next`, because the strings are already being
translated with `Wtf8Buf::from_wide`.
Many things on Windows construct `OsString`s with `Wtf8Buf::from_wide`,
such as `DirEntry::file_name` and `fs::read_link`, so with this patch,
users of those functions can subsequently call `.into_string()` without
paying for an extra scan through the string for surrogates.
r? `@ghost`
Align android `sigaddset` impl with the reference impl from Bionic
In https://github.com/rust-lang/rust/pull/100737 I noticed we were treating the sigset_t as an array of bytes, while referencing code from android (ad8dcd6023/libc/include/android/legacy_signal_inlines.h) which treats it as an array of unsigned long.
That said, the behavior difference is so subtle here that it's not hard to see why nobody noticed. This fixes the implementation to be equivalent to the one in bionic.
Use pointer `is_aligned*` methods
This PR replaces some manual alignment checks with calls to `pointer::{is_aligned, is_aligned_to}` and removes a useless pointer cast.
r? `@scottmcm`
_split off from #100746_
Replace most uses of `pointer::offset` with `add` and `sub`
As PR title says, it replaces `pointer::offset` in compiler and standard library with `pointer::add` and `pointer::sub`. This generally makes code cleaner, easier to grasp and removes (or, well, hides) integer casts.
This is generally trivially correct, `.offset(-constant)` is just `.sub(constant)`, `.offset(usized as isize)` is just `.add(usized)`, etc. However in some cases we need to be careful with signs of things.
r? ````@scottmcm````
_split off from #100746_
add miri-test-libstd support to libstd
- The first commit mirrors what we already have in liballoc.
- The second commit adds some regression tests that only really make sense to be run in Miri, since they rely on Miri's extra checks to detect anything.
- The third commit makes the MPSC tests work in reasonable time in Miri by reducing iteration counts.
- The fourth commit silences some warnings due to code being disabled with `cfg(miri)`
Windows: Load synch functions together
Attempt to load all the required sync functions and fail if any one of them fails.
This fixes a FIXME by going back to optional loading of `WakeByAddressSingle`.
Also reintroduces a macro for optional loading of functions but keeps it separate from the fallback macro rather than having that do two different jobs.
r? `@thomcc`
Expose `Utf8Lossy` as `Utf8Chunks`
This PR changes the feature for `Utf8Lossy` from `str_internals` to `utf8_lossy` and improves the API. This is done to eventually expose the API as stable.
Proposal: rust-lang/libs-team#54
Tracking Issue: #99543
Avoid zeroing a 1kb stack buffer on every call to `std::sys::windows::fill_utf16_buf`
I've also tried to be slightly more careful about integer overflows, although in practice this is likely still not handled ideally.
r? `@ChrisDenton`
Attempt to load all the required sync functions and fail if any one of them fails.
This reintroduces a macro for optional loading of functions but keeps it separate from the fallback macro rather than having that do two different jobs.
Fix HorizonOS regression in FileTimes
The changes in #98246 caused a regression for multiple Newlib-based systems. This is just a fix including HorizonOS to the list of targets which require a workaround.
``@AzureMarker`` ``@ian-h-chamberlain``
r? ``@nagisa``
promote debug_assert to assert when possible and useful
This PR fixed a very old issue https://github.com/rust-lang/rust/issues/94705 to clarify and improve the POSIX error checking, and some of the checks are skipped because can have no benefit, but I'm sure that this can open some interesting discussion.
Fixes https://github.com/rust-lang/rust/issues/94705
cc: `@tavianator`
cc: `@cuviper`
linux: Use `pthread_setname_np` instead of `prctl`
This function is available on Linux since glibc 2.12, musl 1.1.16, and
uClibc 1.0.20. The main advantage over `prctl` is that it properly
represents the pointer argument, rather than a multi-purpose `long`,
so we're better representing strict provenance (#95496).
This function is available on Linux since glibc 2.12, musl 1.1.16, and
uClibc 1.0.20. The main advantage over `prctl` is that it properly
represents the pointer argument, rather than a multi-purpose `long`,
so we're better representing strict provenance (#95496).
Remove Windows function preloading
After `@Mark-Simulacrum` asked me to provide guidance for when optionally imported functions should be preloaded, I realised my justifications were now quite weak. I think the strongest argument that can be made is that it avoids some degree of nondeterminism when calling these functions (in as far as system API calls can be said to be deterministic). However, I don't think that's particularly convincing unless there's a real world use case where it matters. Further discussion with `@thomcc` has strengthened my feeling that preloading isn't really needed.
Note that `WaitOnAddress` needed some adjustment to work without preloading. I opted not to use a macro for this special case as it seemed silly to do so for just one thing (and I don't like macros tbh).
Fix futex module imports on wasm+atomics
The futex modules were rearranged a bit in #98707, which meant that wasm+atomics would no longer compile on nightly. I don’t believe any other targets were impacted by this.
Remove synchronization from Windows `hashmap_random_keys`
Unfortunately using synchronization when generating hashmap keys can prevent it being used in `DllMain`.
~~Fixes #99341~~
kmc-solid: Update `Socket::connect_timeout` to be in line with #78802
Fixes the build failure of the [`*-kmc-solid_*`](https://doc.rust-lang.org/nightly/rustc/platform-support/kmc-solid.html) Tier 3 targets after #78802.
```
error[E0308]: mismatched types
--> library\std\src\sys\solid\net.rs:234:45
|
234 | cvt(netc::connect(self.0.raw(), addrp, len))
| ------------- ^^^^^ expected *-ptr, found union `SocketAddrCRepr`
| |
| arguments to this function are incorrect
|
= note: expected raw pointer `*const sockets::sockaddr`
found union `SocketAddrCRepr`
note: function defined here
--> library\std\src\sys\solid\abi\sockets.rs:173:12
|
173 | pub fn connect(s: c_int, name: *const sockaddr, namelen: socklen_t) -> c_int;
| ^^^^^^^
```
Support setting file accessed/modified timestamps
Add `struct FileTimes` to contain the relevant file timestamps, since
most platforms require setting all of them at once. (This also allows
for future platform-specific extensions such as setting creation time.)
Add `File::set_file_time` to set the timestamps for a `File`.
Implement the `sys` backends for UNIX, macOS (which needs to fall back
to `futimes` before macOS 10.13 because it lacks `futimens`), Windows,
and WASI.
Implement network primitives with ideal Rust layout, not C system layout
This PR is the result of this internals forum thread: https://internals.rust-lang.org/t/why-are-socketaddrv4-socketaddrv6-based-on-low-level-sockaddr-in-6/13321.
Instead of basing `std:::net::{Ipv4Addr, Ipv6Addr, SocketAddrV4, SocketAddrV6}` on system (C) structs, they are encoded in a more optimal and idiomatic Rust way.
This changes the public API of std by introducing structural equality impls for all four types here, which means that `match ipv4addr { SOME_CONSTANT => ... }` will now compile, whereas previously this was an error. No other intentional changes are introduced to public API.
It's possible to observe the current layout of these types (e.g., by pointer casting); most but not all libraries which were found by Crater to do this have had updates issued and affected versions yanked. See report below.
### Benefits of this change
- It will become possible to move these fundamental network types from `std` into `core` ([RFC](https://github.com/rust-lang/rfcs/pull/2832)).
- Some methods that can't be made `const fn`s today can be made `const fn`s with this change.
- `SocketAddrV4` only occupies 6 bytes instead of 16 bytes.
- These simple primitives become easier to read and uses less `unsafe`.
- Makes these types support structural equality, which means you can now (for instance) match an `Ipv4Addr` against a constant
### ~Remaining~ Previous problems
This change obviously changes the memory layout of the types. And it turns out some libraries invalidly assumes the memory layout and does very dangerous pointer casts to convert them. These libraries will have undefined behaviour and perform invalid memory access until patched.
- [x] - `mio` - Issue: https://github.com/tokio-rs/mio/issues/1386.
- [x] `0.7` branch https://github.com/tokio-rs/mio/pull/1388
- [x] `0.7.6` published https://github.com/tokio-rs/mio/pull/1398
- [x] Yank all `0.7` versions older than `0.7.6`
- [x] Report `<0.7.6` to RustSec Advisory Database https://rustsec.org/advisories/RUSTSEC-2020-0081.html
- [x] - `socket2` - Issue: https://github.com/rust-lang/socket2-rs/issues/119.
- [x] `0.3.x` branch https://github.com/rust-lang/socket2-rs/pull/120
- [x] `0.3.16` published
- [x] `master` branch https://github.com/rust-lang/socket2-rs/pull/122
- [x] Yank all `0.3` versions older than `0.3.16`
- [x] Report `<0.3.16` to RustSec Advisory Database https://rustsec.org/advisories/RUSTSEC-2020-0079.html
- [x] - `net2` - Issue: https://github.com/deprecrated/net2-rs/issues/105
- [x] https://github.com/deprecrated/net2-rs/pull/106
- [x] `0.2.36` published
- [x] Yank all `0.2` versions older than `0.2.36`
- [x] Report `<0.2.36` to RustSec Advisory Database https://rustsec.org/advisories/RUSTSEC-2020-0078.html
- [x] - `miow` - Issue: https://github.com/yoshuawuyts/miow/issues/38
- [x] `0.3.x` - https://github.com/yoshuawuyts/miow/pull/39
- [x] `0.3.6` published
- [x] `0.2.x` - https://github.com/yoshuawuyts/miow/pull/40
- [x] `0.2.2` published
- [x] Yanked all `0.2` versions older than `0.2.2`
- [x] Yanked all `0.3` versions older than `0.3.6`
- [x] Report `<0.2.2` and `<0.3.6` to RustSec Advisory Database https://rustsec.org/advisories/RUSTSEC-2020-0080.html
- [x] - `quinn master` (aka what became 0.7) - https://github.com/quinn-rs/quinn/issues/968https://github.com/quinn-rs/quinn/pull/987
- [x] - `quinn 0.6` - https://github.com/quinn-rs/quinn/pull/1045
- [x] - `quinn 0.5` - https://github.com/quinn-rs/quinn/pull/1046
- [x] - Release `0.7.0`, `0.6.2` and `0.5.4`
- [x] - `nb-connect` - https://github.com/smol-rs/nb-connect/issues/1
- [x] - Release `1.0.3`
- [x] - Yank all versions older than `1.0.3`
- [x] - `shadowsocks-rust` - https://github.com/shadowsocks/shadowsocks-rust/issues/462
- [ ] - `rio` - https://github.com/spacejam/rio/issues/44
- [ ] - `seaslug` - https://github.com/spacejam/seaslug/issues/1
#### Fixed crate versions
All crates I have found that assumed the memory layout have been fixed and published. The crates and versions that will continue working even as/if this PR is merged is (please upgrade these to help unblock this PR):
* `net2 0.2.36`
* `socket2 0.3.16`
* `miow 0.2.2`
* `miow 0.3.6`
* `mio 0.7.6`
* `mio 0.6.23` - Never had the invalid assumption itself, but has now been bumped to only allow fixed dependencies (`net2` + `miow`)
* `nb-connect 1.0.3`
* `quinn 0.5.4`
* `quinn 0.6.2`
### Release notes draft
This release changes the memory layout of `Ipv4Addr`, `Ipv6Addr`, `SocketAddrV4` and `SocketAddrV6`. The standard library no longer implements these as the corresponding `libc` structs (`sockaddr_in`, `sockaddr_in6` etc.). This internal representation was never exposed, but some crates relied on it anyway by unsafely transmuting. This change will cause those crates to make invalid memory accesses. Notably `net2 <0.2.36`, `socket2 <0.3.16`, `mio <0.7.6`, `miow <0.3.6` and a few other crates are affected. All known affected crates have been patched and have had fixed versions published over a year ago. If any affected crate is still in your dependency tree, you need to upgrade them before using this version of Rust.
Rewrite Windows `compat_fn` macro
This allows using most delay loaded functions before the init code initializes them. It also only preloads a select few functions, rather than all functions.
This is optimized for the common case where a function is used after already being loaded (or failed to load). The only change in codegen at the call site is to use an atomic load instead of a plain load, which should have negligible or no impact.
I've split the old `compat_fn` macro in two so as not to mix two different use cases. If/when Windows 7 support is dropped `compat_fn_optional` can be removed entirely.
r? rust-lang/libs
This allows using most delay loaded functions before the init code initializes them. It also only preloads a select few functions, rather than all functions.
Co-Authored-By: Mark Rousskov <mark.simulacrum@gmail.com>
kmc-solid: Use `libc::abort` to abort a program
This PR updates the target-specific abort subroutine for the [`*-kmc-solid_*`](https://doc.rust-lang.org/nightly/rustc/platform-support/kmc-solid.html) Tier 3 targets.
The current implementation uses a `hlt` instruction, which is the most direct way to notify a connected debugger but is not the most flexible way. This PR changes it to call the `abort` libc function, making it possible for a system designer to override its behavior as they see fit.
Add cgroupv1 support to available_parallelism
Fixes#97549
My dev machine uses cgroup v2 so I was only able to test that code path. So the v1 code path is written only based on documentation. I could use some help testing that it works on a machine with cgroups v1:
```
$ x.py build --stage 1
# quota.rs
fn main() {
println!("{:?}", std:🧵:available_parallelism());
}
# assuming stage1 is linked in rustup
$ rust +stage1 quota.rs
# spawn a new cgroup scope for the current user
$ sudo systemd-run -p CPUQuota="300%" --uid=$(id -u) -tdS
# should print Ok(3)
$ ./quota
```
If it doesn't work as expected an strace, the contents of `/proc/self/cgroups` and the structure of `/sys/fs/cgroups` would help.
Some language settings can result in unreliable UTF-8 being produced.
This can result in failing to emit the error string, panicking instead.
from_lossy_utf8 allows us to assume these strings usually will be fine.
std: use futex-based locks on Fuchsia
This switches `Condvar` and `RwLock` to the futex-based implementation currently used on Linux and some BSDs. Additionally, `Mutex` now has its own, priority-inheriting implementation based on the mutex in Fuchsia's `libsync`. It differs from the original in that it panics instead of aborting when reentrant locking is detected.
````@rustbot```` ping fuchsia
r? ````@m-ou-se````
stdlib support for Apple WatchOS
This is a follow-up to https://github.com/rust-lang/rust/pull/95243 (Add Apple WatchOS compiler targets) that adds stdlib support for Apple WatchOS.
`@deg4uss3r`
`@nagisa`
Windows: Use `FindFirstFileW` for getting the metadata of locked system files
Fixes#96980
Usually opening a file handle with access set to metadata only will always succeed, even if the file is locked. However some special system files, such as `C:\hiberfil.sys`, are locked by the system in a way that denies even that. So as a fallback we try reading the cached metadata from the directory.
Note that the test is a bit iffy. I don't know if `hiberfil.sys` actually exists in the CI.
r? rust-lang/libs
Add `struct FileTimes` to contain the relevant file timestamps, since
most platforms require setting all of them at once. (This also allows
for future platform-specific extensions such as setting creation time.)
Add `File::set_file_time` to set the timestamps for a `File`.
Implement the `sys` backends for UNIX, macOS (which needs to fall back
to `futimes` before macOS 10.13 because it lacks `futimens`), Windows,
and WASI.
Implement ExitCodeExt for Windows
Fixes#97914
### Motivation:
On Windows it is common for applications to return `HRESULT` (`i32`) or `DWORD` (`u32`) values. These stem from COM based components ([HRESULTS](https://docs.microsoft.com/en-us/windows/win32/api/objbase/nf-objbase-coinitialize)), Win32 errors ([GetLastError](https://docs.microsoft.com/en-us/windows/win32/api/errhandlingapi/nf-errhandlingapi-getlasterror)), GUI applications ([WM_QUIT](https://docs.microsoft.com/en-us/windows/win32/winmsg/wm-quit)) and more. The newly stabilized `ExitCode` provides an excellent fit for propagating these values, because `std::process::exit` does not run deconstructors which can result in errors. However, `ExitCode` currently only implements `From<u8> for ExitCode`, which disallows the full range of `i32`/`u32` values. This pull requests attempts to address that shortcoming by providing windows specific extensions that accept a `u32` value (which covers all possible `HRESULTS` and Win32 errors) analog to [ExitStatusExt::from_raw](https://doc.rust-lang.org/std/os/windows/process/trait.ExitStatusExt.html#tymethod.from_raw).
This was also intended by the original Stabilization https://github.com/rust-lang/rust/pull/93840#issue-1129209143= as pointed out by ``@eggyal`` in https://github.com/rust-lang/rust/issues/97914#issuecomment-1151076755:
> Issues around platform specific representations: We resolved this issue by changing the return type of report from i32 to the opaque type ExitCode. __That way we can change the underlying representation without affecting the API, letting us offer full support for platform specific exit code APIs in the future.__
[Emphasis added]
### API
```rust
/// Windows-specific extensions to [`process::ExitCode`].
///
/// This trait is sealed: it cannot be implemented outside the standard library.
/// This is so that future additional methods are not breaking changes.
#[stable(feature = "windows_process_exit_code_from", since = "1.63.0")]
pub trait ExitCodeExt: Sealed {
/// Creates a new `ExitCode` from the raw underlying `u32` return value of
/// a process.
#[stable(feature = "windows_process_exit_code_from", since = "1.63.0")]
fn from_raw(raw: u32) -> Self;
}
#[stable(feature = "windows_process_exit_code_from", since = "1.63.0")]
impl ExitCodeExt for process::ExitCode {
fn from_raw(raw: u32) -> Self {
process::ExitCode::from_inner(From::from(raw))
}
}
```
### Misc
I apologize in advance if I misplaced any attributes regarding stabilzation, as far as I learned traits are insta-stable so I chose to make them stable. If this is an error, please let me know and I'll correct it. I also added some additional machinery to make it work, analog to [ExitStatus](https://doc.rust-lang.org/std/process/struct.ExitStatus.html#).
EDIT: Proposal: https://github.com/rust-lang/libs-team/issues/48
Usually opening a file handle with access set to metadata only will always succeed, even if the file is locked. However some special system files, such as `C:\hiberfil.sys`, are locked by the system in a way that denies even that. So as a fallback we try reading the cached metadata from the directory.
Remove feature `const_option` from std
This is part of the effort to reduce the number of unstable features used by std. This one is easy as it's only used in one place.
std: use an event-flag-based thread parker on SOLID
`Mutex` and `Condvar` are being replaced by more efficient implementations, which need thread parking themselves (see #93740). Therefore, the generic `Parker` needs to be replaced on all platforms where the new lock implementation will be used, which, after #96393, are SOLID, SGX and Hermit (more PRs coming soon).
SOLID, conforming to the [μITRON specification](http://www.ertl.jp/ITRON/SPEC/FILE/mitron-400e.pdf), has event flags, which are a thread parking primitive very similar to `Parker`. However, they do not make any atomic ordering guarantees (even though those can probably be assumed) and necessitate a system call even when the thread token is already available. Hence, this `Parker`, like the Windows parker, uses an extra atomic state variable.
I future-proofed the code by wrapping the event flag in a `WaitFlag` structure, as both SGX and Hermit can share the Parker implementation, they just have slightly different primitives (SGX uses signals and Hermit has a thread blocking API).
`````@kawadakk````` I assume you are the target maintainer? Could you test this for me?
Mitigate MMIO stale data vulnerability
Intel publicly disclosed the MMIO stale data vulnerability on June 14. To mitigate this vulnerability, compiler changes are required for the `x86_64-fortanix-unknown-sgx` target.
cc: ````@jethrogb````
Windows: Iterative `remove_dir_all`
This will allow better strategies for use of memory and File handles. However, fully taking advantage of that is left to future work.
Note to reviewer: It's probably best to view the `remove_dir_all_recursive` as a new function. The diff is not very helpful (imho).
Add a `is_known_utf8` flag to `Wtf8Buf`, which tracks whether the
string is known to contain UTF-8. This is efficiently computed in many
common situations, such as when a `Wtf8Buf` is constructed from a `String`
or `&str`, or with `Wtf8Buf::from_wide` which is already doing UTF-16
decoding and already checking for surrogates.
This makes `OsString::into_string` O(1) rather than O(N) on Windows in
common cases.
And, it eliminates the need to scan through the string for surrogates in
`Args::next` and `Vars::next`, because the strings are already being
translated with `Wtf8Buf::from_wide`.
Many things on Windows construct `OsString`s with `Wtf8Buf::from_wide`,
such as `DirEntry::file_name` and `fs::read_link`, so with this patch,
users of those functions can subsequently call `.into_string()` without
paying for an extra scan through the string for surrogates.
once cell renamings
This PR does the renamings proposed in https://github.com/rust-lang/rust/issues/74465#issuecomment-1153703128
- Move/rename `lazy::{OnceCell, Lazy}` to `cell::{OnceCell, LazyCell}`
- Move/rename `lazy::{SyncOnceCell, SyncLazy}` to `sync::{OnceLock, LazyLock}`
(I used `Lazy...` instead of `...Lazy` as it seems to be more consistent, easier to pronounce, etc)
```@rustbot``` label +T-libs-api -T-libs
Windows: No panic if function not (yet) available
In some situations (e.g. #97814) it is possible for required functions to be called before they've had a chance to be loaded. Therefore, we make it possible to recover from this situation simply by looking at error codes.
`@rustbot` label +O-windows
Add `#[inline]` to small fns of futex `RwLock`
The important methods like `read` and `write` were already inlined,
which can propagate all the way to inlining in user code, but these
small state functions were left behind as normal calls. They should
almost always be inlined as well, as they're just a few instructions.
Add a `BorrowedFd::try_clone_to_owned` and accompanying documentation
Add a `BorrowedFd::try_clone_to_owned`, which returns a new `OwnedFd` sharing the underlying file description. And similar for `BorrowedHandle` and `BorrowedSocket` on WIndows.
This is similar to the existing `OwnedFd::try_clone`, but it's named differently to reflect that it doesn't return `Result<Self, ...>`. I'm open to suggestions for better names.
Also, extend the `unix::io` documentation to mention that `dup` is permitted on `BorrowedFd`.
This was originally requsted [here](https://github.com/rust-lang/rust/issues/88564#issuecomment-910786081). At the time I wasn't sure whether it was desirable, but it does have uses and it helps clarify the API. The documentation previously didn't rule out using `dup` on a `BorrowedFd`, but the API only offered convenient ways to do it from an `OwnedFd`. With this patch, the API allows one to do `try_clone` on any type where it's permitted.
The important methods like `read` and `write` were already inlined,
which can propagate all the way to inlining in user code, but these
small state functions were left behind as normal calls. They should
almost always be inlined as well, as they're just a few instructions.
STD support for the Nintendo 3DS
Rustc already supports compiling for the Nintendo 3DS using the `armv6k-nintendo-3ds` target (Tier 3). Until now though, only `core` and `alloc` were supported. This PR adds standard library support for the Nintendo 3DS. A notable exclusion is `std::thread` support, which will come in a follow-up PR as it requires more complicated changes.
This has been a joint effort by `@Meziu,` `@ian-h-chamberlain,` myself, and prior work by `@rust3ds` members.
### Background
The Nintendo 3DS (Horizon OS) is a mostly-UNIX looking system, with the caveat that it does not come with a full libc implementation out of the box. On the homebrew side (I'm not under NDA), the libc interface is partially implemented by the [devkitPro](https://devkitpro.org/wiki/devkitPro_pacman) toolchain and a user library like [`libctru`](https://github.com/devkitPro/libctru). This is important because there are [some possible legal barriers](https://github.com/rust-lang/rust/pull/88529#issuecomment-919938396) to linking directly to a library that uses the underlying platform APIs, since they might be considered a trade secret or under NDA.
To get around this, the standard library impl for the 3DS does not directly depend on any platform-level APIs. Instead, it expects standard libc functions to be linked in. The implementation of these libc functions is left to the user. Some functions are provided by the devkitPro toolchain, but in our testing, we used the following to fill in the other functions:
- [`libctru`] - provides more basic APIs, such as `nanosleep`. Linked in by way of [`ctru-sys`](https://github.com/Meziu/ctru-rs/tree/master/ctru-sys).
- [`pthread-3ds`](https://github.com/Meziu/pthread-3ds) - provides pthread APIs for `std::thread`. Implemented using [`libctru`].
- [`linker-fix-3ds`](https://github.com/Meziu/rust-linker-fix-3ds) - fulfills some other missing libc APIs. Implemented using [`libctru`].
For more details, see the `src/doc/rustc/src/platform-support/armv6k-nintendo-3ds.md` file added in this PR.
### Notes
We've already upstreamed changes to the [`libc`] crate to support this PR, as well as the upcoming threading PR. These changes have all been released as of 0.2.121, so we bump the crate version in this PR.
Edit: After some rebases, the version bump has already been merged so it doesn't appear in this PR.
A lot of the changes in this PR are straightforward, and follow in the footsteps of the ESP-IDF target: https://github.com/rust-lang/rust/pull/87666.
The 3DS does not support user space process spawning, so these APIs are unimplemented (similar to ESP-IDF).
[`libctru`]: https://github.com/devkitPro/libctru
[`libc`]: https://github.com/rust-lang/libc
