fix: get the l4re target working again
This is based on work from https://github.com/rust-lang/rust/pull/103966, addressing the review comment by `@m-ou-se` at the time and "fixing" the (probably newly) missing read_buf.
Implement `TryFrom<&OsStr>` for `&str`
Recently when trying to work with `&OsStr` I was surprised to find this `impl` missing.
Since the `to_str` method already existed the actual implementation is fairly non-controversial, except for maybe the choice of the error type. I chose an opaque error here instead of something like `std::str::Utf8Error`, since that would already make a number of assumption about the underlying implementation of `OsStr`.
As this is a trait implementation, it is insta-stable, if I'm not mistaken?
Either way this will need an FCP.
I chose "1.64.0" as the version, since this is unlikely to land before the beta cut-off.
`@rustbot` modify labels: +T-libs-api
API Change Proposal: rust-lang/rust#99031 (accepted)
Avoid unwind across `extern "C"` in `thread_local::fast_local`
This is a minimal fix for #112285, in case we want a simple patch that can be easily to backported if that's desirable.
*(Note: I have another broader cleanup which I've mostly omitted from here to avoid clutter, except for the `Cell` change, which isn't needed to fix UB, but simplifies safety comments).*
The only tier-1 target that this occurs on in a way that seems likely to cause problems in practice linux-gnu, although I believe some folks care about that platform somewhat 😉. I'm unsure how big of an issue this is. I've seen stuff like this behave quite badly, but there's a number of reasons to think this might actually be "fine in practice".
I've hedged my bets and assumed we'll backport this at least to beta but my feeling is that there's not enough evidence this is a problem worth backporting further than that.
### More details
This issue seems to have existed since `thread_local!`'s `const` init functionality was added. It occurs if you have a `const`-initialized thread local for a type that `needs_drop`, the drop panics, and you're on a target with support for static thread locals. In this case, we will end up defining an `extern "C"` function in the user crate rather than in libstd, and because the user crate will not have `#![feature(c_unwind)]` enabled, their panic will not be caught by an auto-inserted abort guard.
In practice, the actual situation where problems are likely[^ub] is somewhat narrower.
On most targets with static thread locals, we manage the TLS dtor list by hand (for reentrancy reasons among others). In these cases, while the users code may panic, we're calling it inside our own `extern "C"` (or `extern "system"`) function, which seems to (at least in practice) catch the panic and convert it to an abort.
However, on a few targets, most notably linux-gnu with recent glibc (but also fuchsia and redox), a tls dtor registration mechanism exists which we can actually use directly, [`__cxa_thread_atexit_impl`](https://github.com/rust-lang/rust/blob/master/library/std/src/sys/unix/thread_local_dtor.rs#L26-L36).
This is the case that seems most likely to be a cause for concern, as now we're passing a function to the system library and panicking out of it in a case where there are may not be Rust frames above it on the call stack (since it's running thread shutdown), and even if there were, it may not be prepared to handle such unwinding. If that's the case, it'd be bad.
Is it? Dunno. The fact that it's a `__cxa_*` function makes me think they probably have considered that the callback could throw but I have no evidence here and it doesn't seem to be written down anywhere, so it's just a guess. (I would not be surprised if someone comes into this thread to tell me how definitely-bad-news it is).
That said, as I said, all this is actually UB! If this isn't a "technically UB but fine in practice", but all bets are off if this is the kind of thing we are telling LLVM about.
[^ub]: This is UB so take that with a grain of salt -- I'm absolutely making assumptions about how the UB will behave "in practice" here, which is almost certainly a mistake.
QNX Neutrino: exponential backoff when fork/spawn needs a retry
Fixes#108594: When retrying, sleep with an exponential duration. When sleep duration is lower than minimum possible sleeping time, yield instead (this will not be often due to the exponential increase of duration).
Minimum possible sleeping time is determined using `libc::clock_getres` but only when spawn/fork failed the first time in a request. This is cached using a LazyLock.
CC `@gh-tr`
r? `@workingjubilee`
`@rustbot` label +O-neutrino
use c literals in compiler and library
Use c literals #108801 in compiler and library
currently blocked on:
* <strike>rustfmt: don't know how to format c literals</strike> nope, nightly one works.
* <strike>bootstrap</strike>
r? `@ghost`
`@rustbot` blocked
This fixes the behavior of sending EOF by pressing Ctrl+Z => Enter in a
windows console.
Previously, that would trip the unpaired surrogate error, whereas now we
correctly detect EOF.
Uplift `clippy::{drop,forget}_{ref,copy}` lints
This PR aims at uplifting the `clippy::drop_ref`, `clippy::drop_copy`, `clippy::forget_ref` and `clippy::forget_copy` lints.
Those lints are/were declared in the correctness category of clippy because they lint on useless and most probably is not what the developer wanted.
## `drop_ref` and `forget_ref`
The `drop_ref` and `forget_ref` lint checks for calls to `std::mem::drop` or `std::mem::forget` with a reference instead of an owned value.
### Example
```rust
let mut lock_guard = mutex.lock();
std::mem::drop(&lock_guard) // Should have been drop(lock_guard), mutex
// still locked
operation_that_requires_mutex_to_be_unlocked();
```
### Explanation
Calling `drop` or `forget` on a reference will only drop the reference itself, which is a no-op. It will not call the `drop` or `forget` method on the underlying referenced value, which is likely what was intended.
## `drop_copy` and `forget_copy`
The `drop_copy` and `forget_copy` lint checks for calls to `std::mem::forget` or `std::mem::drop` with a value that derives the Copy trait.
### Example
```rust
let x: i32 = 42; // i32 implements Copy
std::mem::forget(x) // A copy of x is passed to the function, leaving the
// original unaffected
```
### Explanation
Calling `std::mem::forget` [does nothing for types that implement Copy](https://doc.rust-lang.org/std/mem/fn.drop.html) since the value will be copied and moved into the function on invocation.
-----
Followed the instructions for uplift a clippy describe here: https://github.com/rust-lang/rust/pull/99696#pullrequestreview-1134072751
cc `@m-ou-se` (as T-libs-api leader because the uplifting was discussed in a recent meeting)
Start using `windows sys` for Windows FFI bindings in std
Switch to using windows-sys for FFI. In order to avoid some currently contentious issues, this uses windows-bindgen to generate a smaller set of bindings instead of using the full crate.
Unlike the windows-sys crate, the generated bindings uses `*mut c_void` for handle types instead of `isize`. This to sidestep opsem concerns about mixing pointer types and integers between languages. Note that `SOCKET` remains defined as an integer but instead of being a usize, it's changed to fit the [standard library definition](a41fc00eaf/library/std/src/os/windows/raw.rs (L12-L16)):
```rust
#[cfg(target_pointer_width = "32")]
pub type SOCKET = u32;
#[cfg(target_pointer_width = "64")]
pub type SOCKET = u64;
```
The generated bindings also customizes the `#[link]` imports. I hope to switch to using raw-dylib but I don't want to tie that too closely with the switch to windows-sys.
---
Changes outside of the bindings are, for the most part, fairly minimal (e.g. some differences in `*mut` vs. `*const` or a few types differ). One issue is that our own bindings sometimes mix in higher level types, like `BorrowedHandle`. This is pretty adhoc though.
Fix MXCSR configuration dependent timing
Dependent on the (potentially secret) data some vector instructions operate on, and the content in MXCSR, instruction retirement may be delayed by one cycle. This is a potential side channel.
This PR fixes this vulnerability for the `x86_64-fortanix-unknown-sgx` platform by loading MXCSR with `0x1fbf` through an `xrstor` instruction when the enclave is entered and executing an `lfence` immediately after. Other changes of the MXCSR happen only when the enclave is about to be exited and no vector instructions will be executed before it will actually do so. Users of EDP who change the MXCSR and do wish to defend against this side channel, will need to implement the software mitigation described [here](https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/best-practices/mxcsr-configuration-dependent-timing.html).
cc: `@jethrogb` `@monokles`
Add FreeBSD cpuset support to `std:🧵:available_concurrency`
Use libc::cpuset_getaffinity to determine the CPUs available to the current process.
The existing sysconf and sysctl paths are left as fallback.
Fix `checked_{add,sub}_duration` incorrectly returning `None` when `other` has more than `i64::MAX` seconds
Use `checked_{add,sub}_unsigned` in `checked_{add,sub}_duration` so that the correct result is returned when adding/subtracting durations with more than `i64::MAX` seconds.
avoid duplicating TLS state between test std and realstd
This basically re-lands https://github.com/rust-lang/rust/pull/100201 and https://github.com/rust-lang/rust/pull/106638, which got reverted by https://github.com/rust-lang/rust/pull/110861. This works around 2 Miri limitations:
- Miri doesn't support the magic linker section that our Windows TLS support relies on, and instead knows where in std to find the symbol that stores the thread callback.
- For macOS, Miri only supports at most one destructor to be registered per thread.
The 2nd would not be very hard to fix (though the intended destructor order is unclear); the first would be a lot of work to fix. Neither of these is a problem for regular Rust code, but in the std test suite we have essentially 2 copies of the std code and then these both become issues. To avoid that we have the std test crate import the TLS code from the real std instead of having its own copy.
r? ``````@m-ou-se``````
Replace generic thread parker with explicit no-op parker
With #98391 merged, all platforms supporting threads now have their own parking implementations. Therefore, the generic implementation can be removed. On the remaining platforms (really just WASM without atomics), parking is not supported, so calls to `thread::park` now return instantly, which is [allowed by their API](https://doc.rust-lang.org/nightly/std/thread/fn.park.html). This is a change in behaviour, as spurious wakeups do not currently occur since all platforms guard against them. It is invalid to depend on this, but I'm still going to tag this as libs-api for confirmation.
````@rustbot```` label +T-libs +T-libs-api +A-atomic
r? rust-lang/libs
Some data-independent timing vector instructions may have subtle data-dependent
timing due to MXCSR configuration; dependent on (potentially secret) data
instruction retirement may be delayed by one cycle.
This can be done by simply changing the `\??\` prefix to `\\?\` and then attempting to convert to a user path.
Currently it simply strips off the prefix which could lead to the wrong path being returned (e.g. if it's not a drive path or if the path contains trailing spaces, etc).
Remove `all` in target_thread_local cfg
I think it was left there by mistake after the previous refactoring. I just came across it while rebasing to master.
Make sure the implementation of TcpStream::as_raw_fd is fully inlined
Currently the following function:
```rust
use std::os::fd::{AsRawFd, RawFd};
use std::net::TcpStream;
pub fn as_raw_fd(socket: &TcpStream) -> RawFd {
socket.as_raw_fd()
}
```
Is optimized to the following:
```asm
example::as_raw_fd:
push rax
call qword ptr [rip + <std::net::tcp::TcpStream as std::sys_common::AsInner<std::sys_common::net::TcpStream>>::as_inner@GOTPCREL]
mov rdi, rax
call qword ptr [rip + std::sys_common::net::TcpStream::socket@GOTPCREL]
mov rdi, rax
pop rax
jmp qword ptr [rip + _ZN73_$LT$std..sys..unix..net..Socket$u20$as$u20$std..os..fd..raw..AsRawFd$GT$9as_raw_fd17h633bcf7e481df8bbE@GOTPCREL]
```
I think it would make more sense to inline trivial functions used within `TcpStream::AsRawFd`.
Add 64-bit `time_t` support on 32-bit glibc Linux to `set_times`
Add support to `set_times` for 64-bit `time_t` on 32-bit glibc Linux platforms which have a 32-bit `time_t`. Split from #109773.
Tracking issue: #98245
If opening a directory with `FILE_LIST_DIRECTORY` access fails then we should try opening without requesting that access. We may still be able to delete it if it's empty or a link.
Restructure and rename std thread_local internals to make it less of a maze
Every time I try to work on std's thread local internals, it feels like I'm trying to navigate a confusing maze made of macros, deeply nested modules, and types with multiple names/aliases. Time to clean it up a bit.
This PR:
- Exports `Key` with its own name (`Key`), instead of `__LocalKeyInner`
- Uses `pub macro` to put `__thread_local_inner` into a (unstable, hidden) module, removing `#[macro_export]`, removing it from the crate root.
- Removes the `__` from `__thread_local_inner`.
- Removes a few unnecessary `allow_internal_unstable` features from the macros
- Removes the `libstd_thread_internals` feature. (Merged with `thread_local_internals`.)
- And removes it from the unstable book
- Gets rid of the deeply nested modules for the `Key` definitions (`mod fast` / `mod os` / `mod statik`).
- Turns a `#[cfg]` mess into a single `cfg_if`, now that there's no `#[macro_export]` anymore that breaks with `cfg_if`.
- Simplifies the `cfg_if` conditions to not repeat the conditions.
- Removes useless `normalize-stderr-test`, which were left over from when the `Key` types had different names on different platforms.
- Removes a seemingly unnecessary `realstd` re-export on `cfg(test)`.
This PR changes nothing about the thread local implementation. That's for a later PR. (Which should hopefully be easier once all this stuff is a bit cleaned up.)
Spelling library
Split per https://github.com/rust-lang/rust/pull/110392
I can squash once people are happy w/ the changes. It's really uncommon for large sets of changes to be perfectly acceptable w/o at least some changes.
I probably won't have time to respond until tomorrow or the next day
Fix `std` compilation error for wasi+atomics
Fix https://github.com/rust-lang/rust/issues/109727
It seems that the `unsupported/once.rs` module isn't meant to exist at the same time as the `futex` module, as they have conflicting definitions.
I've solved this by defining the `once` module only if `not(target_feature = "atomics")`.
The `wasm32-unknown-unknown` target [similarly only defines the `once` module if `not(target_feature = "atomics")`](01c4f31927/library/std/src/sys/wasm/mod.rs (L69-L70)).
As show in [this block of code](01c4f31927/library/std/src/sys_common/once/mod.rs (L10-L34)), the `sys::once` module doesn't need to exist if `all(target_arch = "wasm32", target_feature = "atomics")`.
NotFound errors:
* `ERROR_INVALID_DRIVE`: The system cannot find the drive specified
* `ERROR_BAD_NETPATH`: The network path was not found
* `ERROR_BAD_NET_NAME`: The network name cannot be found.
InvalidFilename:
* `ERROR_BAD_PATHNAME`: The specified path is invalid.
linkat() not available in the system headers of Solaris 10
I've installed rustup on x86_64-unknown-linux-gnu and would like to use the target sparcv9-sun-solaris. For this, I have built a gcc from the source code for cross-compiling to sparcv9-sun-solaris2.10 with system headers of Solaris 10.
With the following hello word example:
main.rs:
```rust
fn main() {
println!("Hello, world!");
}
```
I had a compilation error:
```
$ rustc -v --target sparcv9-sun-solaris -C linker=/opt/cross-solaris/gcc730/bin/sparcv9-sun-solaris2.10-gcc main.rs
error: linking with `/opt/cross-solaris/gcc730/bin/sparcv9-sun-solaris2.10-gcc` failed: exit status: 1
|
= note: "/opt/cross-solaris/gcc730/bin/sparcv9-sun-solaris2.10-gcc" "-m64" "/tmp/rustcgebYgj/symbols.o" "main.main.89363361-cgu.0.rcgu.o" "main.main.89363361-cgu.1.rcgu.o" "main.main.89363361-cgu.2.rcgu.o" "main.main.89363361-cgu.3.rcgu.o" "main.main.89363361-cgu.4.rcgu.o" "main.main.89363361-cgu.5.rcgu.o" "main.csypsau9u2r8348.rcgu.o" "-Wl,-z,ignore" "-L" "/home/dlaugt/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/sparcv9-sun-solaris/lib" "-Wl,-Bstatic" "/home/dlaugt/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/sparcv9-sun-solaris/lib/libstd-fa47c8247d587714.rlib" "/home/dlaugt/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/sparcv9-sun-solaris/lib/libpanic_unwind-5c87bbe223e6c2a3.rlib" "/home/dlaugt/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/sparcv9-sun-solaris/lib/libobject-d484934062ff9fbb.rlib" "/home/dlaugt/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/sparcv9-sun-solaris/lib/libmemchr-e8dbd5835abcbf43.rlib" "/home/dlaugt/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/sparcv9-sun-solaris/lib/libaddr2line-909ad09329bde2f9.rlib" "/home/dlaugt/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/sparcv9-sun-solaris/lib/libgimli-4d74a3be929697ac.rlib" "/home/dlaugt/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/sparcv9-sun-solaris/lib/librustc_demangle-47cbe1d7f7271ae1.rlib" "/home/dlaugt/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/sparcv9-sun-solaris/lib/libstd_detect-239fd2d25fb32a00.rlib" "/home/dlaugt/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/sparcv9-sun-solaris/lib/libhashbrown-c4a7ce45fb9dec19.rlib" "/home/dlaugt/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/sparcv9-sun-solaris/lib/libminiz_oxide-fa6bc3d9bfb4e402.rlib" "/home/dlaugt/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/sparcv9-sun-solaris/lib/libadler-419f5a82ddd339a3.rlib" "/home/dlaugt/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/sparcv9-sun-solaris/lib/librustc_std_workspace_alloc-7672b378962c11be.rlib" "/home/dlaugt/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/sparcv9-sun-solaris/lib/libunwind-0f9e07f0a032c000.rlib" "/home/dlaugt/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/sparcv9-sun-solaris/lib/libcfg_if-ede7757c356dfb28.rlib" "/home/dlaugt/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/sparcv9-sun-solaris/lib/liblibc-808d56fbc668148a.rlib" "/home/dlaugt/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/sparcv9-sun-solaris/lib/liballoc-784767fe059ad3fe.rlib" "/home/dlaugt/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/sparcv9-sun-solaris/lib/librustc_std_workspace_core-aa31d7ef0556bbe1.rlib" "/home/dlaugt/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/sparcv9-sun-solaris/lib/libcore-81d07df07db18847.rlib" "/home/dlaugt/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/sparcv9-sun-solaris/lib/libcompiler_builtins-313a510e63006db2.rlib" "-Wl,-Bdynamic" "-lsocket" "-lposix4" "-lpthread" "-lresolv" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lsendfile" "-llgrp" "-L" "/home/dlaugt/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/sparcv9-sun-solaris/lib" "-o" "main" "-nodefaultlibs"
= note: /opt/cross-solaris/gcc730/lib/gcc/sparcv9-sun-solaris2.10/7.3.0/../../../../sparcv9-sun-solaris2.10/bin/ld: warning: -z ignore ignored.
/home/dlaugt/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/sparcv9-sun-solaris/lib/libstd-fa47c8247d587714.rlib(std-fa47c8247d587714.std.5c42d2c1-cgu.0.rcgu.o): In function `std::sys::unix::fs:🔗:h3683dfbfbb4995cb':
/rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/sys/unix/fs.rs:1407: undefined reference to `linkat'
collect2: error: ld returned 1 exit status
= help: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
= note: use the `-l` flag to specify native libraries to link
= note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)
```
linkat() is not available in the system headers of Solaris 10. The hello word example works fine when I build/use rust with this PR change.
don't splice from files into pipes in io::copy
This fixes potential data ordering issue where a write performed after a copy operation could become visible in the copy even though it signaled completion.
I assumed that by not setting `SPLICE_F_MOVE` we would be safe and the kernel would do a copy in kernel space and we could avoid the read-write syscall and copy-to/from-userspace costs. But apparently that flag only makes a difference when splicing from a pipe, but not when splicing into it.
Context: https://lkml.org/lkml/2023/2/9/673
Initial support for loongarch64-unknown-linux-gnu
Hi, We hope to add a new port in rust for LoongArch.
LoongArch intro
LoongArch is a RISC style ISA which is independently designed by Loongson
Technology in China. It is divided into two versions, the 32-bit version (LA32)
and the 64-bit version (LA64). LA64 applications have application-level
backward binary compatibility with LA32 applications. LoongArch is composed of
a basic part (Loongson Base) and an expanded part. The expansion part includes
Loongson Binary Translation (LBT), Loongson VirtualiZation (LVZ), Loongson SIMD
EXtension (LSX) and Loongson Advanced SIMD EXtension(LASX).
Currently the LA464 processor core supports LoongArch ISA and the Loongson
3A5000 processor integrates 4 64-bit LA464 cores. LA464 is a four-issue 64-bit
high-performance processor core. It can be used as a single core for high-end
embedded and desktop applications, or as a basic processor core to form an
on-chip multi-core system for server and high-performance machine applications.
Documentations:
ISA:
https://loongson.github.io/LoongArch-Documentation/LoongArch-Vol1-EN.html
ABI:
https://loongson.github.io/LoongArch-Documentation/LoongArch-ELF-ABI-EN.html
More docs can be found at:
https://loongson.github.io/LoongArch-Documentation/README-EN.html
Since last year, we have locally adapted two versions of rust, rust1.41 and rust1.57, and completed the test locally.
I'm not sure if I'm submitting all the patches at once, so I split up the patches and here's one of the commits
Fix buffer overrun in bootstrap and (test-only) symlink_junction
I don't think these can be hit in practice, due to their inputs being valid paths. It's also not security-sensitive code, but just... bad vibes.
I think this is still not really the right way to do this (in terms of path correctness), but is no worse than it was.
r? `@ChrisDenton`
Use `getentropy()` instead of `/dev/urandom` on Emscripten
`/dev/urandom` is usually available on Emscripten, except when using
the special `NODERAWFS` filesystem backend, which replaces all normal
filesystem access with direct Node.js operations.
Since this filesystem backend directly access the filesystem on the
OS, it is not recommended to depend on `/dev/urandom`, especially
when trying to run the Wasm binary on OSes that are not Unix-based.
This can be considered a non-functional change, since Emscripten
implements `/dev/urandom` in the same way as `getentropy()` when not
linking with `-sNODERAWFS`.
Use random `HashMap` keys on Hermit
Initializing the keys with random data provided by the libOS avoids HashDOS attacks and similar issues.
CC `@stlankes`
Support TLS access into dylibs on Windows
This allows access to `#[thread_local]` in upstream dylibs on Windows by introducing a MIR shim to return the address of the thread local. Accesses that go into an upstream dylib will call the MIR shim to get the address of it.
`convert_tls_rvalues` is introduced in `rustc_codegen_ssa` which rewrites MIR TLS accesses to dummy calls which are replaced with calls to the MIR shims when the dummy calls are lowered to backend calls.
A new `dll_tls_export` target option enables this behavior with a `false` value which is set for Windows platforms.
This fixes https://github.com/rust-lang/rust/issues/84933.
`OsStr` has historically kept its implementation details private out of
concern for locking us into a specific encoding on Windows.
This is an alternative to #95290 which proposed specifying the encoding on Windows. Instead, this
only specifies that for cross-platform code, `OsStr`'s encoding is a superset of UTF-8 and defines
rules for safely interacting with it
At minimum, this can greatly simplify the `os_str_bytes` crate and every
arg parser that interacts with `OsStr` directly (which is most of those
that support invalid UTF-8).
fix typo in the creation of OpenOption for RustyHermit
Due to this typo we have to build a workaround for issue hermitcore/libhermit-rs#191.
RustyHermit is a tier 3 platform and backward compatibility does not have to be guaranteed.
Windows: make `Command` prefer non-verbatim paths
When spawning Commands, the path we use can end up being queried using `env::current_exe` (or the equivalent in other languages). Not all applications handle these paths properly therefore we should have a stronger preference for non-verbatim paths when spawning processes.
Implement read_buf for a few more types
Implement read_buf for TcpStream, Stdin, StdinLock, ChildStdout,
ChildStderr (and internally for AnonPipe, Handle, Socket), so
that it skips buffer initialization.
The other provided methods like read_to_string and read_to_end are
implemented in terms of read_buf and so benefit from the optimization
as well.
This commit also implements read_vectored and is_read_vectored where
applicable.
Distribute libntdll.a with windows-gnu toolchains
This allows the OS loader to load essential functions (e.g. read/write file) at load time instead of lazily doing so at runtime.
r? libs
Due to this typo we have to build a workaround for issue
hermitcore/libhermit-rs#191.
RustyHermit is a tier 3 platform and backward compatibility does
not have to be guaranteed.
`/dev/urandom` is usually available on Emscripten, except when using
the special `NODERAWFS` filesystem backend, which replaces all normal
filesystem access with direct Node.js operations.
Since this filesystem backend directly access the filesystem on the
OS, it is not recommended to depend on `/dev/urandom`, especially
when trying to run the Wasm binary on OSes that are not Unix-based.
This can be considered a non-functional change, since Emscripten
implements `/dev/urandom` in the same way as `getentropy()` when not
linking with `-sNODERAWFS`.
use `as_ptr` to determine the address of atomics
The PR #107736 renamed atomic `as_mut_ptr` to `as_ptr`. Consequently, the futex implementation of the tier-3 platform `RutyHermit` has to use this new interface. In addition, this PR removes also an unused import.
This allows removing all the platform-dependent code from `library/std/src/thread/local.rs` and `library/std/src/thread/mod.rs`
Signed-off-by: Ayush Singh <ayushsingh1325@gmail.com>
Split the __thread_local_inner macro to make it more readable. Also move
everything to crate::sys::common::thread_local.
Signed-off-by: Ayush Singh <ayushsingh1325@gmail.com>
Move __thread_local_inner macro in crate:🧵:local to crate::sys.
Currently, the tidy check does not fail for `library/std/src/thread/local.rs` even though it contains platform specific code. This is beacause target_family did not exist at the time the tidy checks were written [1].
[1]: https://github.com/rust-lang/rust/pull/105861#discussion_r1125841678
Signed-off-by: Ayush Singh <ayushsingh1325@gmail.com>
Implement read_buf for TcpStream, Stdin, StdinLock, ChildStdout,
ChildStderr (and internally for AnonPipe, Handle, Socket), so
that it skips buffer initialization.
The other provided methods like read_to_string and read_to_end are
implemented in terms of read_buf and so benefit from the optimization
as well.
This commit also implements read_vectored and is_read_vectored where
applicable.
Add vectored positioned I/O on Unix
Add methods for vectored I/O with an offset on `File` for `unix` under `#![feature(unix_file_vectored_at)]`.
The new methods are wrappers around `preadv` and `pwritev`.
Tracking issue: #89517
Add support for QNX Neutrino to standard library
This change:
- adds standard library support for QNX Neutrino (7.1).
- upgrades `libc` to version `0.2.139` which supports QNX Neutrino
`@gh-tr`
⚠️ Backtraces on QNX require https://github.com/rust-lang/backtrace-rs/pull/507 which is not yet merged! (But everything else works without these changes) ⚠️
Tested mainly with a x86_64 virtual machine (see qnx-nto.md) and partially with an aarch64 hardware (some tests fail due to constrained resources).
[stdio][windows] Use MBTWC and WCTMB
`MultiByteToWideChar` and `WideCharToMultiByte` are extremely well optimized, and therefore should probably be used when we know we can (specifically in the Windows stdio stuff).
Fixes#107092
add support of RustyHermit's BSD socket layer
RustyHermit is a tier 3 platform and publishes a new kernel interface. The new version supports a common BSD socket layer. By supporting this interface, the implementation of `std` can be harmonized to other operating systems. In `sys_common/mod.rs` we remove only a special case for RustyHermit. All changes are done in the RustyHermit specific directories.
To realize this socket layer, the handling of file descriptors is also harmonized to other operating systems.
RustHermit publishs a new kernel interface and supports
a common BSD socket layer. By supporting this interface,
the implementation can be harmonized to other operating systems.
To realize this socket layer, the handling of file descriptors
is also harmonized to other operating systems.
As reported in sunfishcode/is-terminal#18, there are situations where
`GetFileInformationByHandleEx` can write a file name length that is
longer than the provided buffer. To avoid deferencing memory past the
end of the buffer, use a bounds-checked function to form a slice to
the buffer and handle the out-of-bounds case.
This ports the fix from sunfishcode/is-terminal#19 to std's `is_terminal`
implementation.
Windows: Quote more batch file arguments
Make sure to always quote batch file arguments that contain command prompt special characters.
Additionally add `/d` command line parameter to disable any autorun scripts that may change the way variable expansion works. This makes it more consistent across systems and may help avoid surprises.
## Background Info
[`CreateProcess`](https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessw) with the `lpApplicationName` set can only be used to run `.exe` files and not script files such as `.bat`. However, for historical reasons, we do have special handling so that `.bat` files will be correctly run with `cmd.exe` as the application.
In Windows, command line arguments are passed as a single string (not an array). Applications can parse this string however they like but most follow the standard MSVC C/C++ convention. But `cmd.exe` uses different argument parsing rules to other Windows programs (because it emulates old DOS). This PR aims to help smooth over some of the differences.
r? libs
Make sure to quote batch file arguments that contain command prompt special characters.
Additionally add `/d` command line parameter to disable any commands that may change the way variable expansion works.
Optimize `LazyLock` size
The initialization function was unnecessarily stored separately from the data to be initialized. Since both cannot exist at the same time, a `union` can be used, with the `Once` acting as discriminant. This unfortunately requires some extra methods on `Once` so that `Drop` can be implemented correctly and efficiently.
`@rustbot` label +T-libs +A-atomic
Added another error to be processed in fallback
This pull request addresses the problem of Rust not being able to read file/directory metadata because the current user doesn't have permission to read the file and are thus inaccessible.
One particular example is `System Volume Information`. But any example can be made by having a file/directory, which the current user can't access even though the system does allow to view the metadata, which is handled by the fallback.
The fallback exists to get the metadata but it was limited to one error type. Having added ERROR_ACCESS_DENIED per Chris Denton's suggestion, file/directory properties are now properly read.
Solution suggested by Chris Denton https://github.com/nushell/nushell/issues/6857#issuecomment-1426847135
Use associated items of `char` instead of freestanding items in `core::char`
The associated functions and constants on `char` have been stable since 1.52 and the freestanding items have soft-deprecated since 1.62 (https://github.com/rust-lang/rust/pull/95566). This PR ~~marks them as "deprecated in future", similar to the integer and floating point modules (`core::{i32, f32}` etc)~~ replaces all uses of `core::char::*` with `char::*` to prepare for future deprecation of `core::char::*`.
Stop at the first `NULL` argument when iterating `argv`
Some C commandline parsers (e.g. GLib and Qt) are replacing already handled arguments in `argv` with `NULL` and move them to the end. That means that `argc` might be bigger than the actual number of non-`NULL` pointers in `argv` at this point.
To handle this we simply stop iterating at the first `NULL` argument.
`argv` is also guaranteed to be `NULL`-terminated so any non-`NULL` arguments after the first `NULL` can safely be ignored.
Fixes https://github.com/rust-lang/rust/issues/105999
Use `__wasilibc_get_environ()` to read the environment variable list
from wasi-libc instead of using `environ`. `environ` is a global
variable which effectively requires wasi-libc to initialize the
environment variables eagerly, and `__wasilibc_get_environ()` is
specifically designed to be an alternative that lets wasi-libc
intiailize its environment variables lazily.
This should have the side effect of fixing at least some of the cases
of #107635.
library/std/sys_common: Define MIN_ALIGN for m68k-unknown-linux-gnu
This PR adds the missing definition of MIN_ALIGN for the m68k-unknown-linux target.
Avoid __cxa_thread_atexit_impl on Emscripten
- Fixes https://github.com/rust-lang/rust/issues/91628.
- Fixes https://github.com/emscripten-core/emscripten/issues/15722.
See discussion in both issues.
The TL;DR is that weak linkage causes LLVM to produce broken Wasm, presumably due to pointer mismatch. The code is casting a void pointer to a function pointer with specific signature, but Wasm is very strict about function pointer compatibility, so the resulting code is invalid.
Ideally LLVM should catch this earlier in the process rather than emit invalid Wasm, but it currently doesn't and this is an easy and valid fix, given that Emcripten doesn't have `__cxa_thread_atexit_impl` these days anyway.
Unfortunately, I can't add a regression test as even after looking into this issue for a long time, I couldn't reproduce it with any minimal Rust example, only with extracted LLVM IR or on a large project involving Rust + C++.
Do not use box syntax in `std`
See #94970 and #49733. About half of the `box` instances in `std` do not even need to allocate, the other half can simply be replaced with `Box::new`.
`@rustbot` label +T-libs
r? rust-lang/libs
Remove various double spaces in the libraries.
I was just pretty bothered by this when reading the source for a function, and was suggested to check if this happened elsewhere.
Stop probing for statx unless necessary
As is the current toy program:
fn main() -> std::io::Result<()> {
use std::fs;
let metadata = fs::metadata("foo.txt")?;
assert!(!metadata.is_dir());
Ok(())
}
... observed under strace will issue:
[snip]
statx(0, NULL, AT_STATX_SYNC_AS_STAT, STATX_ALL, NULL) = -1 EFAULT (Bad address) statx(AT_FDCWD, "foo.txt", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=0, ...}) = 0
While statx is not necessarily always present, checking for it can be delayed to the first error condition. Said condition may very well never happen, in which case the check got avoided altogether.
Note this is still suboptimal as there still will be programs issuing it, but bulk of the problem is removed.
Tested by forbidding the syscall for the binary and observing it correctly falls back to newfstatat.
While here tidy up the commentary, in particular by denoting some problems with the current approach.
- Fixes https://github.com/rust-lang/rust/issues/91628.
- Fixes https://github.com/emscripten-core/emscripten/issues/15722.
See discussion in both issues.
The TL;DR is that weak linkage causes LLVM to produce broken Wasm, presumably due to pointer mismatch. The code is casting a void pointer to a function pointer with specific signature, but Wasm is very strict about function pointer compatibility, so the resulting code is invalid.
Ideally LLVM should catch this earlier in the process rather than emit invalid Wasm, but it currently doesn't and this is an easy and valid fix, given that Emcripten doesn't have `__cxa_thread_atexit_impl` these days anyway.
Unfortunately, I can't add a regression test as even after looking into this issue for a long time, I couldn't reproduce it with any minimal Rust example, only with extracted LLVM IR or on a large project involving Rust + C++.
r? @alexcrichton
As is the current toy program:
fn main() -> std::io::Result<()> {
use std::fs;
let metadata = fs::metadata("foo.txt")?;
assert!(!metadata.is_dir());
Ok(())
}
... observed under strace will issue:
[snip]
statx(0, NULL, AT_STATX_SYNC_AS_STAT, STATX_ALL, NULL) = -1 EFAULT (Bad address)
statx(AT_FDCWD, "foo.txt", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=0, ...}) = 0
While statx is not necessarily always present, checking for it can be
delayed to the first error condition. Said condition may very well never
happen, in which case the check got avoided altogether.
Note this is still suboptimal as there still will be programs issuing
it, but bulk of the problem is removed.
Tested by forbidding the syscall for the binary and observing it
correctly falls back to newfstatat.
While here tidy up the commentary, in particular by denoting some
problems with the current approach.
Even where actually running processes is not supported.
Needed for the next commit.
The manual trait implementations now belong on ExitStatusError,
which still can't exist.
Unify id-based thread parking implementations
Multiple platforms currently use thread-id-based parking implementations (NetBSD and SGX[^1]). Even though the strategy does not differ, these are duplicated for each platform, as the id is encoded into an atomic thread variable in different ways for each platform.
Since `park` is only called by one thread, it is possible to move the thread id into a separate field. By ensuring that the field is only written to once, before any other threads access it, these accesses can be unsynchronized, removing any restrictions on the size and niches of the thread id.
This PR also renames the internal `thread_parker` modules to `thread_parking`, as that name now better reflects their contents. I hope this does not add too much reviewing noise.
r? `@m-ou-se`
`@rustbot` label +T-libs
[^1]: SOLID supports this as well, I will switch it over in a follow-up PR.
kmc-solid: Fix memory ordering in thread operations
Fixes two memory ordering issues in the thread state machine (`ThreadInner::lifecycle`) of the [`*-kmc-solid_*`](https://doc.rust-lang.org/nightly/rustc/platform-support/kmc-solid.html) Tier 3 targets.
1. When detaching a thread that is still running (i.e., the owner updates `lifecycle` first, and the child updates it next), the first update did not synchronize-with the second update, resulting in a data race between the first update and the deallocation of `ThreadInner` by the child thread.
2. When joining on a thread, the joiner has to pass its own task ID to the joinee in order to be woken up later, but in doing so, it did not synchronize-with the read operation, creating possible sequences of execution where the joinee wakes up an incorrect or non-existent task.
Both issue are theoretical and most likely have never manifested in practice because of the stronger guarantees provided by the Arm memory model (particularly due to its barrier-based definition). Compiler optimizations could have subverted this, but the inspection of compiled code did not reveal such optimizations taking place.
Bump master bootstrap compiler
This PR bumps the bootstrap compiler to the beta created earlier this week, cherry-picks the stabilization version number updates, and updates the `cfg(bootstrap)`s.
r? `@Mark-Simulacrum`
available_parallelism: Gracefully handle zero value cfs_period_us
There seem to be some scenarios where the cgroup cpu quota field `cpu.cfs_period_us` can contain `0`. This field is used to determine the "amount" of parallelism suggested by the function `std:🧵:available_parallelism`
A zero value of this field cause a panic when `available_parallelism()` is invoked. This issue was detected by the call from binaries built by `cargo test`. I really don't feel like `0` is a good value for `cpu.cfs_period_us`, but I also don't think applications should panic if this value is seen.
This panic started happening with rust 1.64.0.
This case is gracefully handled by other projects which read this information: [num_cpus](e437b9d908/src/linux.rs (L207-L210)), [ninja](https://github.com/ninja-build/ninja/pull/2174/files), [dotnet](c4341d45ac/src/coreclr/pal/src/misc/cgroup.cpp (L481-L483))
Before this change, running `cargo test` in environments configured as described above would trigger this panic:
```
$ RUST_BACKTRACE=1 cargo test
Finished test [unoptimized + debuginfo] target(s) in 3.55s
Running unittests src/main.rs (target/debug/deps/x-9a42e145aca2934d)
thread 'main' panicked at 'attempt to divide by zero', library/std/src/sys/unix/thread.rs:546:70
stack backtrace:
0: rust_begin_unwind
1: core::panicking::panic_fmt
2: core::panicking::panic
3: std::sys::unix:🧵:cgroups::quota
4: std::sys::unix:🧵:available_parallelism
5: std:🧵:available_parallelism
6: test::helpers::concurrency::get_concurrency
7: test::console::run_tests_console
8: test::test_main
9: test::test_main_static
10: x::main
at ./src/main.rs:1:1
11: core::ops::function::FnOnce::call_once
at /tmp/rust-1.64-1.64.0-1/library/core/src/ops/function.rs:248:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
error: test failed, to rerun pass '--bin x'
```
I've tested this change in an environment which has the bad (questionable?) setup and rebuilding the test executable against a fixed std library fixes the panic.
Use correct clock in `park_timeout` on Horizon
Horizon does not support using `CLOCK_MONOTONIC` with condition variables, so use the system time instead.
Some C commandline parsers (e.g. GLib and Qt) are replacing already
handled arguments in `argv` with `NULL` and move them to the end. That
means that `argc` might be bigger than the actual number of non-`NULL`
pointers in `argv` at this point.
To handle this we simply stop iterating at the first `NULL` argument.
`argv` is also guaranteed to be `NULL`-terminated so any non-`NULL`
arguments after the first `NULL` can safely be ignored.
Fixes https://github.com/rust-lang/rust/issues/105999
Use a more efficient `Once` on platforms without threads
The current implementation uses an atomic queue and spins rather than panicking when calling `call_once` recursively. Since concurrency is not supported on platforms like WASM, `Once` can be implemented much more efficiently using just a single non-atomic state variable.
fs: Fix#50619 (again) and add a regression test
Bug #50619 was fixed by adding an end_of_stream flag in #50630.
Unfortunately, that fix only applied to the readdir_r() path. When I
switched Linux to use readdir() in #92778, I inadvertently reintroduced
the bug on that platform. Other platforms that had always used
readdir() were presumably never fixed.
This patch enables end_of_stream for all platforms, and adds a
Linux-specific regression test that should hopefully prevent the bug
from being reintroduced again.
Allow blocking `Command::output`
### Problem
Currently, `Command::output` is internally implemented using `Command::spawn`. This is problematic because some targets (like UEFI) do not actually support multitasking and thus block while the program is executing. This coupling does not make much sense as `Command::output` is supposed to block until the execution is complete anyway and thus does not need to rely on a non-blocking `Child` or any other intermediate.
### Solution
This PR moves the implementation of `Command::output` to `std::sys`. This means targets can choose to implement only `Command::output` without having to implement `Command::spawn`.
### Additional Information
This was originally conceived when working on https://github.com/rust-lang/rust/pull/100316. Currently, the only target I know about that will benefit from this change is UEFI.
This PR can also be used to implement more efficient `Command::output` since the intermediate `Process` is not actually needed anymore, but that is outside the scope of this PR.
Since this is not a public API change, I'm not sure if an RFC is needed or not.
Use more LFS functions.
On Linux, use mmap64, open64, openat64, and sendfile64 in place of their non-LFS counterparts.
This is relevant to #94173.
With these changes (together with rust-lang/backtrace-rs#501), the simple binaries I produce with rustc seem to have no non-LFS functions, so maybe #94173 is fixed. But I can't be sure if I've missed something and maybe some non-LFS functions could sneak in somehow.
Bug #50619 was fixed by adding an end_of_stream flag in #50630.
Unfortunately, that fix only applied to the readdir_r() path. When I
switched Linux to use readdir() in #92778, I inadvertently reintroduced
the bug on that platform. Other platforms that had always used
readdir() were presumably never fixed.
This patch enables end_of_stream for all platforms, and adds a
Linux-specific regression test that should hopefully prevent the bug
from being reintroduced again.
