Port outline-atomics to rust
This has a very long history, summarized in
https://github.com/rust-lang/rust/issues/109064. This port is a very
minimal subset of `aarch64/lse.S` from LLVM's compiler-rt. In
particular, it is missing the following:
1. Any form of runtime dispatch between LL/SC and LSE.
Determining which version of the intrinsics to use
requires one of the following:
i) `getauxval` from glibc. It's unclear whether `compiler_builtins` is
allowed to depend on libc at all, and musl doesn't even support
getauxval. Don't enshrine the requirement "de-facto" by making it
required for outline-atomics.
ii) kernel support. Linux and FreeBSD have limited support, but it
requires an extremely recent kernel version and doesn't work at all under QEMU (https://github.com/rust-lang/rust/issues/109064#issuecomment-1494939904).
Instead, we hard-code LL/SC intrinsics. Users who want LSE support
should use the LLVM compiler-rt (if you're building from source in
rust-lang/rust, make sure you have `src/llvm-project` checked out
locally. the goal is to soon add a new `optimized-compiler-builtins`
option so this is easier to discover).
2. The global `___aarch64_have_lse_atomics` CTOR, required to do runtime
dispatch. Thom Chiviolani has this to say about global CTORs:
> static ctors are problems because we are pretty eager about dead code elim
> in general if you have a module that isnt directly reference we will probably not have its static ctors
> also, while llvm has a super robust way to have a static ctor (theres s special "appending global" to use for c++), we dont use that and just have people make a #[used] static in a special section
> 1. the robust way kinda requires rust knowing that the argument is a static ctor (maybe a #[rustc_static_ctor] attribute). it also would be... finnicky, since on windows we actually care beyond being a static ctor, that we run as part in a specific group of ctors, which means a very specific section (one for TLS and the other for, uh, i dont remember)
> 2. we still actually have to codegen the cgu that isn't referenced. but maybe we could remember that it has that attribute and use that
So while this is possible in theory, it's decidedly non-trivial, and
needs invasive changes to rust itself. In any case, it doesn't matter
until we decide the story around libc.
3. The 16-byte (i128) version of compare_and_swap. This wouldn't be
*too* hard to add, but it would be hard to test. The way I tested the
existing code was not just with unit tests but also by loading it as a
path dependency and running `x test core` - the latter caught several
bugs the unit tests didn't catch (because I originally wrote the tests
wrong). So I am slightly nervous about adding a 16-byte version that is
much more poorly tested than the other intrinsics.
2023-06-25 21:07:00 +00:00
|
|
|
#![feature(decl_macro)] // so we can use pub(super)
|
2025-06-13 05:27:47 +00:00
|
|
|
#![feature(macro_metavar_expr_concat)]
|
2024-04-13 08:32:49 +00:00
|
|
|
#![cfg(all(target_arch = "aarch64", target_os = "linux", not(feature = "no-asm")))]
|
Port outline-atomics to rust
This has a very long history, summarized in
https://github.com/rust-lang/rust/issues/109064. This port is a very
minimal subset of `aarch64/lse.S` from LLVM's compiler-rt. In
particular, it is missing the following:
1. Any form of runtime dispatch between LL/SC and LSE.
Determining which version of the intrinsics to use
requires one of the following:
i) `getauxval` from glibc. It's unclear whether `compiler_builtins` is
allowed to depend on libc at all, and musl doesn't even support
getauxval. Don't enshrine the requirement "de-facto" by making it
required for outline-atomics.
ii) kernel support. Linux and FreeBSD have limited support, but it
requires an extremely recent kernel version and doesn't work at all under QEMU (https://github.com/rust-lang/rust/issues/109064#issuecomment-1494939904).
Instead, we hard-code LL/SC intrinsics. Users who want LSE support
should use the LLVM compiler-rt (if you're building from source in
rust-lang/rust, make sure you have `src/llvm-project` checked out
locally. the goal is to soon add a new `optimized-compiler-builtins`
option so this is easier to discover).
2. The global `___aarch64_have_lse_atomics` CTOR, required to do runtime
dispatch. Thom Chiviolani has this to say about global CTORs:
> static ctors are problems because we are pretty eager about dead code elim
> in general if you have a module that isnt directly reference we will probably not have its static ctors
> also, while llvm has a super robust way to have a static ctor (theres s special "appending global" to use for c++), we dont use that and just have people make a #[used] static in a special section
> 1. the robust way kinda requires rust knowing that the argument is a static ctor (maybe a #[rustc_static_ctor] attribute). it also would be... finnicky, since on windows we actually care beyond being a static ctor, that we run as part in a specific group of ctors, which means a very specific section (one for TLS and the other for, uh, i dont remember)
> 2. we still actually have to codegen the cgu that isn't referenced. but maybe we could remember that it has that attribute and use that
So while this is possible in theory, it's decidedly non-trivial, and
needs invasive changes to rust itself. In any case, it doesn't matter
until we decide the story around libc.
3. The 16-byte (i128) version of compare_and_swap. This wouldn't be
*too* hard to add, but it would be hard to test. The way I tested the
existing code was not just with unit tests but also by loading it as a
path dependency and running `x test core` - the latter caught several
bugs the unit tests didn't catch (because I originally wrote the tests
wrong). So I am slightly nervous about adding a 16-byte version that is
much more poorly tested than the other intrinsics.
2023-06-25 21:07:00 +00:00
|
|
|
|
|
|
|
/// Translate a byte size to a Rust type.
|
|
|
|
macro int_ty {
|
|
|
|
(1) => { i8 },
|
|
|
|
(2) => { i16 },
|
|
|
|
(4) => { i32 },
|
|
|
|
(8) => { i64 },
|
|
|
|
(16) => { i128 }
|
|
|
|
}
|
|
|
|
|
|
|
|
mod cas {
|
|
|
|
pub(super) macro test($_ordering:ident, $bytes:tt, $name:ident) {
|
|
|
|
#[test]
|
|
|
|
fn $name() {
|
2025-03-18 09:52:53 +00:00
|
|
|
builtins_test::fuzz_2(10000, |expected: super::int_ty!($bytes), new| {
|
Port outline-atomics to rust
This has a very long history, summarized in
https://github.com/rust-lang/rust/issues/109064. This port is a very
minimal subset of `aarch64/lse.S` from LLVM's compiler-rt. In
particular, it is missing the following:
1. Any form of runtime dispatch between LL/SC and LSE.
Determining which version of the intrinsics to use
requires one of the following:
i) `getauxval` from glibc. It's unclear whether `compiler_builtins` is
allowed to depend on libc at all, and musl doesn't even support
getauxval. Don't enshrine the requirement "de-facto" by making it
required for outline-atomics.
ii) kernel support. Linux and FreeBSD have limited support, but it
requires an extremely recent kernel version and doesn't work at all under QEMU (https://github.com/rust-lang/rust/issues/109064#issuecomment-1494939904).
Instead, we hard-code LL/SC intrinsics. Users who want LSE support
should use the LLVM compiler-rt (if you're building from source in
rust-lang/rust, make sure you have `src/llvm-project` checked out
locally. the goal is to soon add a new `optimized-compiler-builtins`
option so this is easier to discover).
2. The global `___aarch64_have_lse_atomics` CTOR, required to do runtime
dispatch. Thom Chiviolani has this to say about global CTORs:
> static ctors are problems because we are pretty eager about dead code elim
> in general if you have a module that isnt directly reference we will probably not have its static ctors
> also, while llvm has a super robust way to have a static ctor (theres s special "appending global" to use for c++), we dont use that and just have people make a #[used] static in a special section
> 1. the robust way kinda requires rust knowing that the argument is a static ctor (maybe a #[rustc_static_ctor] attribute). it also would be... finnicky, since on windows we actually care beyond being a static ctor, that we run as part in a specific group of ctors, which means a very specific section (one for TLS and the other for, uh, i dont remember)
> 2. we still actually have to codegen the cgu that isn't referenced. but maybe we could remember that it has that attribute and use that
So while this is possible in theory, it's decidedly non-trivial, and
needs invasive changes to rust itself. In any case, it doesn't matter
until we decide the story around libc.
3. The 16-byte (i128) version of compare_and_swap. This wouldn't be
*too* hard to add, but it would be hard to test. The way I tested the
existing code was not just with unit tests but also by loading it as a
path dependency and running `x test core` - the latter caught several
bugs the unit tests didn't catch (because I originally wrote the tests
wrong). So I am slightly nervous about adding a 16-byte version that is
much more poorly tested than the other intrinsics.
2023-06-25 21:07:00 +00:00
|
|
|
let mut target = expected.wrapping_add(10);
|
|
|
|
assert_eq!(
|
2023-07-02 19:43:25 +00:00
|
|
|
unsafe {
|
|
|
|
compiler_builtins::aarch64_linux::$name::$name(expected, new, &mut target)
|
|
|
|
},
|
Port outline-atomics to rust
This has a very long history, summarized in
https://github.com/rust-lang/rust/issues/109064. This port is a very
minimal subset of `aarch64/lse.S` from LLVM's compiler-rt. In
particular, it is missing the following:
1. Any form of runtime dispatch between LL/SC and LSE.
Determining which version of the intrinsics to use
requires one of the following:
i) `getauxval` from glibc. It's unclear whether `compiler_builtins` is
allowed to depend on libc at all, and musl doesn't even support
getauxval. Don't enshrine the requirement "de-facto" by making it
required for outline-atomics.
ii) kernel support. Linux and FreeBSD have limited support, but it
requires an extremely recent kernel version and doesn't work at all under QEMU (https://github.com/rust-lang/rust/issues/109064#issuecomment-1494939904).
Instead, we hard-code LL/SC intrinsics. Users who want LSE support
should use the LLVM compiler-rt (if you're building from source in
rust-lang/rust, make sure you have `src/llvm-project` checked out
locally. the goal is to soon add a new `optimized-compiler-builtins`
option so this is easier to discover).
2. The global `___aarch64_have_lse_atomics` CTOR, required to do runtime
dispatch. Thom Chiviolani has this to say about global CTORs:
> static ctors are problems because we are pretty eager about dead code elim
> in general if you have a module that isnt directly reference we will probably not have its static ctors
> also, while llvm has a super robust way to have a static ctor (theres s special "appending global" to use for c++), we dont use that and just have people make a #[used] static in a special section
> 1. the robust way kinda requires rust knowing that the argument is a static ctor (maybe a #[rustc_static_ctor] attribute). it also would be... finnicky, since on windows we actually care beyond being a static ctor, that we run as part in a specific group of ctors, which means a very specific section (one for TLS and the other for, uh, i dont remember)
> 2. we still actually have to codegen the cgu that isn't referenced. but maybe we could remember that it has that attribute and use that
So while this is possible in theory, it's decidedly non-trivial, and
needs invasive changes to rust itself. In any case, it doesn't matter
until we decide the story around libc.
3. The 16-byte (i128) version of compare_and_swap. This wouldn't be
*too* hard to add, but it would be hard to test. The way I tested the
existing code was not just with unit tests but also by loading it as a
path dependency and running `x test core` - the latter caught several
bugs the unit tests didn't catch (because I originally wrote the tests
wrong). So I am slightly nervous about adding a 16-byte version that is
much more poorly tested than the other intrinsics.
2023-06-25 21:07:00 +00:00
|
|
|
expected.wrapping_add(10),
|
|
|
|
"return value should always be the previous value",
|
|
|
|
);
|
|
|
|
assert_eq!(
|
|
|
|
target,
|
|
|
|
expected.wrapping_add(10),
|
|
|
|
"shouldn't have changed target"
|
|
|
|
);
|
|
|
|
|
|
|
|
target = expected;
|
|
|
|
assert_eq!(
|
2023-07-02 19:43:25 +00:00
|
|
|
unsafe {
|
|
|
|
compiler_builtins::aarch64_linux::$name::$name(expected, new, &mut target)
|
|
|
|
},
|
Port outline-atomics to rust
This has a very long history, summarized in
https://github.com/rust-lang/rust/issues/109064. This port is a very
minimal subset of `aarch64/lse.S` from LLVM's compiler-rt. In
particular, it is missing the following:
1. Any form of runtime dispatch between LL/SC and LSE.
Determining which version of the intrinsics to use
requires one of the following:
i) `getauxval` from glibc. It's unclear whether `compiler_builtins` is
allowed to depend on libc at all, and musl doesn't even support
getauxval. Don't enshrine the requirement "de-facto" by making it
required for outline-atomics.
ii) kernel support. Linux and FreeBSD have limited support, but it
requires an extremely recent kernel version and doesn't work at all under QEMU (https://github.com/rust-lang/rust/issues/109064#issuecomment-1494939904).
Instead, we hard-code LL/SC intrinsics. Users who want LSE support
should use the LLVM compiler-rt (if you're building from source in
rust-lang/rust, make sure you have `src/llvm-project` checked out
locally. the goal is to soon add a new `optimized-compiler-builtins`
option so this is easier to discover).
2. The global `___aarch64_have_lse_atomics` CTOR, required to do runtime
dispatch. Thom Chiviolani has this to say about global CTORs:
> static ctors are problems because we are pretty eager about dead code elim
> in general if you have a module that isnt directly reference we will probably not have its static ctors
> also, while llvm has a super robust way to have a static ctor (theres s special "appending global" to use for c++), we dont use that and just have people make a #[used] static in a special section
> 1. the robust way kinda requires rust knowing that the argument is a static ctor (maybe a #[rustc_static_ctor] attribute). it also would be... finnicky, since on windows we actually care beyond being a static ctor, that we run as part in a specific group of ctors, which means a very specific section (one for TLS and the other for, uh, i dont remember)
> 2. we still actually have to codegen the cgu that isn't referenced. but maybe we could remember that it has that attribute and use that
So while this is possible in theory, it's decidedly non-trivial, and
needs invasive changes to rust itself. In any case, it doesn't matter
until we decide the story around libc.
3. The 16-byte (i128) version of compare_and_swap. This wouldn't be
*too* hard to add, but it would be hard to test. The way I tested the
existing code was not just with unit tests but also by loading it as a
path dependency and running `x test core` - the latter caught several
bugs the unit tests didn't catch (because I originally wrote the tests
wrong). So I am slightly nervous about adding a 16-byte version that is
much more poorly tested than the other intrinsics.
2023-06-25 21:07:00 +00:00
|
|
|
expected
|
|
|
|
);
|
|
|
|
assert_eq!(target, new, "should have updated target");
|
|
|
|
});
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2023-06-26 13:54:47 +00:00
|
|
|
macro test_cas16($_ordering:ident, $name:ident) {
|
|
|
|
cas::test!($_ordering, 16, $name);
|
|
|
|
}
|
|
|
|
|
Port outline-atomics to rust
This has a very long history, summarized in
https://github.com/rust-lang/rust/issues/109064. This port is a very
minimal subset of `aarch64/lse.S` from LLVM's compiler-rt. In
particular, it is missing the following:
1. Any form of runtime dispatch between LL/SC and LSE.
Determining which version of the intrinsics to use
requires one of the following:
i) `getauxval` from glibc. It's unclear whether `compiler_builtins` is
allowed to depend on libc at all, and musl doesn't even support
getauxval. Don't enshrine the requirement "de-facto" by making it
required for outline-atomics.
ii) kernel support. Linux and FreeBSD have limited support, but it
requires an extremely recent kernel version and doesn't work at all under QEMU (https://github.com/rust-lang/rust/issues/109064#issuecomment-1494939904).
Instead, we hard-code LL/SC intrinsics. Users who want LSE support
should use the LLVM compiler-rt (if you're building from source in
rust-lang/rust, make sure you have `src/llvm-project` checked out
locally. the goal is to soon add a new `optimized-compiler-builtins`
option so this is easier to discover).
2. The global `___aarch64_have_lse_atomics` CTOR, required to do runtime
dispatch. Thom Chiviolani has this to say about global CTORs:
> static ctors are problems because we are pretty eager about dead code elim
> in general if you have a module that isnt directly reference we will probably not have its static ctors
> also, while llvm has a super robust way to have a static ctor (theres s special "appending global" to use for c++), we dont use that and just have people make a #[used] static in a special section
> 1. the robust way kinda requires rust knowing that the argument is a static ctor (maybe a #[rustc_static_ctor] attribute). it also would be... finnicky, since on windows we actually care beyond being a static ctor, that we run as part in a specific group of ctors, which means a very specific section (one for TLS and the other for, uh, i dont remember)
> 2. we still actually have to codegen the cgu that isn't referenced. but maybe we could remember that it has that attribute and use that
So while this is possible in theory, it's decidedly non-trivial, and
needs invasive changes to rust itself. In any case, it doesn't matter
until we decide the story around libc.
3. The 16-byte (i128) version of compare_and_swap. This wouldn't be
*too* hard to add, but it would be hard to test. The way I tested the
existing code was not just with unit tests but also by loading it as a
path dependency and running `x test core` - the latter caught several
bugs the unit tests didn't catch (because I originally wrote the tests
wrong). So I am slightly nervous about adding a 16-byte version that is
much more poorly tested than the other intrinsics.
2023-06-25 21:07:00 +00:00
|
|
|
mod swap {
|
|
|
|
pub(super) macro test($_ordering:ident, $bytes:tt, $name:ident) {
|
|
|
|
#[test]
|
|
|
|
fn $name() {
|
2025-03-18 09:52:53 +00:00
|
|
|
builtins_test::fuzz_2(10000, |left: super::int_ty!($bytes), mut right| {
|
Port outline-atomics to rust
This has a very long history, summarized in
https://github.com/rust-lang/rust/issues/109064. This port is a very
minimal subset of `aarch64/lse.S` from LLVM's compiler-rt. In
particular, it is missing the following:
1. Any form of runtime dispatch between LL/SC and LSE.
Determining which version of the intrinsics to use
requires one of the following:
i) `getauxval` from glibc. It's unclear whether `compiler_builtins` is
allowed to depend on libc at all, and musl doesn't even support
getauxval. Don't enshrine the requirement "de-facto" by making it
required for outline-atomics.
ii) kernel support. Linux and FreeBSD have limited support, but it
requires an extremely recent kernel version and doesn't work at all under QEMU (https://github.com/rust-lang/rust/issues/109064#issuecomment-1494939904).
Instead, we hard-code LL/SC intrinsics. Users who want LSE support
should use the LLVM compiler-rt (if you're building from source in
rust-lang/rust, make sure you have `src/llvm-project` checked out
locally. the goal is to soon add a new `optimized-compiler-builtins`
option so this is easier to discover).
2. The global `___aarch64_have_lse_atomics` CTOR, required to do runtime
dispatch. Thom Chiviolani has this to say about global CTORs:
> static ctors are problems because we are pretty eager about dead code elim
> in general if you have a module that isnt directly reference we will probably not have its static ctors
> also, while llvm has a super robust way to have a static ctor (theres s special "appending global" to use for c++), we dont use that and just have people make a #[used] static in a special section
> 1. the robust way kinda requires rust knowing that the argument is a static ctor (maybe a #[rustc_static_ctor] attribute). it also would be... finnicky, since on windows we actually care beyond being a static ctor, that we run as part in a specific group of ctors, which means a very specific section (one for TLS and the other for, uh, i dont remember)
> 2. we still actually have to codegen the cgu that isn't referenced. but maybe we could remember that it has that attribute and use that
So while this is possible in theory, it's decidedly non-trivial, and
needs invasive changes to rust itself. In any case, it doesn't matter
until we decide the story around libc.
3. The 16-byte (i128) version of compare_and_swap. This wouldn't be
*too* hard to add, but it would be hard to test. The way I tested the
existing code was not just with unit tests but also by loading it as a
path dependency and running `x test core` - the latter caught several
bugs the unit tests didn't catch (because I originally wrote the tests
wrong). So I am slightly nervous about adding a 16-byte version that is
much more poorly tested than the other intrinsics.
2023-06-25 21:07:00 +00:00
|
|
|
let orig_right = right;
|
|
|
|
assert_eq!(
|
2023-07-02 19:43:25 +00:00
|
|
|
unsafe { compiler_builtins::aarch64_linux::$name::$name(left, &mut right) },
|
Port outline-atomics to rust
This has a very long history, summarized in
https://github.com/rust-lang/rust/issues/109064. This port is a very
minimal subset of `aarch64/lse.S` from LLVM's compiler-rt. In
particular, it is missing the following:
1. Any form of runtime dispatch between LL/SC and LSE.
Determining which version of the intrinsics to use
requires one of the following:
i) `getauxval` from glibc. It's unclear whether `compiler_builtins` is
allowed to depend on libc at all, and musl doesn't even support
getauxval. Don't enshrine the requirement "de-facto" by making it
required for outline-atomics.
ii) kernel support. Linux and FreeBSD have limited support, but it
requires an extremely recent kernel version and doesn't work at all under QEMU (https://github.com/rust-lang/rust/issues/109064#issuecomment-1494939904).
Instead, we hard-code LL/SC intrinsics. Users who want LSE support
should use the LLVM compiler-rt (if you're building from source in
rust-lang/rust, make sure you have `src/llvm-project` checked out
locally. the goal is to soon add a new `optimized-compiler-builtins`
option so this is easier to discover).
2. The global `___aarch64_have_lse_atomics` CTOR, required to do runtime
dispatch. Thom Chiviolani has this to say about global CTORs:
> static ctors are problems because we are pretty eager about dead code elim
> in general if you have a module that isnt directly reference we will probably not have its static ctors
> also, while llvm has a super robust way to have a static ctor (theres s special "appending global" to use for c++), we dont use that and just have people make a #[used] static in a special section
> 1. the robust way kinda requires rust knowing that the argument is a static ctor (maybe a #[rustc_static_ctor] attribute). it also would be... finnicky, since on windows we actually care beyond being a static ctor, that we run as part in a specific group of ctors, which means a very specific section (one for TLS and the other for, uh, i dont remember)
> 2. we still actually have to codegen the cgu that isn't referenced. but maybe we could remember that it has that attribute and use that
So while this is possible in theory, it's decidedly non-trivial, and
needs invasive changes to rust itself. In any case, it doesn't matter
until we decide the story around libc.
3. The 16-byte (i128) version of compare_and_swap. This wouldn't be
*too* hard to add, but it would be hard to test. The way I tested the
existing code was not just with unit tests but also by loading it as a
path dependency and running `x test core` - the latter caught several
bugs the unit tests didn't catch (because I originally wrote the tests
wrong). So I am slightly nervous about adding a 16-byte version that is
much more poorly tested than the other intrinsics.
2023-06-25 21:07:00 +00:00
|
|
|
orig_right
|
|
|
|
);
|
|
|
|
assert_eq!(left, right);
|
|
|
|
});
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
macro_rules! test_op {
|
|
|
|
($mod:ident, $( $op:tt )* ) => {
|
|
|
|
mod $mod {
|
|
|
|
pub(super) macro test {
|
|
|
|
($_ordering:ident, $bytes:tt, $name:ident) => {
|
|
|
|
#[test]
|
|
|
|
fn $name() {
|
2025-03-18 09:52:53 +00:00
|
|
|
builtins_test::fuzz_2(10000, |old, val| {
|
Port outline-atomics to rust
This has a very long history, summarized in
https://github.com/rust-lang/rust/issues/109064. This port is a very
minimal subset of `aarch64/lse.S` from LLVM's compiler-rt. In
particular, it is missing the following:
1. Any form of runtime dispatch between LL/SC and LSE.
Determining which version of the intrinsics to use
requires one of the following:
i) `getauxval` from glibc. It's unclear whether `compiler_builtins` is
allowed to depend on libc at all, and musl doesn't even support
getauxval. Don't enshrine the requirement "de-facto" by making it
required for outline-atomics.
ii) kernel support. Linux and FreeBSD have limited support, but it
requires an extremely recent kernel version and doesn't work at all under QEMU (https://github.com/rust-lang/rust/issues/109064#issuecomment-1494939904).
Instead, we hard-code LL/SC intrinsics. Users who want LSE support
should use the LLVM compiler-rt (if you're building from source in
rust-lang/rust, make sure you have `src/llvm-project` checked out
locally. the goal is to soon add a new `optimized-compiler-builtins`
option so this is easier to discover).
2. The global `___aarch64_have_lse_atomics` CTOR, required to do runtime
dispatch. Thom Chiviolani has this to say about global CTORs:
> static ctors are problems because we are pretty eager about dead code elim
> in general if you have a module that isnt directly reference we will probably not have its static ctors
> also, while llvm has a super robust way to have a static ctor (theres s special "appending global" to use for c++), we dont use that and just have people make a #[used] static in a special section
> 1. the robust way kinda requires rust knowing that the argument is a static ctor (maybe a #[rustc_static_ctor] attribute). it also would be... finnicky, since on windows we actually care beyond being a static ctor, that we run as part in a specific group of ctors, which means a very specific section (one for TLS and the other for, uh, i dont remember)
> 2. we still actually have to codegen the cgu that isn't referenced. but maybe we could remember that it has that attribute and use that
So while this is possible in theory, it's decidedly non-trivial, and
needs invasive changes to rust itself. In any case, it doesn't matter
until we decide the story around libc.
3. The 16-byte (i128) version of compare_and_swap. This wouldn't be
*too* hard to add, but it would be hard to test. The way I tested the
existing code was not just with unit tests but also by loading it as a
path dependency and running `x test core` - the latter caught several
bugs the unit tests didn't catch (because I originally wrote the tests
wrong). So I am slightly nervous about adding a 16-byte version that is
much more poorly tested than the other intrinsics.
2023-06-25 21:07:00 +00:00
|
|
|
let mut target = old;
|
|
|
|
let op: fn(super::int_ty!($bytes), super::int_ty!($bytes)) -> _ = $($op)*;
|
|
|
|
let expected = op(old, val);
|
2023-07-02 19:43:25 +00:00
|
|
|
assert_eq!(old, unsafe { compiler_builtins::aarch64_linux::$name::$name(val, &mut target) }, "{} should return original value", stringify!($name));
|
Port outline-atomics to rust
This has a very long history, summarized in
https://github.com/rust-lang/rust/issues/109064. This port is a very
minimal subset of `aarch64/lse.S` from LLVM's compiler-rt. In
particular, it is missing the following:
1. Any form of runtime dispatch between LL/SC and LSE.
Determining which version of the intrinsics to use
requires one of the following:
i) `getauxval` from glibc. It's unclear whether `compiler_builtins` is
allowed to depend on libc at all, and musl doesn't even support
getauxval. Don't enshrine the requirement "de-facto" by making it
required for outline-atomics.
ii) kernel support. Linux and FreeBSD have limited support, but it
requires an extremely recent kernel version and doesn't work at all under QEMU (https://github.com/rust-lang/rust/issues/109064#issuecomment-1494939904).
Instead, we hard-code LL/SC intrinsics. Users who want LSE support
should use the LLVM compiler-rt (if you're building from source in
rust-lang/rust, make sure you have `src/llvm-project` checked out
locally. the goal is to soon add a new `optimized-compiler-builtins`
option so this is easier to discover).
2. The global `___aarch64_have_lse_atomics` CTOR, required to do runtime
dispatch. Thom Chiviolani has this to say about global CTORs:
> static ctors are problems because we are pretty eager about dead code elim
> in general if you have a module that isnt directly reference we will probably not have its static ctors
> also, while llvm has a super robust way to have a static ctor (theres s special "appending global" to use for c++), we dont use that and just have people make a #[used] static in a special section
> 1. the robust way kinda requires rust knowing that the argument is a static ctor (maybe a #[rustc_static_ctor] attribute). it also would be... finnicky, since on windows we actually care beyond being a static ctor, that we run as part in a specific group of ctors, which means a very specific section (one for TLS and the other for, uh, i dont remember)
> 2. we still actually have to codegen the cgu that isn't referenced. but maybe we could remember that it has that attribute and use that
So while this is possible in theory, it's decidedly non-trivial, and
needs invasive changes to rust itself. In any case, it doesn't matter
until we decide the story around libc.
3. The 16-byte (i128) version of compare_and_swap. This wouldn't be
*too* hard to add, but it would be hard to test. The way I tested the
existing code was not just with unit tests but also by loading it as a
path dependency and running `x test core` - the latter caught several
bugs the unit tests didn't catch (because I originally wrote the tests
wrong). So I am slightly nervous about adding a 16-byte version that is
much more poorly tested than the other intrinsics.
2023-06-25 21:07:00 +00:00
|
|
|
assert_eq!(expected, target, "{} should store to target", stringify!($name));
|
|
|
|
});
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
};
|
|
|
|
}
|
|
|
|
|
|
|
|
test_op!(add, |left, right| left.wrapping_add(right));
|
|
|
|
test_op!(clr, |left, right| left & !right);
|
|
|
|
test_op!(xor, std::ops::BitXor::bitxor);
|
|
|
|
test_op!(or, std::ops::BitOr::bitor);
|
2025-06-13 05:27:47 +00:00
|
|
|
use compiler_builtins::{foreach_bytes, foreach_ordering};
|
Port outline-atomics to rust
This has a very long history, summarized in
https://github.com/rust-lang/rust/issues/109064. This port is a very
minimal subset of `aarch64/lse.S` from LLVM's compiler-rt. In
particular, it is missing the following:
1. Any form of runtime dispatch between LL/SC and LSE.
Determining which version of the intrinsics to use
requires one of the following:
i) `getauxval` from glibc. It's unclear whether `compiler_builtins` is
allowed to depend on libc at all, and musl doesn't even support
getauxval. Don't enshrine the requirement "de-facto" by making it
required for outline-atomics.
ii) kernel support. Linux and FreeBSD have limited support, but it
requires an extremely recent kernel version and doesn't work at all under QEMU (https://github.com/rust-lang/rust/issues/109064#issuecomment-1494939904).
Instead, we hard-code LL/SC intrinsics. Users who want LSE support
should use the LLVM compiler-rt (if you're building from source in
rust-lang/rust, make sure you have `src/llvm-project` checked out
locally. the goal is to soon add a new `optimized-compiler-builtins`
option so this is easier to discover).
2. The global `___aarch64_have_lse_atomics` CTOR, required to do runtime
dispatch. Thom Chiviolani has this to say about global CTORs:
> static ctors are problems because we are pretty eager about dead code elim
> in general if you have a module that isnt directly reference we will probably not have its static ctors
> also, while llvm has a super robust way to have a static ctor (theres s special "appending global" to use for c++), we dont use that and just have people make a #[used] static in a special section
> 1. the robust way kinda requires rust knowing that the argument is a static ctor (maybe a #[rustc_static_ctor] attribute). it also would be... finnicky, since on windows we actually care beyond being a static ctor, that we run as part in a specific group of ctors, which means a very specific section (one for TLS and the other for, uh, i dont remember)
> 2. we still actually have to codegen the cgu that isn't referenced. but maybe we could remember that it has that attribute and use that
So while this is possible in theory, it's decidedly non-trivial, and
needs invasive changes to rust itself. In any case, it doesn't matter
until we decide the story around libc.
3. The 16-byte (i128) version of compare_and_swap. This wouldn't be
*too* hard to add, but it would be hard to test. The way I tested the
existing code was not just with unit tests but also by loading it as a
path dependency and running `x test core` - the latter caught several
bugs the unit tests didn't catch (because I originally wrote the tests
wrong). So I am slightly nervous about adding a 16-byte version that is
much more poorly tested than the other intrinsics.
2023-06-25 21:07:00 +00:00
|
|
|
compiler_builtins::foreach_cas!(cas::test);
|
2023-06-26 13:54:47 +00:00
|
|
|
compiler_builtins::foreach_cas16!(test_cas16);
|
Port outline-atomics to rust
This has a very long history, summarized in
https://github.com/rust-lang/rust/issues/109064. This port is a very
minimal subset of `aarch64/lse.S` from LLVM's compiler-rt. In
particular, it is missing the following:
1. Any form of runtime dispatch between LL/SC and LSE.
Determining which version of the intrinsics to use
requires one of the following:
i) `getauxval` from glibc. It's unclear whether `compiler_builtins` is
allowed to depend on libc at all, and musl doesn't even support
getauxval. Don't enshrine the requirement "de-facto" by making it
required for outline-atomics.
ii) kernel support. Linux and FreeBSD have limited support, but it
requires an extremely recent kernel version and doesn't work at all under QEMU (https://github.com/rust-lang/rust/issues/109064#issuecomment-1494939904).
Instead, we hard-code LL/SC intrinsics. Users who want LSE support
should use the LLVM compiler-rt (if you're building from source in
rust-lang/rust, make sure you have `src/llvm-project` checked out
locally. the goal is to soon add a new `optimized-compiler-builtins`
option so this is easier to discover).
2. The global `___aarch64_have_lse_atomics` CTOR, required to do runtime
dispatch. Thom Chiviolani has this to say about global CTORs:
> static ctors are problems because we are pretty eager about dead code elim
> in general if you have a module that isnt directly reference we will probably not have its static ctors
> also, while llvm has a super robust way to have a static ctor (theres s special "appending global" to use for c++), we dont use that and just have people make a #[used] static in a special section
> 1. the robust way kinda requires rust knowing that the argument is a static ctor (maybe a #[rustc_static_ctor] attribute). it also would be... finnicky, since on windows we actually care beyond being a static ctor, that we run as part in a specific group of ctors, which means a very specific section (one for TLS and the other for, uh, i dont remember)
> 2. we still actually have to codegen the cgu that isn't referenced. but maybe we could remember that it has that attribute and use that
So while this is possible in theory, it's decidedly non-trivial, and
needs invasive changes to rust itself. In any case, it doesn't matter
until we decide the story around libc.
3. The 16-byte (i128) version of compare_and_swap. This wouldn't be
*too* hard to add, but it would be hard to test. The way I tested the
existing code was not just with unit tests but also by loading it as a
path dependency and running `x test core` - the latter caught several
bugs the unit tests didn't catch (because I originally wrote the tests
wrong). So I am slightly nervous about adding a 16-byte version that is
much more poorly tested than the other intrinsics.
2023-06-25 21:07:00 +00:00
|
|
|
compiler_builtins::foreach_swp!(swap::test);
|
|
|
|
compiler_builtins::foreach_ldadd!(add::test);
|
|
|
|
compiler_builtins::foreach_ldclr!(clr::test);
|
|
|
|
compiler_builtins::foreach_ldeor!(xor::test);
|
|
|
|
compiler_builtins::foreach_ldset!(or::test);
|