Rollup merge of #94713 - clarfonthey:is_char_surrogate, r=scottmcm

Add u16::is_utf16_surrogate

Right now, there are methods in the standard library for encoding and decoding UTF-16, but at least for the moment, there aren't any methods specifically for `u16` to help work with UTF-16 data. Since the full logic already exists, this wouldn't really add any code, just expose what's already there.

This method in particular is useful for working with the data returned by Windows `OsStrExt::encode_wide`. Initially, I was planning to also offer a `TryFrom<u16> for char`, but decided against it for now. There is plenty of code in rustc that could be rewritten to use this method, but I only checked within the standard library to replace them.

I think that offering more UTF-16-related methods to u16 would be useful, but I think this one is a good start. For example, one useful method might be `u16::is_pattern_whitespace`, which would check if something is the Unicode `Pattern_Whitespace` category. We can get away with this because all of the `Pattern_Whitespace` characters are in the basic multilingual plane, and hence we don't need to check for surrogates.
This commit is contained in:
Dylan DPC 2022-03-23 03:05:31 +01:00 committed by GitHub
commit 25acd9331e
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
3 changed files with 30 additions and 3 deletions

View File

@ -91,7 +91,7 @@ impl<I: Iterator<Item = u16>> Iterator for DecodeUtf16<I> {
None => self.iter.next()?,
};
if u < 0xD800 || 0xDFFF < u {
if !u.is_utf16_surrogate() {
// SAFETY: not a surrogate
Some(Ok(unsafe { from_u32_unchecked(u as u32) }))
} else if u >= 0xDC00 {
@ -125,7 +125,7 @@ impl<I: Iterator<Item = u16>> Iterator for DecodeUtf16<I> {
// buf is empty, no additional elements from it.
None => (0, 0),
// `u` is a non surrogate, so it's always an additional character.
Some(u) if u < 0xD800 || 0xDFFF < u => (1, 1),
Some(u) if !u.is_utf16_surrogate() => (1, 1),
// `u` is a leading surrogate (it can never be a trailing surrogate and
// it's a surrogate due to the previous branch) and `self.iter` is empty.
//

View File

@ -93,7 +93,7 @@
#![warn(missing_docs)]
#![allow(explicit_outlives_requirements)]
//
// Library features for const fns:
// Library features:
#![feature(const_align_offset)]
#![feature(const_align_of_val)]
#![feature(const_alloc_layout)]
@ -146,6 +146,8 @@
#![feature(ptr_metadata)]
#![feature(slice_ptr_get)]
#![feature(str_internals)]
#![feature(utf16_extra)]
#![feature(utf16_extra_const)]
#![feature(variant_count)]
#![feature(const_array_from_ref)]
#![feature(const_slice_from_ref)]

View File

@ -820,6 +820,31 @@ impl u16 {
uint_impl! { u16, u16, i16, NonZeroU16, 16, 65535, 4, "0xa003", "0x3a", "0x1234", "0x3412", "0x2c48",
"[0x34, 0x12]", "[0x12, 0x34]", "", "" }
widening_impl! { u16, u32, 16, unsigned }
/// Checks if the value is a Unicode surrogate code point, which are disallowed values for [`char`].
///
/// # Examples
///
/// ```
/// #![feature(utf16_extra)]
///
/// let low_non_surrogate = 0xA000u16;
/// let low_surrogate = 0xD800u16;
/// let high_surrogate = 0xDC00u16;
/// let high_non_surrogate = 0xE000u16;
///
/// assert!(!low_non_surrogate.is_utf16_surrogate());
/// assert!(low_surrogate.is_utf16_surrogate());
/// assert!(high_surrogate.is_utf16_surrogate());
/// assert!(!high_non_surrogate.is_utf16_surrogate());
/// ```
#[must_use]
#[unstable(feature = "utf16_extra", issue = "94919")]
#[rustc_const_unstable(feature = "utf16_extra_const", issue = "94919")]
#[inline]
pub const fn is_utf16_surrogate(self) -> bool {
matches!(self, 0xD800..=0xDFFF)
}
}
#[lang = "u32"]