mirror of
https://github.com/rust-lang/rust.git
synced 2025-04-30 20:17:50 +00:00
![]() `Cursor` is currently hidden, and the main tokenization path uses `rustc_lexer::first_token` which involves constructing a new `Cursor` for every single token, which is weird. Also, `first_token` also can't handle empty input, so callers have to check for that first. This commit makes `Cursor` public, so `StringReader` can contain a `Cursor`, which results in a simpler structure. The commit also changes `StringReader::advance_token` so it returns an `Option<Token>`, simplifying the the empty input case. |
||
---|---|---|
.. | ||
src | ||
Cargo.toml |