rust/compiler/rustc_lexer/Cargo.toml

[package]
name = "rustc_lexer"
version = "0.1.0"
license = "MIT OR Apache-2.0"
edition = "2021"

repository = "https://github.com/rust-lang/rust/"
description = """
Rust lexer used by rustc. No stability guarantees are provided.
"""

# Note: do not remove this blank `[lib]` section.
# This will be used when publishing this crate as `rustc-ap-rustc_lexer`.
[lib]

# Note that this crate purposefully does not depend on other rustc crates
[dependencies]
unicode-xid = "0.2.0"
unic-emoji-char = "0.9.0"

[dev-dependencies]
expect-test = "1.4.0"
Introduce rustc_lexer The idea here is to make a reusable library out of the existing rust-lexer, by separating out pure lexing and rustc-specific concerns, like spans, error reporting an interning. So, rustc_lexer operates directly on `&str`, produces simple tokens which are a pair of type-tag and a bit of original text, and does not report errors, instead storing them as flags on the token. 2019-05-06 08:53:40 +00:00			`[package]`
			`name = "rustc_lexer"`
			`version = "0.1.0"`
fill metadata in rustc_lexer's Cargo.toml We publish this to crates.io, so having non-empty meta is useful 2019-09-05 20:03:50 +00:00			`license = "MIT OR Apache-2.0"`
Migrate to 2021 2021-09-19 16:49:55 +00:00			`edition = "2021"`
Introduce rustc_lexer The idea here is to make a reusable library out of the existing rust-lexer, by separating out pure lexing and rustc-specific concerns, like spans, error reporting an interning. So, rustc_lexer operates directly on `&str`, produces simple tokens which are a pair of type-tag and a bit of original text, and does not report errors, instead storing them as flags on the token. 2019-05-06 08:53:40 +00:00
fill metadata in rustc_lexer's Cargo.toml We publish this to crates.io, so having non-empty meta is useful 2019-09-05 20:03:50 +00:00			`repository = "https://github.com/rust-lang/rust/"`
			`description = """`
			`Rust lexer used by rustc. No stability guarantees are provided.`
			`"""`

Add comment 2019-07-27 13:10:09 +00:00			# Note: do not remove this blank `[lib]` section.
			# This will be used when publishing this crate as `rustc-ap-rustc_lexer`.
Add lib section to rustc_lexer's Cargo.toml 2019-07-27 08:21:42 +00:00			`[lib]`
remove XID and Pattern_White_Space unicode tables from libcore They are only used by rustc_lexer, and are not needed elsewhere. So we move the relevant definitions into rustc_lexer (while the actual unicode data comes from the unicode-xid crate) and make the rest of the compiler use it. 2019-07-21 11:50:39 +00:00
			`# Note that this crate purposefully does not depend on other rustc crates`
			`[dependencies]`
			`unicode-xid = "0.2.0"`
Tokenize emoji as if they were valid indentifiers In the lexer, consider emojis to be valid identifiers and reject them later to avoid knock down parse errors. 2021-08-29 08:34:23 +00:00			`unic-emoji-char = "0.9.0"`
Introduce expect snapshot testing library into rustc Snapshot testing is a technique for writing maintainable unit tests. Unlike usual `assert_eq!` tests, snapshot tests allow to automatically upgrade expected values on test failure. In a sense, snapshot tests are inline-version of our beloved UI-tests. Example: ![expect](https://user-images.githubusercontent.com/1711539/90888810-3bcc8180-e3b7-11ea-9626-d06e89e1a0bb.gif) A particular library we use, `expect_test` provides an `expect!` macro, which creates a sort of self-updating string literal (by using `file!` macro). Self-update is triggered by setting `UPDATE_EXPECT` environmental variable (this info is printed during the test failure). This library was extracted from rust-analyzer, where we use it for most of our tests. There are some other, more popular snapshot testing libraries: * https://github.com/mitsuhiko/insta * https://github.com/aaronabramov/k9 The main differences of `expect` are: * first-class snapshot objects (so, tests can be written as functions, rather than as macros) * focus on inline-snapshots (but file snapshots are also supported) * restricted feature set (only `assert_eq` and `assert_debug_eq`) * no extra runtime (ie, no `cargo insta`) See https://github.com/rust-analyzer/rust-analyzer/pull/5101 for a an extended comparison. It is unclear if this testing style will stick with rustc in the long run. At the moment, rustc is mainly tested via integrated UI tests. But in the library-ified world, unit-tests will become somewhat more important (that's why use use `rustc_lexer` library-ified library as an example in this PR). Given that the cost of removal shouldn't be too high, it probably makes sense to just see if this flies! 2020-08-21 12:03:50 +00:00
			`[dev-dependencies]`
update cpufeatures, swap difference to dissimilar 2022-11-22 14:41:06 +00:00			`expect-test = "1.4.0"`