Expand description
Lexical shape of each supported language.
Languages differ in a small number of lexical decisions - how a comment starts, which quotes open a string, whether a backslash escapes, whether indentation is significant - and agree on everything else. Describing those differences as data keeps one tokenizer correct for all of them instead of one hand-written scanner per language.
Structs§
- Syntax
- The lexical rules of one language.
Enums§
- Language
- A language this crate can tokenize.