Expand description
§Generic Tokenizer
This crate provides a generic tokenizer that includes line and column numbers. This is useful for things like higher-level tokenizers, parsers, and language servers that need to track line and column numbers as they go.
Structs§
- Char
Locations - Location
- NonZero
- A wrapper around usize guaranteeing that this value is nonzero
- Span
- Tokenizer