Expand description
A tokenizer for styx
Structs§
- Span
- A span representing a range in the source text.
- Token
- A token with its kind, span, and source text slice.
- Tokenizer
- A tokenizer that produces tokens from Styx source text.
Enums§
- Token
Kind - The kind of a token.