Crate tokenizer_lib[−][src]
Expand description
Tokenization utilities for building parsers in Rust
Structs
A queue which can be used as a sender and reader. Use this for buffering all the tokens before reading
A token queue which has a backing generator/lexer which is called when needed by parsing logic
A token queue used for doing lexing and parsing on different threads. Will send tokens between threads
A structure with a piece of data and some additional data such as a position
Traits
A reader over a sequence of tokens
Trait for a sender that can append a token to a sequence