Crate tokenizer_lib
source ·Expand description
Tokenization utilities for building parsers in Rust
Structs
A queue which can be used as a sender and reader. Use this for buffering all the tokens before reading
A token queue which has a backing generator/lexer which is called when needed by parsing logic
A wrapping struct for the cache around
GeneratorTokenQueue. Use as the second parameter
in the generator/lexer functionA token queue used for doing lexing and parsing on different threads. Will send tokens between threads
A structure with a piece of data and some additional data such as a position
Traits
A reader over a sequence of tokens
Trait for a sender that can append a token to a sequence
PartialEq is required for comparing tokens with TokenReader::expect_next