Expand description

Tokenization utilities for building parsers in Rust


A queue which can be used as a sender and reader. Use this for buffering all the tokens before reading

A token queue which has a backing generator/lexer which is called when needed by parsing logic

A wrapping struct for the cache around GeneratorTokenQueue. Use as the second parameter in the generator/lexer function

A token queue used for doing lexing and parsing on different threads. Will send tokens between threads

A structure with a piece of data and some additional data such as a position


A reader over a sequence of tokens

Trait for a sender that can append a token to a sequence

PartialEq is required for comparing tokens with TokenReader::expect_next