Crate tokenizer_lib[][src]

Expand description

Tokenization utilities for building parsers in Rust

Structs

A queue which can be used as a sender and reader. Use this for buffering all the tokens before reading

A token queue which has a backing generator/lexer which is called when needed by parsing logic

A wrapping struct for the cache around GeneratorTokenQueue. Use as the second parameter in the generator/lexer function

A token queue used for doing lexing and parsing on different threads. Will send tokens between threads

A structure with a piece of data and some additional data such as a position

Traits

A reader over a sequence of tokens

Trait for a sender that can append a token to a sequence