Crate tokenizer_lib

Structs

ParseError	A error struct with a reason and position
Span	Represents a start and end of something in a sequence
StaticTokenChannel	A synchronous "channel" which can be used as a sender and reader. Will buffer all tokens into a VecDeque before reading
StreamedTokenReader
StreamedTokenSender
Token	Defines a Token of type T and with a Position

Traits

TokenReader	Trait for a reader which returns tokens over a current sequence
TokenSender	Trait for a sender that can append a token to a sequence

Functions

get_streamed_token_channel

Will return a TokenSender and TokenReader for use when lexing and parsing in separate threads Unlike StaticTokenChannel it does not buffer all the tokens before parsing can begin