Crate token_dict

Re-exports

pub use dict::TokenDict;

pub use token::Token;

Modules

dict

module for token dictionary

token

module for Token type

Structs

UTF8CharIter

iterator for live converting utf8 to chars, returning errors for all bytes that aren’t part of a valid character. useful for lazily detokenizing into a string

Traits

Val

trait for unifying primitives and their references