Crate token_dict 

Re-exports

pub use dict::TokenDict;

pub use token::Token;

Modules

dict

module for token dictionary

token

module for Token type

Structs

UTF8CharIter

iterator for live converting utf8 to chars, returning errors for all bytes that aren’t part of a valid character. useful for lazily detokenizing into a string

Traits

Val

trait for unifying primitives and their references

Functions

dict_to_file_lines

write a TokenDict to a file as lines. Assumes tokens are valid UTF8 and single line

file_lines_to_dict

read tokens separated by line from a file into a TokenDict. Assumes tokens are valid UTF8 and single line