Module mako::tokenization
source · [−]Expand description
Tokenization module handles all tokenization and untokenization
Re-exports
pub use hf_tokenizers::Tokenizer as HFTokenizer;Modules
The core of tokenizers, written in Rust.
Provides an implementation of today’s most used tokenizers, with a focus on performance and
versatility.
Structs
Traits
A trait to implement for all tokenizers, contains basic tokenizing and untokenizing functions