Skip to main content

Module tokenizer

dynamo_bench::coding

Module tokenizer 

Source

Structs§

HfTokenizerFactory
HfTokenizerWorker
LazyTokenizer
A worker created on its first encode, so an export whose turns all carry usage loads no tokenizer.

Traits§

TokenizerFactory
TokenizerWorker