Skip to main content

Module huggingface

ferrum_tokenizer::implementations

Module huggingface 

Source
Expand description

HuggingFace tokenizer implementation

Structsยง

HuggingFaceTokenizer
HuggingFace tokenizer wrapper
HuggingFaceTokenizerFactory
HuggingFace tokenizer factory
IncrementalState
Incremental decoding state