Crate tokengeex 

Structs

CrlfProcessor

Replaces occurences of \r\n by \n.

Lattice

Structure to implement Viterbi algorithm to find the best encoding, or sample from all possible encodings of a given sentence.

LocalTask

Model

Node

A node from the lattice, that helps reconstruct the underlying String

ScoredToken

A token and its score.

Task

Tokenizer

Trie

TrieIterator

VecPool

Enums

Error

ProcessorWrapper

UnicodeProcessor

Unicode normalizer.

Traits

Processor

A processor is a step of the tokenization pipeline. It can be used to transform input sequences before they are fed to the model and to transform the output sequences after they are generated by the model.

Functions

make_vocab

mb_per_sec

new_default_vocab

par_chunk_size

Type Aliases

Result

Token

An arbitrary sequence of bytes. Almost always valid UTF-8 but not guaranteed.

TokenID

A numerical ID for a token. Cannot be larger than u32::MAX.