general_sam::utils

Module tokenize

Available on crate feature utils only.

Expand description

Greedy tokenizer.

Modules§

trietrie

Structs§

GreedyTokenizer: Greedy tokenizer with a general suffix automaton of the vocabulary.
OwnedGeneralSam