Module general_sam::utils::tokenize

Available on crate feature utils only.

Greedy tokenizer.

Modules

• trietrie

Structs

• GreedyTokenizer
  Greedy tokenizer with a general suffix automaton of the vocabulary.
• OwnedGeneralSAM