Module general_sam::utils::tokenize

source ·
Available on crate feature utils only.
Expand description

Greedy tokenizer.

Modules

Structs

  • GreedyTokenizer
    Greedy tokenizer with a general suffix automaton of the vocabulary.