Struct Tokenizer

pub struct Tokenizer {
    pub bpe: BytePairEncoding,
    pub pre: Option<Pretokenizer>,
}

A byte-pair encoding tokenizer that supports a pre-tokenization regex. The direct methods on this type pre-tokenize the input text and should produce the same output as the tiktoken tokenizers. The type gives access to the regex and underlying byte-pair encoding if needed. Note that using the byte-pair encoding directly does not take the regex into account and may result in output that differs from tiktoken.

Fields

bpe: BytePairEncoding

The byte-pair encoding for this tokenizer.

pre: Option<Pretokenizer>

The pattern regex used to split the input.

Implementations

impl Tokenizer

pub fn new(bpe: BytePairEncoding, pat: Option<&str>) -> Result<Self, BuildError>

Build a tokenizer with an optional pretokenization regex pattern.

pub fn new_lookahead( bpe: BytePairEncoding, patterns: &[(&str, bool)], ) -> Result<Self, BuildError>

Build a tokenizer with pretokenization regex patterns. If the boolean for a pattern is true, the pattern is assumed to be a look-ahead pattern with exactly one look-ahead character!

pub fn count(&self, text: &str) -> usize

Count the number of tokens produced when encoding the text. Applies pre-tokenization before counting.

pub fn count_till_limit(&self, text: &str, token_limit: usize) -> Option<usize>

Returns the token count iff the total token count stays below the specified token_limit. Otherwise, it returns none. This function can be faster than Self::count` when the token limit is much smaller than the provided text. Applies pre-tokenization before counting.

pub fn encode(&self, text: &str) -> Vec<u32>

Returns the tokens for the encoding of the given text. Applies pre-tokenization before encoding.

pub fn decode(&self, tokens: &[u32]) -> Option<String>

Returns the text corresponding to the given encoding if it is valid UTF-8. Otherwise, returns none.

pub fn split<'a>(&'a self, text: &'a str) -> impl Iterator<Item = &'a str> + 'a

Returns an iterator with the text pieces resulting from pre-tokenization. If this tokenizer does not have pre-tokenization, the iterator returns the full text.