Struct Tokenizer

pub struct Tokenizer {
    pub bpe: BytePairEncoding,
    pub pre: Option<Pretokenizer>,
    /* private fields */
}

A byte-pair encoding tokenizer that supports a pre-tokenization regex. The direct methods on this type pre-tokenize the input text and should produce the same output as the tiktoken tokenizers. The type gives access to the regex and underlying byte-pair encoding if needed. Note that using the byte-pair encoding directly does not take the regex into account and may result in output that differs from tiktoken.

Fields

bpe: BytePairEncoding

The byte-pair encoding for this tokenizer.

pre: Option<Pretokenizer>

The pattern regex used to split the input.

Implementations

impl Tokenizer

pub fn new( bpe: BytePairEncoding, pat: Option<&str>, nfc: bool, ) -> Result<Self, BuildError>

Build a tokenizer with an optional pretokenization regex pattern.

pub fn new_lookahead( bpe: BytePairEncoding, patterns: &[(&str, bool)], nfc: bool, ) -> Result<Self, BuildError>

Build a tokenizer with pretokenization regex patterns. If the boolean for a pattern is true, the pattern is assumed to be a look-ahead pattern with exactly one look-ahead character!

pub fn count<'a, I: Normalizable<'a>>(&self, text: I) -> usize

Count the number of tokens produced when encoding the text. Applies pre-tokenization before counting.

pub fn count_till_limit( &self, text: &NormalizedString<'_>, token_limit: usize, ) -> Option<usize>

Returns the token count iff the total token count stays below the specified token_limit. Otherwise, it returns none. This function can be faster than Self::count` when the token limit is much smaller than the provided text. Applies pre-tokenization before counting.

Note: This function assumes that the text is already normalized, so that this function can run in roughly O(token_limit) time.

pub fn encode<'a, I: Normalizable<'a>>(&self, text: I) -> Vec<u32>

Returns the tokens for the encoding of the given text. Applies pre-tokenization before encoding.

pub fn decode(&self, tokens: &[u32]) -> Option<String>

Returns the text corresponding to the given encoding if it is valid UTF-8. Otherwise, returns none.

pub fn split<'a>(&'a self, text: &'a str) -> impl Iterator<Item = &'a str>

Returns an iterator with the text pieces resulting from pre-tokenization. If this tokenizer does not have pre-tokenization, the iterator returns the full text.

pub fn normalize<'a, I: Normalizable<'a>>( &self, text: I, ) -> NormalizedString<'a>

Returns the normalized text if the tokenizer requires normalization. If the input was already normalized, this function is a noop.