Struct BytePairEncoding

pub struct BytePairEncoding { /* private fields */ }

Representation of the byte pair dictionary. This struct provides various conversions. We put all of them into a single struct so that they can be reused by different implementations.

Implementations

impl BytePairEncoding

pub fn from_dictionary( tokens: impl IntoIterator<Item = Vec<u8>>, hash_factor: Option<u64>, ) -> BytePairEncoding

Construct a BytePairEncoding instance from an iterator that enumerates all tokens. A suitable hash factor may be necessary to prevent hash collisions, which can be found using [find_hash_factor_for_dictionary].

The recommended approach is to store the serialized value and reuse that, to prevent repeating the cost of computing the hash factor and encoding.

pub fn num_tokens(&self) -> usize

Return the number of tokens in this BPE dictionary.

pub fn token_bytes(&self, token_id: u32) -> &[u8]

Converts a token id into its corresponding token bytes. Panics if the token_id is not within the valid 0..num_tokens() range!

pub fn token_len(&self, token_id: u32) -> usize

Returns the length of the decoded byte slice of a token.

pub fn decode_tokens(&self, tokens: &[u32]) -> Vec<u8>

Decode a sequence of tokens back to its original byte sequence. Note: we don’t return here a str, since not every token sequence corresponds to a valid utf8 sequence.

pub fn count(&self, text: &[u8]) -> usize

Counts the number tokens produced when encoding the text.

pub fn count_till_limit(&self, text: &[u8], token_limit: usize) -> Option<usize>

Returns the token count iff the total token count stays below the specified token_limit. Otherwise, it returns none. This function can be faster than count when the token_limit is much smaller than the provided text.

pub fn encode_via_table(&self, text: &[u8]) -> Vec<u32>

pub fn encode_via_backtracking(&self, text: &[u8]) -> Vec<u32>

pub fn encode_via_bitfield(&self, text: &[u8]) -> Vec<u32>

pub fn encode_greedy(&self, text: &[u8]) -> Vec<u32>

It is not recommended to use this function, since it doesn’t output the correct BPE encoded sequence.

pub fn encode_minimal(&self, text: &[u8]) -> Vec<u32>

This function computes the shortest possible encoding sequence which will usually differ from the tokenization produced by the original BPE algorithm.

Trait Implementations

impl<'de> Deserialize<'de> for BytePairEncoding

fn deserialize<__D>( __deserializer: __D, ) -> Result<BytePairEncoding, <__D as Deserializer<'de>>::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for BytePairEncoding

fn serialize<__S>( &self, __serializer: __S, ) -> Result<<__S as Serializer>::Ok, <__S as Serializer>::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.