Struct Vocabulary

pub struct Vocabulary {
    pub id_to_token: Vec<Vec<u8>>,
    pub id_to_token_score: Vec<f32>,
    pub token_to_id: HashMap<Vec<u8>, TokenId>,
    pub max_token_length: usize,
}

The vocabulary used by a model.

Fields

id_to_token: Vec<Vec<u8>>

Maps every integer (index) token ID to its corresponding token.

id_to_token_score: Vec<f32>

Maps every integer (index) token ID to corresponding score.

token_to_id: HashMap<Vec<u8>, TokenId>

Maps a token to a token ID.

max_token_length: usize

The longest token in this vocabulary.

Implementations

impl Vocabulary

pub fn push_token(&mut self, id: TokenId, content: Vec<u8>, score: f32)

Add a token to the vocabulary.

The token added must have id directly after the last token in the vocabulary.

Panics

• This function can panic if id does not correspond to the next token in the vocabulary. That is, if there are already n tokens in the vocabulary, then id must be n.

pub fn token(&self, idx: usize) -> &[u8]

Converts a token index to the token it represents in this vocabulary.

pub fn tokenize<'a>( &'a self, text: &str, bos: bool, ) -> Result<Vec<(&'a [u8], TokenId)>, InferenceError>

Tokenize a text with this vocabulary.

bos controls whether a beginning-of-string token should be inserted.

Trait Implementations

impl Clone for Vocabulary

fn clone(&self) -> Vocabulary

Returns a duplicate of the value.

const fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Vocabulary

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Vocabulary

fn default() -> Vocabulary

Returns the “default value” for a type.