Struct LlamaVocab 

pub struct LlamaVocab<'model> { /* private fields */ }

A safe wrapper around llama_vocab.

The vocabulary is owned by the LlamaModel it was retrieved from, so dropping a LlamaVocab does not free the underlying llama_vocab.

Implementations

impl<'model> LlamaVocab<'model>

pub fn vocab_type(&self) -> VocabType

The type of the vocabulary.

pub fn n_tokens(&self) -> i32

The number of tokens the model was trained on.

pub fn text(&self, token: LlamaToken) -> Option<&CStr>

The text representation of a token, if any.

pub fn score(&self, token: LlamaToken) -> f32

The score of a token.

pub fn attr(&self, token: LlamaToken) -> LlamaTokenAttrs

The attributes of a token.

pub fn is_eog(&self, token: LlamaToken) -> bool

Check if the token is supposed to end generation (end-of-generation, eg. EOS, EOT, etc.)

pub fn is_control(&self, token: LlamaToken) -> bool

Identify if the token is a control token or a render-able token.

pub fn bos(&self) -> LlamaToken

The beginning of stream token.

pub fn eos(&self) -> LlamaToken

The end of stream token.

pub fn eot(&self) -> LlamaToken

The end of turn token.

pub fn sep(&self) -> LlamaToken

The separator token.

pub fn nl(&self) -> LlamaToken

The newline token.

pub fn pad(&self) -> LlamaToken

The padding token.

pub fn mask(&self) -> LlamaToken

The mask token.

pub fn should_add_bos(&self) -> bool

Whether to add the beginning of stream token when tokenizing.

pub fn should_add_eos(&self) -> bool

Whether to add the end of stream token when tokenizing.

pub fn should_add_sep(&self) -> bool

Whether to add the separator token when tokenizing.

pub fn suppress_tokens(&self) -> &[LlamaToken]

Model-specific suppress tokens.

(gguf key: tokenizer.ggml.suppress_tokens)

pub fn fim_pre(&self) -> LlamaToken

The fill-in-the-middle prefix token.

pub fn fim_suf(&self) -> LlamaToken

The fill-in-the-middle suffix token.

pub fn fim_mid(&self) -> LlamaToken

The fill-in-the-middle middle token.

pub fn fim_pad(&self) -> LlamaToken

The fill-in-the-middle padding token.

pub fn fim_rep(&self) -> LlamaToken

The fill-in-the-middle repo token.

pub fn fim_sep(&self) -> LlamaToken

The fill-in-the-middle separator token.

pub fn tokenize( &self, text: &[u8], add_special: bool, parse_special: bool, ) -> Vec<LlamaToken>

Tokenize text to a Vec of tokens.

This is a convenience wrapper on top of tokenize_into.

Example

Tokenize a string of text.

let tokens = vocab.tokenize("Hello, World!".as_bytes(), true, true);

pub fn tokenize_into( &self, text: &[u8], buffer: &mut Vec<LlamaToken>, add_special: bool, parse_special: bool, )

Tokenize text into a user-provided buffer.

The output is placed into the vector’s spare capacity, and the Vec will be resized if the output doesn’t fit.

Setting add_special allows this function to add BOS and EOS tokens if the model is configured to do so.

Setting parse_special allows this function to tokenize special and/or control tokens which otherwise are not exposed and treated as plaintext. Setting it does not insert a leading space.

pub fn detokenize( &self, tokens: &[LlamaToken], remove_special: bool, unparse_special: bool, ) -> Vec<u8>

Detokenize a list of tokens.

This is a convenience wrapper on top of detokenize_into.

pub fn detokenize_into( &self, tokens: &[LlamaToken], buffer: &mut Vec<u8>, remove_special: bool, unparse_special: bool, )

Detokenize a list of tokens into a user-provided buffer.

This is the inverse of tokenization.

The output is placed into the vector’s spare capacity, and the Vec will be resized if the output doesn’t fit.

Setting remove_special allows this function to remove BOS and EOS tokens if the model is configured to do so.

Special tokens are rendered in the output if unparse_special is set.

Example

Convert a list of tokens to a string.

When converting tokens while streaming, you probably want to use token_to_piece instead.

let token_bytes = vocab.detokenize(&tokens, true, true);
let (token_string, _had_errors) = encoding_rs::UTF_8.decode_without_bom_handling(&token_bytes);
print!("{token_string}");

pub fn token_to_piece( &self, token: LlamaToken, special: bool, lstrip: Option<NonZeroU16>, ) -> Vec<u8>

Detokenize a single token.

This is a convenience method on top of token_to_piece_into.

pub fn token_to_piece_into( &self, token: LlamaToken, buffer: &mut Vec<u8>, special: bool, lstrip: Option<NonZeroU16>, )

Detokenize a single token into a caller-provided buffer.

The output is placed into the vector’s spare capacity, and the Vec will be resized if the output doesn’t fit.

Special tokens are rendered in the output if special is set.

This will skip up to lstrip leading spaces before copying, which can be useful when encoding/decoding multiple tokens with ‘add_space_prefix’.

Example

Convert a stream of tokens to a string, with proper partial UTF-8 decoding (such as for emoji support) and reusing buffers.

let mut buffer = Vec::new();
let mut decoder = encoding_rs::UTF_8.new_decoder();
let mut token_string = String::new();
for token in tokens {
    // Clear buffers.
    buffer.clear();
    token_string.clear();

    // Get the token bytes.
    vocab.token_to_piece_into(token, &mut buffer, true, None);

    // Ensure the size of the string buffer is large enough.
    let max_len = decoder.max_utf8_buffer_length(buffer.len()).unwrap();
    token_string.reserve(max_len);

    // Grab the parts of the buffer that's valid (and leave the rest
    // for the next token to reuse).
    let (result, read, _) = decoder.decode_to_string(&buffer, &mut token_string, false);
    assert!(
        matches!(result, encoding_rs::CoderResult::InputEmpty) && read == buffer.len(),
        "UTF-8 decoder capacity bound must consume the complete token"
    );

    // Print the resulting partially decoded token.
    println!("{token} --> {token_string}");
}

pub fn tokens(&self) -> impl Iterator<Item = LlamaToken>

Get all tokens in the vocabulary.

Trait Implementations

impl<'model> Debug for LlamaVocab<'model>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Send for LlamaVocab<'_>

impl Sync for LlamaVocab<'_>