Struct Vocabulary 

pub struct Vocabulary { /* private fields */ }

Vocabulary for a large language model.

Implementations

impl Vocabulary

pub fn new(eos_token_id: u32) -> Self

Creates an empty vocabulary.

pub fn tokens(&self) -> &HashMap<Vec<u8>, Vec<u32>>

Returns all tokens with their token ids in vocabulary.

pub fn token_ids(&self, token: impl AsRef<[u8]>) -> Option<&[u32]>

Returns all token ids per provided token if available in the vocabulary.

pub fn eos_token_id(&self) -> u32

Gets the identifier of the special end of the sentence token.

pub fn try_insert( &mut self, token: impl Into<Vec<u8>>, id: u32, ) -> Result<(), Error>

Inserts a token with the given id. Idempotent: repeating (token, id) is a no-op. A different id under the same bytes is still kept.

pub fn remove(&mut self, token: impl Into<Vec<u8>>)

Removes a given token from the vocabulary.

pub fn len(&self) -> usize

Returns ordinary token ids plus EOS, not the sparse mask width. Overflowing counts saturate instead of wrapping.

pub fn is_empty(&self) -> bool

True when no ordinary tokens are present (EOS is not counted).

Trait Implementations

impl Clone for Vocabulary

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Vocabulary

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Vocabulary

fn default() -> Self

Returns the “default value” for a type.

impl Display for Vocabulary

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for Vocabulary

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for Vocabulary