Struct tokenizers::tokenizer::AddedVocabulary

pub struct AddedVocabulary { /* private fields */ }

A vocabulary built on top of the Model

This provides a way to add new vocabulary to a Tokenizer that has already been trained, in a previous process, maybe by someone else. This is especially interesting in the case of fine-tunings, where we want to finetune a model while adding some new functionalities using some new special tokens, or maybe add some tokens in the case of unknown tokens, etc.

One of the reasons we need to handle these tokens outside of the model is simply that for many models, it is not possible to add new tokens after the training process. For example, using BPE, the training process generates merges pairs along the vocabulary, and any token in the vocabulary can be decomposed in other tokens, down to the original alphabet. If we were to add new tokens after this training process, we couldn’t make sure the merges pairs exist as required.

Implementations

impl AddedVocabulary

pub fn new() -> Self

pub fn len(&self) -> usize

Size of the additional vocabulary

pub fn is_empty(&self) -> bool

Whether or not this vocabulary is empty

pub fn get_vocab(&self) -> &HashMap<String, u32>

Get the additional vocabulary

pub fn get_added_tokens_decoder(&self) -> &HashMap<u32, AddedToken>

Get the additional vocabulary with the AddedTokens

pub fn token_to_id(&self, token: &str, model: &impl Model) -> Option<u32>

Get the id matching one of our token if it exists

pub fn id_to_token(&self, id: u32, model: &impl Model) -> Option<String>

Get the token matching the given id if it exists

pub fn set_encode_special_tokens(&mut self, value: bool)

pub fn get_encode_special_tokens(&self) -> bool

pub fn is_special_token(&self, token: &str) -> bool

Check if a token is a special token

pub fn add_special_tokens<N: Normalizer>( &mut self, tokens: &[AddedToken], model: &impl Model, normalizer: Option<&N> ) -> usize

Add some special tokens to the vocabulary

pub fn add_tokens<N: Normalizer>( &mut self, tokens: &[AddedToken], model: &impl Model, normalizer: Option<&N> ) -> usize

Add some tokens to the vocabulary

pub fn extract_and_normalize<N: Normalizer>( &self, normalizer: Option<&N>, sequence: &str ) -> PreTokenizedString

Extract the additional vocabulary from the given sentence, normalizing it along the way.

Some tokens should match against their normalized representation, as well as the non-normalized one. For example, when we expect to extract the token yesterday in the input sentence I read a book Yesterday, if the normalizer is supposed to lowercase everything, we expect a match.

Trait Implementations

impl Clone for AddedVocabulary

fn clone(&self) -> AddedVocabulary

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for AddedVocabulary

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for AddedVocabulary

fn default() -> Self

Returns the “default value” for a type.

impl Serialize for AddedVocabulary

fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>

where S: Serializer,

Serialize this value into the given Serde serializer.