Struct rust_tokenizers::vocab::BpePairVocab

pub struct BpePairVocab {
    pub values: HashMap<(String, String), i64>,
}

Byte pair Encoding Vocab

BPE vocab containing the merges (dictionary of pairs with their priority) used to merge pairs together. This vocabulary element is used on BPE tokenizers such as GPT2 or RoBERTa. This vocabulary is not meant to be used directly, but rather as part of a BPE Tokenizer.

Fields

values: HashMap<(String, String), i64>

Implementations

impl BpePairVocab

pub fn from_file<P: AsRef<Path>>( path: P ) -> Result<BpePairVocab, TokenizerError>

Create a new BpePairVocab from a flat file containing merges in the format first_element second_element) The indices are implied by the lien position of each pair in the merges file. The first line needs to be a header and is skipped.

Example

use rust_tokenizers::vocab::{BpePairVocab, Vocab};
let path = "path/to/file";

let bpe_vocab = BpePairVocab::from_file(path);

pub fn from_sentencepiece_file<P: AsRef<Path>>( path: P ) -> Result<BpePairVocab, TokenizerError>

Create a new BpePairVocab from a SentencePiece file containing a BPE model.

Example

use rust_tokenizers::vocab::{BpePairVocab, Vocab};
let path = "path/to/spiece.model";

let bpe_vocab = BpePairVocab::from_sentencepiece_file(path);

pub fn byte_pair_to_id(&self, byte_pair: &BpePairRef<'_>) -> Option<&i64>

Gets the id of a “byte pair” in the merges vocab. Returns an optional index for the pair if it is found in the vocabulary.

Example

use rust_tokenizers::vocab::{BpePairRef, BpePairVocab, Vocab};
let path = "path/to/file";

let bpe_vocab = BpePairVocab::from_file(path).unwrap();

let query = BpePairRef {
    byte_1: &"won".to_string(),
    byte_2: &"derful".to_string(),
};
let id = bpe_vocab.byte_pair_to_id(&query);

Trait Implementations

impl Clone for BpePairVocab

fn clone(&self) -> BpePairVocab

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for BpePairVocab

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.