Struct SentencePieceBpeModel 

pub struct SentencePieceBpeModel { /* private fields */ }

SentencePiece BPE Model

Model for SentencePiece BPE tokenizer. This model performs SentencePiece BPE decomposition using a priority queue and consecutive merges.

Expects a SentencePiece protobuf file when created from file.

Implementations

impl SentencePieceBpeModel

pub fn from_file<P: AsRef<Path>>( path: P, ) -> Result<SentencePieceBpeModel, TokenizerError>

Creates a SentencePiece BPE Model from a protobuf file.

Example

use rust_tokenizers::vocab::SentencePieceBpeModel;
let path = "path/to/spiece.model";

let sentence_piece_model = SentencePieceBpeModel::from_file(path);

pub fn tokenize_to_tokens(&self, initial_token: TokenRef<'_>) -> Vec<Token>

Tokenizes an input sequence into an array of Tokens by merging adjacent symbols present in the merges list.

Example

use rust_tokenizers::vocab::SentencePieceBpeModel;
use rust_tokenizers::TokenRef;
let path = "path/to/spiece.model";

let sentence_piece_bpe_model = SentencePieceBpeModel::from_file(path).unwrap();
let token = TokenRef::new("hello", &[0, 1, 2, 3]);
let tokenized_output = sentence_piece_bpe_model.tokenize_to_tokens(token);

pub fn populate_masks(&self, tokens: &mut [Token], whitespace_token: char)

Populates the mask field for a sequence of sub-tokens generated by a SentencePiece model. These masks are not generated as part of the standard unigram decomposition and must be added afterwards. Mutates the tokens in-place.

Arguments

• tokens (&mut [Token]): tokens to get the masks from
• whitespace_char (char): whitespace character to identify whether a token is a continuation token or not.

Example

use rust_tokenizers::vocab::SentencePieceBpeModel;
use rust_tokenizers::TokenRef;
let path = "path/to/spiece.model";
let sentence_piece_model = SentencePieceBpeModel::from_file(path).unwrap();

let token = TokenRef::new("hello", &[0, 1, 2, 3]);
let mut sub_tokens = sentence_piece_model.tokenize_to_tokens(token);
let sub_tokens_with_masks = sentence_piece_model.populate_masks(&mut sub_tokens, ' ');