Struct ByteLevelBpeTokenizer 

pub struct ByteLevelBpeTokenizer {
    pub vocab: HashMap<String, u32>,
    pub id_to_token: Vec<String>,
    pub merges: Vec<(String, String)>,
    pub byte_encoder: HashMap<u8, char>,
    pub byte_decoder: HashMap<char, u8>,
}

GPT-2-style byte-level BPE tokenizer.

Every input byte is first mapped to a unique unicode character via the GPT-2 byte→unicode table, so the BPE algorithm operates on unicode character sequences. This makes the tokenizer vocabulary guaranteed to be lossless and eliminates any [UNK] token for arbitrary UTF-8 input.

Fields

vocab: HashMap<String, u32>

token string → integer id

id_to_token: Vec<String>

integer id → token string

merges: Vec<(String, String)>

ordered merge rules (left_piece, right_piece)

byte_encoder: HashMap<u8, char>

byte → unicode char

byte_decoder: HashMap<char, u8>

unicode char → byte (inverse of byte_encoder)

Implementations

impl ByteLevelBpeTokenizer

pub fn train(texts: &[&str], config: ByteLevelBpeConfig) -> Self

Train a new ByteLevelBpeTokenizer from raw text slices.

Pre-tokenises on whitespace boundaries and prepends Ġ (U+0120) to every word that is not at the beginning of the pre-tokenised sequence.

impl ByteLevelBpeTokenizer

pub fn encode(&self, text: &str) -> Vec<u32>

Encode text to a sequence of token IDs.

Applies the same whitespace pre-tokenisation + byte-encoding + BPE merges as during training.

pub fn decode(&self, ids: &[u32]) -> String

Decode a sequence of token IDs back to a UTF-8 string.

This is lossless: decode(encode(text)) == text for any valid UTF-8.

impl ByteLevelBpeTokenizer

pub fn save_vocab(&self, vocab_path: &str, merges_path: &str) -> Result<()>

Save vocabulary (HuggingFace JSON format) and merge rules to separate files.

The vocab file is a JSON object mapping token strings to integer IDs. The merges file contains one merge rule per line: left right.

pub fn load(vocab_path: &str, merges_path: &str) -> Result<Self>

Load a tokenizer from a HuggingFace-format vocab JSON and merges text file.

pub fn vocab_size(&self) -> usize

Return the vocabulary size.

pub fn id_to_token(&self, id: u32) -> Option<&str>

Look up the token string for an ID.

pub fn token_to_id(&self, token: &str) -> Option<u32>

Look up the ID for a token string.

Trait Implementations

impl Clone for ByteLevelBpeTokenizer

fn clone(&self) -> ByteLevelBpeTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ByteLevelBpeTokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.