Struct HfBpeTokenizer 

pub struct HfBpeTokenizer { /* private fields */ }

Byte Pair Encoding tokenizer.

Implements subword tokenization using BPE algorithm.

Implementations

impl BpeTokenizer

pub fn new(config: BpeConfig) -> BpeTokenizer

Create a new BPE tokenizer with given config

pub fn from_huggingface<P>(path: P) -> Result<BpeTokenizer, AprenderError>

where P: AsRef<Path>,

Load tokenizer from a HuggingFace tokenizer.json file path.

Parses the HuggingFace tokenizer.json format, extracting:

• model.vocab (token-to-ID mapping)
• model.merges (ordered BPE merge rules)
• added_tokens (special tokens like <|endoftext|>, <|im_start|>)

The byte encoder for UTF-8 byte-level BPE is built automatically.

Arguments

• path - Path to a HuggingFace tokenizer.json file

Returns

A fully loaded BpeTokenizer with vocabulary, merge rules, and special tokens.

Errors

Returns error if the file cannot be read or the JSON is malformed.

Example

use aprender::text::bpe::BpeTokenizer;

let tokenizer = BpeTokenizer::from_huggingface("path/to/tokenizer.json")
    .expect("failed to load tokenizer");
assert!(tokenizer.vocab_size() > 0);
let ids = tokenizer.encode("Hello world");
assert!(!ids.is_empty());

pub fn from_huggingface_json(json: &str) -> Result<BpeTokenizer, AprenderError>

Load tokenizer from a HuggingFace tokenizer.json string.

This is the in-memory counterpart of from_huggingface. Useful when the JSON has already been read into a string (e.g., from an HTTP response).

Arguments

• json - JSON string in HuggingFace tokenizer.json format

Returns

A fully loaded BpeTokenizer.

Errors

Returns error if JSON parsing fails or the structure is invalid.

pub fn from_vocab_merges<P, Q>( vocab_path: P, merges_path: Q, ) -> Result<BpeTokenizer, AprenderError>

where P: AsRef<Path>, Q: AsRef<Path>,

Load tokenizer from legacy GPT-2/RoBERTa format (vocab.json + merges.txt).

CodeBERT and other RoBERTa-family models use this format instead of the unified tokenizer.json. The vocab.json maps tokens to IDs, and merges.txt contains ordered BPE merge rules (one per line, #version header skipped).

Arguments

• vocab_path - Path to vocab.json ({"token": id, ...})
• merges_path - Path to merges.txt (header + one pair1 pair2 per line)

Returns

A fully loaded BpeTokenizer with vocabulary and merge rules.

Errors

Returns error if files cannot be read or JSON is malformed.

Example

use aprender::text::bpe::BpeTokenizer;

let tokenizer = BpeTokenizer::from_vocab_merges(
    "path/to/vocab.json",
    "path/to/merges.txt",
).expect("failed to load tokenizer");
assert!(tokenizer.vocab_size() > 0);

pub fn gpt2_base() -> BpeTokenizer

Create tokenizer with GPT-2 base vocabulary (stub)

Note

Real implementation requires loading vocabulary files.

pub fn add_special_token(&mut self, token: &str, id: u32)

Add a special token

pub fn add_merge(&mut self, first: &str, second: &str)

Add a merge rule

pub fn vocab_size(&self) -> usize

Get vocabulary size

pub fn token_to_id(&self, token: &str) -> Option<u32>

Get token ID for a token

pub fn id_to_token(&self, id: u32) -> Option<&str>

Get token for an ID

pub fn is_special_token(&self, token: &str) -> bool

Check if token is a special token

pub fn encode(&self, text: &str) -> Vec<u32>

Encode text to token IDs.

Arguments

• text - Text to encode

Returns

Vector of token IDs

pub fn decode(&self, ids: &[u32]) -> String

Decode token IDs to text.

Arguments

• ids - Token IDs to decode

Returns

Decoded text

pub fn encode_checked(&self, text: &str) -> Result<Vec<u32>, AprenderError>

Encode text to token IDs with error handling.

Errors

Returns error if encoding fails.

pub fn decode_checked(&self, ids: &[u32]) -> Result<String, AprenderError>

Decode token IDs to text with error handling.

Errors

Returns error if decoding fails.

Trait Implementations

impl Clone for BpeTokenizer

fn clone(&self) -> BpeTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for BpeTokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Default for BpeTokenizer

fn default() -> BpeTokenizer

Returns the “default value” for a type.