Struct BPETokenizer 

pub struct BPETokenizer { /* private fields */ }

BPE (Byte Pair Encoding) tokenizer

Implementations

impl BPETokenizer

pub fn new(config: TokenizerConfig) -> Self

Create a new BPE tokenizer

pub fn vocab(&self) -> &HashMap<String, TokenId>

Borrow the learned token → id vocabulary map.

Exposed so callers (e.g. apr tokenize train) can emit the HuggingFace vocab.json artifact mandated by contracts/tokenizer-bpe-v1.yaml without serializing the whole BPETokenizer struct. Read-only by design — training and encoding continue to own the HashMap.

pub fn merges(&self) -> &[(String, String)]

Borrow the ordered list of learned merge rules ((left, right) pairs in merge order).

Exposed so callers can write the HuggingFace merges.txt artifact. The order is load-bearing: merges.txt consumers apply pairs top-to-bottom.

pub fn save(&self, path: &str) -> Result<()>

Save tokenizer to file

pub fn load(path: &str) -> Result<Self>

Load tokenizer from file

pub fn from_vocab_merges( vocab_path: &str, merges_path: &str, config: TokenizerConfig, ) -> Result<Self>

Reconstruct a trained BPETokenizer from the HuggingFace-style pair of vocab.json + merges.txt emitted by apr tokenize train.

Format

• vocab.json: JSON object mapping token string → token id (u32). Order is informational; the loader inverts the map to build id_to_token.
• merges.txt: leading #version: 0.2\n header line, then one merge per line in apply order. Each line is "<left> <right>" with a single ASCII space separator (tokens in the aprender-train hex representation never contain spaces, so space-split is unambiguous).

Parameters

• vocab_path: path to vocab.json
• merges_path: path to merges.txt
• config: caller-supplied config (normalization, special tokens, etc.) since those fields are not recorded in the HF-style files. MUST match the config used at training time — wrong normalization here produces silently-wrong encodings.

Invariants

• C-PRETOK-BIN INV-PRETOK-001: every loaded vocab id < returned tokenizer’s vocab_size().
• Every merge’s (left, right) concatenation is present in the loaded vocab (otherwise applying the merge during encode would produce a token the vocab cannot resolve). Enforced; mismatch returns an error.

Trait Implementations

impl Clone for BPETokenizer

fn clone(&self) -> BPETokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for BPETokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for BPETokenizer

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for BPETokenizer

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl Tokenizer for BPETokenizer

fn train(&mut self, corpus: &[&str]) -> Result<()>

Train the tokenizer on a corpus

fn encode(&self, text: &str) -> Result<Vec<TokenId>>

Encode text to token IDs

fn decode(&self, ids: &[TokenId]) -> Result<String>

Decode token IDs to text

fn vocab_size(&self) -> usize

Get vocabulary size

fn is_trained(&self) -> bool

Check if tokenizer is trained

fn id_to_token(&self, id: TokenId) -> Option<&str>

Get token for ID

fn token_to_id(&self, token: &str) -> Option<TokenId>

Get ID for token