Struct BpeDataset 

pub struct BpeDataset {
    pub tokens: Vec<usize>,
    pub tokenizer: BpeTokenizer,
}

BPE-tokenized dataset. Each token is a subword (avg ~4-5 chars). A 128-position context window sees ~500-600 characters instead of 128.

Fields

tokens: Vec<usize>

tokenizer: BpeTokenizer

Implementations

impl BpeDataset

pub fn from_text(text: &str, target_vocab: usize) -> Self

Build from text with BPE tokenization. target_vocab: number of subword tokens to learn (512, 1024, 2048).

pub fn from_jsonl(path: &Path, target_vocab: usize) -> Result<Self>

Load from a JSONL file with BPE tokenization.

pub fn from_file(path: &Path, target_vocab: usize) -> Result<Self>

Load from a plain text file with BPE tokenization.

pub fn vocab_size(&self) -> usize

pub fn len(&self) -> usize

pub fn decode(&self, tokens: &[usize]) -> String