Struct TokenDataset 

pub struct TokenDataset {
    pub tokens: Vec<usize>,
    pub tokenizer: CharTokenizer,
}

A tokenized text dataset for training.

Fields

tokens: Vec<usize>

tokenizer: CharTokenizer

Implementations

impl TokenDataset

pub fn from_text(text: &str) -> Self

Build from text (tokenizes immediately).

pub fn shakespeare() -> Self

Build from the built-in Shakespeare corpus.

pub fn vocab_size(&self) -> usize

Vocabulary size.

pub fn len(&self) -> usize

Total token count.

pub fn sample_window(&self, length: usize, seed: u64) -> &[usize]

Sample a deterministic window of length tokens. The start position is derived from seed for reproducibility.

pub fn num_windows(&self, length: usize) -> usize

Number of possible windows of given length.

pub fn decode(&self, tokens: &[usize]) -> String

Decode token indices back to text.

pub fn from_jsonl(path: &Path) -> Result<Self>

Load from a JSONL file where each line is a quoted text string. Lines are unescaped (\n → newline) and concatenated with newline separators.

pub fn from_file(path: &Path) -> Result<Self>

Load from a plain text file.

pub fn from_dir(path: &Path) -> Result<Self>

Load from a directory of .txt files (concatenated with newline separators).

pub fn from_path(path: &Path) -> Result<Self>

Auto-detect format from path: .jsonl, .txt file, or directory of .txt files.