Struct CharTokenizer 

pub struct CharTokenizer { /* private fields */ }

Implementations

impl CharTokenizer

pub fn from_corpus(text: &str) -> Self

Build the vocab from the sorted unique characters of text (nanoGPT’s rule). Ids are indices into that sorted order.

pub fn chars(&self) -> &[char]

The vocabulary in id order.

pub fn unknown_chars(&self, text: &str) -> Vec<char>

Characters of text that are outside the vocabulary, deduplicated and in first-appearance order. Callers that want to warn a user before CharTokenizer::encode rejects the input use this.

pub fn encode_lossy(&self, text: &str) -> Vec<u32>

Encode, silently dropping characters outside the vocabulary. Pair with CharTokenizer::unknown_chars to tell the user what was dropped — the strict CharTokenizer::encode is the default for good reason.

pub fn to_json(&self) -> String

{"<char>": id} — the same shape as GPT-2’s vocab.json, so the same loader path and the same filename work for both tokenizers.

pub fn from_json(s: &str) -> Result<Self>

pub fn save_json(&self, path: impl AsRef<Path>) -> Result<()>

Write vocab.json next to a checkpoint. Re-deriving the vocab at inference time from a different corpus would silently shift every id, so it ships with the weights.

pub fn from_json_file(path: impl AsRef<Path>) -> Result<Self>

Trait Implementations

impl Tokenizer for CharTokenizer

fn encode(&self, text: &str) -> Result<Vec<u32>>

Unknown characters are an error rather than a silent drop: a prompt containing them would otherwise generate a plausible continuation of something the user did not type. See CharTokenizer::encode_lossy.

fn decode_bytes(&self, ids: &[u32]) -> Vec<u8>

A char vocab has no partial-UTF-8 problem — every token is a whole character — but the streaming path calls this for both tokenizers.

fn decode(&self, ids: &[u32]) -> String

fn vocab_size(&self) -> usize