Struct CharTokenizer 

pub struct CharTokenizer {
    pub char_to_idx: Vec<(char, usize)>,
    pub idx_to_char: Vec<char>,
    pub vocab_size: usize,
}

Character-level tokenizer. Maps unique characters to indices.

Fields

char_to_idx: Vec<(char, usize)>

idx_to_char: Vec<char>

vocab_size: usize

Implementations

impl CharTokenizer

pub fn from_text(text: &str) -> Self

Build tokenizer from a text corpus. Vocabulary = unique characters.

pub fn encode(&self, text: &str) -> Vec<usize>

pub fn decode(&self, tokens: &[usize]) -> String

Trait Implementations

impl Clone for CharTokenizer

fn clone(&self) -> CharTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.