Struct TextVocabulary 

pub struct TextVocabulary { /* private fields */ }

Character vocabulary for text generation tasks.

Maps characters to indices and vice versa.

Implementations

impl TextVocabulary

pub fn from_text(text: &str) -> Self

Create vocabulary from text, extracting unique characters.

pub fn from_chars(chars: &[char]) -> Self

Create vocabulary from explicit character list.

pub fn char_to_index(&self, ch: char) -> Option<usize>

Get index for a character.

pub fn index_to_char(&self, idx: usize) -> Option<char>

Get character for an index.

pub fn size(&self) -> usize

Get vocabulary size.

pub fn contains(&self, ch: char) -> bool

Check if character is in vocabulary.

pub fn chars(&self) -> Vec<char>

Get all characters in vocabulary order.

pub fn encode(&self, text: &str) -> Vec<usize>

Encode string to indices.

pub fn decode(&self, indices: &[usize]) -> String

Decode indices to string.

Trait Implementations

impl Clone for TextVocabulary

fn clone(&self) -> TextVocabulary

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TextVocabulary

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.