Struct UnigramTokenizer 

pub struct UnigramTokenizer { /* private fields */ }

A simplified unigram-style tokenizer.

Implementations

impl UnigramTokenizer

pub fn new(vocab: HashMap<String, f32>) -> Self

Creates a new unigram tokenizer from a vocabulary with scores.

pub fn from_tokens(tokens: &[&str]) -> Self

Creates a tokenizer from a list of tokens (equal scores).

Trait Implementations

impl Clone for UnigramTokenizer

fn clone(&self) -> UnigramTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for UnigramTokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Tokenizer for UnigramTokenizer

fn tokenize(&self, text: &str) -> Vec<String>

Tokenizes a string into tokens.

fn encode(&self, text: &str, vocab: &Vocab) -> Vec<usize>

Tokenizes and encodes to indices using a vocabulary.