Skip to main content

SubwordTokenizer

textprep::subword

Trait SubwordTokenizer 

Source 
pub trait SubwordTokenizer: Send + Sync {
    // Required method
    fn tokenize(&self, text: &str) -> Vec<u32>;
}
Expand description

Trait for subword tokenizers.

Returns a sequence of token IDs for input text.

Required Methods§

Source

fn tokenize(&self, text: &str) -> Vec<u32>

Implementors§