Skip to main content

Model

morpheme::traits

Trait Model 

Source 
pub trait Model {
    // Required methods
    fn tokenize(&self, sequence: &str) -> Result<Vec<Token>>;
    fn token_to_id(&self, token: &str) -> Option<u32>;
    fn id_to_token(&self, id: u32) -> Option<String>;
    fn vocab(&self) -> HashMap<String, u32>;
    fn vocab_size(&self) -> usize;
}
Expand description

Maps a piece of text to tokens.

Required Methods§

Source

fn tokenize(&self, sequence: &str) -> Result<Vec<Token>>

Tokenize sequence. Token offsets are byte offsets into sequence.

Source

fn token_to_id(&self, token: &str) -> Option<u32>

Id of token, if it is in the vocabulary.

Source

fn id_to_token(&self, id: u32) -> Option<String>

Token for id, if it is in the vocabulary.

Source

fn vocab(&self) -> HashMap<String, u32>

The full vocabulary.

Source

fn vocab_size(&self) -> usize

Vocabulary size.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§