pub trait Tokenizer: Send + Sync {
// Required methods
fn encode(&self, text: &str) -> Vec<TokenId>;
fn decode(&self, tokens: &[TokenId]) -> String;
fn vocab_size(&self) -> usize;
fn token_to_piece(&self, token: TokenId) -> Option<&str>;
fn piece_to_token(&self, piece: &str) -> Option<TokenId>;
}Expand description
Standard abstract interface for tokenizers in spec-stream.
Required Methods§
Sourcefn encode(&self, text: &str) -> Vec<TokenId>
fn encode(&self, text: &str) -> Vec<TokenId>
Encodes a UTF-8 text string into a sequence of token IDs.
Sourcefn decode(&self, tokens: &[TokenId]) -> String
fn decode(&self, tokens: &[TokenId]) -> String
Decodes a sequence of token IDs back into a UTF-8 string.
Sourcefn vocab_size(&self) -> usize
fn vocab_size(&self) -> usize
Total vocabulary size.
Sourcefn token_to_piece(&self, token: TokenId) -> Option<&str>
fn token_to_piece(&self, token: TokenId) -> Option<&str>
Maps a token ID to its string piece representation, if present.
Sourcefn piece_to_token(&self, piece: &str) -> Option<TokenId>
fn piece_to_token(&self, piece: &str) -> Option<TokenId>
Maps a string piece to its token ID, if present.
Dyn Compatibility§
This trait is dyn compatible.
In older versions of Rust, dyn compatibility was called "object safety".