Skip to main content

Tokenizer

Trait Tokenizer 

Source
pub trait Tokenizer: Send + Sync {
    // Required methods
    fn encode(&self, text: &str) -> Vec<TokenId>;
    fn decode(&self, tokens: &[TokenId]) -> String;
    fn vocab_size(&self) -> usize;
    fn token_to_piece(&self, token: TokenId) -> Option<&str>;
    fn piece_to_token(&self, piece: &str) -> Option<TokenId>;
}
Expand description

Standard abstract interface for tokenizers in spec-stream.

Required Methods§

Source

fn encode(&self, text: &str) -> Vec<TokenId>

Encodes a UTF-8 text string into a sequence of token IDs.

Source

fn decode(&self, tokens: &[TokenId]) -> String

Decodes a sequence of token IDs back into a UTF-8 string.

Source

fn vocab_size(&self) -> usize

Total vocabulary size.

Source

fn token_to_piece(&self, token: TokenId) -> Option<&str>

Maps a token ID to its string piece representation, if present.

Source

fn piece_to_token(&self, piece: &str) -> Option<TokenId>

Maps a string piece to its token ID, if present.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§