Trait TokenizerEncoder 

pub trait TokenizerEncoder: Send + Sync {
    // Required method
    fn encode_ids(&self, text: &str) -> Result<Vec<u32>, String>;
}

Encode text into model token ids.

Implemented for Tokenizer; other tokenizers can implement it to back a prompt encoding.

Required Methods

fn encode_ids(&self, text: &str) -> Result<Vec<u32>, String>

Encode text into token ids without added special tokens.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementations on Foreign Types

impl TokenizerEncoder for Tokenizer

fn encode_ids(&self, text: &str) -> Result<Vec<u32>, String>

impl<T: TokenizerEncoder> TokenizerEncoder for Arc<T>

fn encode_ids(&self, text: &str) -> Result<Vec<u32>, String>

Implementors