Trait TokenizerExt 

pub trait TokenizerExt: Tokenizer {
    // Provided methods
    fn encode_with(
        &self,
        text: &str,
        cfg: &EncodeConfig,
    ) -> Result<Encoding, TokenizerError> { ... }
    fn encode_batch(
        &self,
        texts: &[&str],
        cfg: &EncodeConfig,
    ) -> Result<Vec<Encoding>, TokenizerError> { ... }
}

Higher-level encoding conveniences (special tokens, padding, truncation, batching) available on every Tokenizer.

Provided Methods

fn encode_with( &self, text: &str, cfg: &EncodeConfig, ) -> Result<Encoding, TokenizerError>

Encode a single sequence, applying special-token insertion, truncation and (fixed) padding from cfg.

Padding::Longest has no effect on a single sequence — use encode_batch for that.

Errors

Propagates Tokenizer::encode errors, or returns TokenizerError::MalformedFile if fixed padding is requested without a pad token id.

fn encode_batch( &self, texts: &[&str], cfg: &EncodeConfig, ) -> Result<Vec<Encoding>, TokenizerError>

Encode a batch of sequences, applying special tokens and truncation to each and then the batch-level Padding strategy across all of them.

Errors

Propagates Tokenizer::encode errors, or returns TokenizerError::MalformedFile if padding is requested without a pad token id.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors

impl<T: Tokenizer + ?Sized> TokenizerExt for T

Provided-method extensions shared by every Tokenizer.