Trait Tokenizer 

pub trait Tokenizer: Encoder + Decoder {
    // Provided methods
    fn validate_prefix_cache(&self) -> Result<()> { ... }
    fn with_options(self, options: TokenizerOptions) -> Self
       where Self: Sized { ... }
}

Provided Methods

fn validate_prefix_cache(&self) -> Result<()>

Validate that this tokenizer can be safely wrapped in the prefix cache.

Implementations must explicitly opt in by returning Ok(()), or return an error explaining why the tokenizer is incompatible.

fn with_options(self, options: TokenizerOptions) -> Self

where Self: Sized,

Apply construction-time TokenizerOptions.

The default implementation ignores the options — correct for tokenizers with no applicable option.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors

impl Tokenizer for CachedTokenizer

impl Tokenizer for FastTokenizer

impl Tokenizer for HuggingFaceTokenizer

impl Tokenizer for TikTokenTokenizer