Trait TokenFilter

pub trait TokenFilter:
    'static
    + Send
    + Sync {
    type Tokenizer<T: Tokenizer>: Tokenizer;

    // Required method
    fn transform<T: Tokenizer>(self, tokenizer: T) -> Self::Tokenizer<T>;
}

Trait for the pluggable components of Tokenizers.

Required Associated Types

type Tokenizer<T: Tokenizer>: Tokenizer

The Tokenizer type returned by this filter, typically parametrized by the underlying Tokenizer.

Required Methods

fn transform<T: Tokenizer>(self, tokenizer: T) -> Self::Tokenizer<T>

Wraps a Tokenizer and returns a new one.

Dyn Compatibility

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety", so this trait is not object safe.

Implementors