Trait TokenFilter 

pub trait TokenFilter:
    'static
    + Send
    + Sync
    + TokenFilterClone {
    // Required method
    fn transform<'a>(
        &self,
        token_stream: BoxTokenStream<'a>,
    ) -> BoxTokenStream<'a>;
}

Trait for the pluggable components of Tokenizers.

Required Methods

fn transform<'a>(&self, token_stream: BoxTokenStream<'a>) -> BoxTokenStream<'a>

Wraps a token stream and returns the modified one.

Implementors