summavy_tokenizer_api

Trait TokenFilter

Source 
pub trait TokenFilter:
    'static
    + Send
    + Sync
    + TokenFilterClone {
    // Required method
    fn transform<'a>(
        &self,
        token_stream: BoxTokenStream<'a>,
    ) -> BoxTokenStream<'a>;
}
Expand description

Trait for the pluggable components of Tokenizers.

Required Methods§

Source

fn transform<'a>(&self, token_stream: BoxTokenStream<'a>) -> BoxTokenStream<'a>

Wraps a token stream and returns the modified one.

Implementors§