Struct tantivy_analysis_contrib::commons::LengthTokenFilter

pub struct LengthTokenFilter { /* private fields */ }

Available on crate feature commons only.

This TokenFilter filters tokens that don’t match a min or a max length (inclusive).

use tantivy_analysis_contrib::commons::LengthTokenFilter;

let length_token_filter = LengthTokenFilter::new(Some(4), Some(10));

Example

In this example, tokens There, 1 and token are filtered out because they are too short or too long.

use tantivy::tokenizer::{WhitespaceTokenizer, TextAnalyzer, Token};
use tantivy_analysis_contrib::commons::LengthTokenFilter;

let mut tmp = TextAnalyzer::builder(WhitespaceTokenizer::default())
   .filter(LengthTokenFilter::new(Some(2), Some(4)))
   .build();
let mut token_stream = tmp.token_stream("There is 1 token");

let token = token_stream.next().expect("A token should be present.");
assert_eq!(token.text, "is".to_string());

assert_eq!(None, token_stream.next());

Implementations

impl LengthTokenFilter

pub fn new(min: Option<usize>, max: Option<usize>) -> Self

Get a new token filter.

Parameters :

• min : minimum length a token should have (inclusive)
• max : maximum length a token should have (inclusive)

Trait Implementations

impl Clone for LengthTokenFilter

fn clone(&self) -> LengthTokenFilter

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for LengthTokenFilter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl TokenFilter for LengthTokenFilter

type Tokenizer<T: Tokenizer> = LengthFilterWrapper<T>

The Tokenizer type returned by this filter, typically parametrized by the underlying Tokenizer.

fn transform<T: Tokenizer>(self, token_stream: T) -> Self::Tokenizer<T>

Wraps a Tokenizer and returns a new one.

impl Copy for LengthTokenFilter