Struct tantivy_analysis_contrib::commons::EdgeNgramTokenFilter

pub struct EdgeNgramTokenFilter { /* private fields */ }

Available on crate feature commons only.

Token filter that produce ngram from the start of the token. For example, Quick will generate Q, Qu, Qui, Quic, …etc.

It is configure with two parameters:

• min edge-ngram: the number of maximum characters (e.g. with min=3, Quick will generate Qui, Quic and Quick). It must be greater than 0.
• max edge-ngram: the number of maximum characters (e.g. with max=3, Quick will generate Q, Qu and Qui. It is optional, and there is no maximum then it will generate up to the end of the token.

Example

use std::num::NonZeroUsize;
use tantivy::tokenizer::{WhitespaceTokenizer, TextAnalyzer, Token};
use tantivy_analysis_contrib::commons::EdgeNgramTokenFilter;

let mut tmp = TextAnalyzer::builder(WhitespaceTokenizer::default())
   .filter(EdgeNgramTokenFilter::new(NonZeroUsize::new(2).unwrap(), NonZeroUsize::new(4), false)?)
   .build();
let mut token_stream = tmp.token_stream("Quick");

let token = token_stream.next().expect("A token should be present.");
assert_eq!(token.text, "Qu".to_string());
let token = token_stream.next().expect("A token should be present.");
assert_eq!(token.text, "Qui".to_string());
let token = token_stream.next().expect("A token should be present.");
assert_eq!(token.text, "Quic".to_string());

assert_eq!(None, token_stream.next());

This token filter is useful to do a “starts with” therefor a “search as you type”.

It is also easy to have an efficient “ends with” by adding the ReverseTokenFilter before the edge ngram filter.

How to use it

To use it, you should have another pipeline at search time that does not include the edge-ngram filter. Otherwise, you’ll get irrelevant results. Please see the example in source repository for a way to do it.

Implementations

impl EdgeNgramTokenFilter

pub fn new( min: NonZeroUsize, max: Option<NonZeroUsize>, keep_original_token: bool ) -> Result<Self, EdgeNgramError>

Create a new EdgeNgramTokenFilter with the min and max ngram provided.

Parameters

• min : minimum edge-ngram.
• max : maximum edge-ngram. It must be greater or equals to min. Provide None for unlimited.
• keep_original_token: the complete token will also be output if the length is greater than max.

Trait Implementations

impl Clone for EdgeNgramTokenFilter

fn clone(&self) -> EdgeNgramTokenFilter

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for EdgeNgramTokenFilter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<NonZero<usize>> for EdgeNgramTokenFilter

fn from(ngram: NonZeroUsize) -> Self

Converts to this type from the input type.

impl Hash for EdgeNgramTokenFilter

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl Ord for EdgeNgramTokenFilter

fn cmp(&self, other: &EdgeNgramTokenFilter) -> Ordering

This method returns an Ordering between self and other.

fn max(self, other: Self) -> Self

where Self: Sized,

Compares and returns the maximum of two values.

fn min(self, other: Self) -> Self

where Self: Sized,

Compares and returns the minimum of two values.

fn clamp(self, min: Self, max: Self) -> Self

where Self: Sized + PartialOrd,

Restrict a value to a certain interval.

impl PartialEq for EdgeNgramTokenFilter

fn eq(&self, other: &EdgeNgramTokenFilter) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialOrd for EdgeNgramTokenFilter

fn partial_cmp(&self, other: &EdgeNgramTokenFilter) -> Option<Ordering>

This method returns an ordering between self and other values if one exists.

fn lt(&self, other: &Rhs) -> bool

This method tests less than (for self and other) and is used by the < operator.

fn le(&self, other: &Rhs) -> bool

This method tests less than or equal to (for self and other) and is used by the <= operator.

fn gt(&self, other: &Rhs) -> bool

This method tests greater than (for self and other) and is used by the > operator.

fn ge(&self, other: &Rhs) -> bool

This method tests greater than or equal to (for self and other) and is used by the >= operator.

impl TokenFilter for EdgeNgramTokenFilter

type Tokenizer<T: Tokenizer> = EdgeNgramFilterWrapper<T>

The Tokenizer type returned by this filter, typically parametrized by the underlying Tokenizer.

fn transform<T: Tokenizer>(self, tokenizer: T) -> Self::Tokenizer<T>

Wraps a Tokenizer and returns a new one.

impl Copy for EdgeNgramTokenFilter

impl Eq for EdgeNgramTokenFilter

impl StructuralPartialEq for EdgeNgramTokenFilter