Struct tantivy::tokenizer::NgramTokenizer

pub struct NgramTokenizer { /* fields omitted */ }

Tokenize the text by splitting words into n-grams of the given size(s)

With this tokenizer, the position is always 0. Beware however, in presence of multiple value for the same field, the position will be POSITION_GAP * index of value.

Example 1: hello would be tokenized as (min_gram: 2, max_gram: 3, prefix_only: false)

Term	he	hel	el	ell	ll	llo	lo
Position	0	0	0	0	0	0	0
Offsets	0,2	0,3	1,3	1,4	2,4	2,5	3,5

Example 2: hello would be tokenized as (min_gram: 2, max_gram: 5, prefix_only: true)

Term	he	hel	hell	hello
Position	0	0	0	0
Offsets	0,2	0,3	0,4	0,5

Example 3: hεllo (non-ascii) would be tokenized as (min_gram: 2, max_gram: 5, prefix_only: true)

Term	hε	hεl	hεll	hεllo
Position	0	0	0	0
Offsets	0,3	0,4	0,5	0,6

Example

use tantivy::tokenizer::*;
let tokenizer = NgramTokenizer::new(2, 3, false);
let mut stream = tokenizer.token_stream("hello");
{
    let token = stream.next().unwrap();
    assert_eq!(token.text, "he");
    assert_eq!(token.offset_from, 0);
    assert_eq!(token.offset_to, 2);
}
{
  let token = stream.next().unwrap();
    assert_eq!(token.text, "hel");
    assert_eq!(token.offset_from, 0);
    assert_eq!(token.offset_to, 3);
}
{
  let token = stream.next().unwrap();
    assert_eq!(token.text, "el");
    assert_eq!(token.offset_from, 1);
    assert_eq!(token.offset_to, 3);
}
{
  let token = stream.next().unwrap();
    assert_eq!(token.text, "ell");
    assert_eq!(token.offset_from, 1);
    assert_eq!(token.offset_to, 4);
}
{
  let token = stream.next().unwrap();
    assert_eq!(token.text, "ll");
    assert_eq!(token.offset_from, 2);
    assert_eq!(token.offset_to, 4);
}
{
  let token = stream.next().unwrap();
    assert_eq!(token.text, "llo");
    assert_eq!(token.offset_from, 2);
    assert_eq!(token.offset_to, 5);
}
{
  let token = stream.next().unwrap();
  assert_eq!(token.text, "lo");
  assert_eq!(token.offset_from, 3);
  assert_eq!(token.offset_to, 5);
}
assert!(stream.next().is_none());

Methods

impl NgramTokenizer

pub fn new(
    min_gram: usize,
    max_gram: usize,
    prefix_only: bool
) -> NgramTokenizer

Configures a new Ngram tokenizer

pub fn all_ngrams(min_gram: usize, max_gram: usize) -> NgramTokenizer

Create a NGramTokenizer which generates tokens for all inner ngrams.

This is as opposed to only prefix ngrams .

pub fn prefix_only(min_gram: usize, max_gram: usize) -> NgramTokenizer

Create a NGramTokenizer which only generates tokens for the prefix ngrams.

Trait Implementations

impl<'a> Tokenizer<'a> for NgramTokenizer

type TokenStreamImpl = NgramTokenStream<'a>

Type associated to the resulting tokenstream tokenstream.

fn token_stream(&self, text: &'a str) -> Self::TokenStreamImpl

Creates a token stream for a given str.

fn filter<NewFilter>(
    self,
    new_filter: NewFilter
) -> ChainTokenizer<NewFilter, Self> where
    NewFilter: TokenFilter<Self::TokenStreamImpl>, 

Appends a token filter to the current tokenizer.

impl Clone for NgramTokenizer

fn clone(&self) -> NgramTokenizer

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.