Struct tantivy::tokenizer::RegexTokenizer

pub struct RegexTokenizer { /* private fields */ }

Tokenize the text by using a regex pattern to split. Each match of the regex emits a distinct token, empty tokens will not be emitted. Anchors such as \A will match the text from the part where the last token was emitted or the beginning of the complete text if no token was emitted yet.

Example: 'aaa' bbb 'ccc' 'ddd' with the pattern '(?:\w*)' will be tokenized as followed:

Term	aaa	ccc	ddd
Position	1	2	3
Offsets	0,5	10,15	16,21

Example

use tantivy::tokenizer::*;

let mut tokenizer = RegexTokenizer::new(r"'(?:\w*)'").unwrap();
let mut stream = tokenizer.token_stream("'aaa' bbb 'ccc' 'ddd'");
{
    let token = stream.next().unwrap();
    assert_eq!(token.text, "'aaa'");
    assert_eq!(token.offset_from, 0);
    assert_eq!(token.offset_to, 5);
}
{
  let token = stream.next().unwrap();
    assert_eq!(token.text, "'ccc'");
    assert_eq!(token.offset_from, 10);
    assert_eq!(token.offset_to, 15);
}
{
  let token = stream.next().unwrap();
    assert_eq!(token.text, "'ddd'");
    assert_eq!(token.offset_from, 16);
    assert_eq!(token.offset_to, 21);
}
assert!(stream.next().is_none());

Implementations

impl RegexTokenizer

pub fn new(regex_pattern: &str) -> Result<RegexTokenizer>

Creates a new RegexTokenizer.

Trait Implementations

impl Clone for RegexTokenizer

fn clone(&self) -> RegexTokenizer

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Tokenizer for RegexTokenizer

type TokenStream<'a> = RegexTokenStream<'a>

The token stream returned by this Tokenizer.

fn token_stream<'a>(&'a mut self, text: &'a str) -> RegexTokenStream<'a>

Creates a token stream for a given str.