Struct tantivy::tokenizer::SplitCompoundWords

pub struct SplitCompoundWords { /* private fields */ }

A TokenFilter which splits compound words into their parts based on a given dictionary.

Words only will be split if they can be fully decomposed into consecutive matches into the given dictionary.

This is mostly useful to split compound nouns common to many Germanic languages into their constituents.

Example

The quality of the dictionary determines the quality of the splits, e.g. the missing stem “back” of “backen” implies that “brotbackautomat” is not split in the following example.

use tantivy::tokenizer::{SimpleTokenizer, SplitCompoundWords, TextAnalyzer};

let mut tokenizer =
       TextAnalyzer::builder(SimpleTokenizer::default())
       .filter(
           SplitCompoundWords::from_dictionary([
                "dampf", "schiff", "fahrt", "brot", "backen", "automat",
           ])
           .unwrap()
       )
       .build();
{
    let mut stream = tokenizer.token_stream("dampfschifffahrt");
    assert_eq!(stream.next().unwrap().text, "dampf");
    assert_eq!(stream.next().unwrap().text, "schiff");
    assert_eq!(stream.next().unwrap().text, "fahrt");
    assert_eq!(stream.next(), None);
}
let mut stream = tokenizer.token_stream("brotbackautomat");
assert_eq!(stream.next().unwrap().text, "brotbackautomat");
assert_eq!(stream.next(), None);

Implementations

impl SplitCompoundWords

pub fn from_dictionary<I, P>(dict: I) -> Result<Self>

where I: IntoIterator<Item = P>, P: AsRef<[u8]>,

Create a filter from a given dictionary.

The dictionary will be used to construct an AhoCorasick automaton with reasonable defaults. See from_automaton if more control over its construction is required.

pub fn from_automaton(dict: AhoCorasick) -> Self

Create a filter from a given automaton.

The automaton should use one of the leftmost-first match kinds and it should not be anchored.

Trait Implementations

impl Clone for SplitCompoundWords

fn clone(&self) -> SplitCompoundWords

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl TokenFilter for SplitCompoundWords

type Tokenizer<T: Tokenizer> = SplitCompoundWordsFilter<T>

The Tokenizer type returned by this filter, typically parametrized by the underlying Tokenizer.

fn transform<T: Tokenizer>(self, tokenizer: T) -> SplitCompoundWordsFilter<T>

Wraps a Tokenizer and returns a new one.