Struct DecompositionAhoCorasick

pub struct DecompositionAhoCorasick { /* private fields */ }

Decompose compound words into their parts found in a given dictionary. Useful for small or on the fly generated dictionaries.

This implementation is based on the aho_corasick crate.

This will decompose only if a word can be fully resolved using the dictionary, any texts containing unknown “words” will be passed through as-is.

This is an adaption of the SplitCompoundWords filter from tantivy for this crate.

Implementations

impl DecompositionAhoCorasick

pub fn from_dictionary<I, P>(dict: I) -> Result<Self, BuildError>

where I: IntoIterator<Item = P>, P: AsRef<[u8]>,

Create a filter from a given dictionary.

The dictionary will be used to construct an AhoCorasick automaton with reasonable defaults. See from_automaton if more control over its construction is required.

pub fn from_automaton(dictionary: AhoCorasick) -> Self

Create a filter from a given automaton.

The automaton should use one of the leftmost-first match kinds and it should not be anchored.

Trait Implementations

impl Clone for DecompositionAhoCorasick

fn clone(&self) -> DecompositionAhoCorasick

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DecompositionAhoCorasick

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Segmenter for DecompositionAhoCorasick

type SubdivisionIter<'a> = IntoIter<SegmentedToken<'a>>

The iterator type returned by the subdivide function if it has multiple results.

fn subdivide<'a>( &self, token: SegmentedToken<'a>, ) -> UseOrSubdivide<SegmentedToken<'a>, Self::SubdivisionIter<'a>>

A method that should split the given token into zero, one or more subtokens.