Struct DecompositionFst

pub struct DecompositionFst<D: AsRef<[u8]>> { /* private fields */ }

Decompose compound words into their parts found in a given dictionary. Useful for compressed, memory mapped dictionaries.

This implementation is based on the fst crate.

This will decompose only if a word can be fully resolved using the dictionary, any texts containing unknown “words” will be passed through as-is.

This is an adaption of the FstSegmenter from charabia for this crate. Unlike the charabia crate, this does not do splitting into unknown segments using heuristics or character splitting.

Implementations

impl DecompositionFst<Vec<u8>>

pub fn from_dictionary<I, P>(dict: I) -> Result<Self, Error>

where I: IntoIterator<Item = P>, P: AsRef<[u8]>,

Convenience contructor for DecompositionFst, takes a list of lexicographically ordered words to recognize as valid parts for decomposition.

If you’re using this constructor outside of testing and development please have a look at the DecompositionAhoCorasick implementation as it is more likely to fit your usecase of an in-memory automaton matcher.

impl<D> DecompositionFst<D>

where D: AsRef<[u8]>,

pub fn from_fst(dictionary: Fst<D>) -> Self

Construct this decomposer from an existing Fst struct.

This is the recommended way as it allows you to choose the best datastore that fits you usecase.

Trait Implementations

impl<D: Clone + AsRef<[u8]>> Clone for DecompositionFst<D>

fn clone(&self) -> DecompositionFst<D>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<D> Segmenter for DecompositionFst<D>

where D: AsRef<[u8]>,

type SubdivisionIter<'a> = IntoIter<SegmentedToken<'a>>

The iterator type returned by the subdivide function if it has multiple results.

fn subdivide<'a>( &self, token: SegmentedToken<'a>, ) -> UseOrSubdivide<SegmentedToken<'a>, IntoIter<SegmentedToken<'a>>>

A method that should split the given token into zero, one or more subtokens.