Trait Segmenter

pub trait Segmenter {
    type SubdivisionIter<'a>: Iterator<Item = SegmentedToken<'a>>;

    // Required method
    fn subdivide<'a>(
        &self,
        token: SegmentedToken<'a>,
    ) -> UseOrSubdivide<SegmentedToken<'a>, Self::SubdivisionIter<'a>> ;
}

Allows the segmenter to be part of a SubdivisionMap iterator and be chained using the chain_segmenter methods.

Required Associated Types

type SubdivisionIter<'a>: Iterator<Item = SegmentedToken<'a>>

The iterator type returned by the subdivide function if it has multiple results.

Setting this to IntoIter<SegmentedToken<'a>> for std::vec::IntoIter is a sane default. Allowing you to generate a valid iterator calling into_iter() on a Vec.

Required Methods

fn subdivide<'a>( &self, token: SegmentedToken<'a>, ) -> UseOrSubdivide<SegmentedToken<'a>, Self::SubdivisionIter<'a>>

A method that should split the given token into zero, one or more subtokens.

Have a look at the UseOrSubdivide enum, which is similar to an Option, but can also return an owned iterator with multiple items in it.

Dyn Compatibility

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety", so this trait is not object safe.

Implementors

impl Segmenter for DecompositionAhoCorasick

type SubdivisionIter<'a> = IntoIter<SegmentedToken<'a>>

impl Segmenter for NaiveWordSplitter

type SubdivisionIter<'a> = IntoIter<SegmentedToken<'a>>

impl Segmenter for UnicodeSentenceSplitter

type SubdivisionIter<'a> = IntoIter<SegmentedToken<'a>>

impl Segmenter for UnicodeWordSplitter

type SubdivisionIter<'a> = IntoIter<SegmentedToken<'a>>

impl<D> Segmenter for DecompositionFst<D>

where D: AsRef<[u8]>,

type SubdivisionIter<'a> = IntoIter<SegmentedToken<'a>>

impl<T> Segmenter for T

where T: Augmenter,

type SubdivisionIter<'a> = IntoIter<SegmentedToken<'a>>