Enum TokenizerMode 

pub enum TokenizerMode {
    Whitespace,
    WordBoundary,
    NGram,
}

Tokenization strategy.

Variants

Whitespace

Split on whitespace characters.

WordBoundary

Split on non-alphanumeric boundaries.

NGram

Generate character n-grams of a fixed size.

Trait Implementations

impl Clone for TokenizerMode

fn clone(&self) -> TokenizerMode

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for TokenizerMode

impl Debug for TokenizerMode

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for TokenizerMode

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for TokenizerMode

impl PartialEq for TokenizerMode

fn eq(&self, other: &TokenizerMode) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for TokenizerMode