Enum TokenizerSpec 

pub enum TokenizerSpec {
    Named(String),
    Lex(LexOptions),
}

Parsed form of a tokenizer name in the schema: a registered name (simple, en_stem, …) or a lex(...) spec. The canonical string form is stored in FieldEntry::tokenizer.

Variants

Named(String)

Lex(LexOptions)

Implementations

impl TokenizerSpec

pub fn parse(spec: &str) -> Result<TokenizerSpec, String>

Parse a tokenizer name or lex(...) spec.

pub fn lex(&self) -> Option<&LexOptions>

The options of a lex(...) spec.

pub fn hint_field(&self) -> Option<&str>

Field whose values hint the tokenizer, for lex specs with by.

pub fn keeps_original(&self) -> bool

Whether the spec indexes originals next to their variants, so exact (phrase, term) queries match the written form and match queries the stem.

pub fn dynamic_tokenizer(&self) -> Option<BoxedTokenizer>

Build the tokenizer described by a lex spec.

Trait Implementations

impl Clone for TokenizerSpec

fn clone(&self) -> TokenizerSpec

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenizerSpec

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for TokenizerSpec

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for TokenizerSpec

impl PartialEq for TokenizerSpec

fn eq(&self, other: &TokenizerSpec) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for TokenizerSpec