Enum TokenizerSpec 

pub enum TokenizerSpec {
    Named(String),
    DynamicStem {
        by: String,
        default: Option<Language>,
        stop_words: bool,
        segmenter: Segmenter,
    },
}

Parsed form of a tokenizer name in the schema.

Plain names refer to a registered tokenizer (simple, en_stem, …). stem(by: <field>, default: <language|simple>, stop_words: <bool>) declares a DynamicStemmer whose per-document hint is read from <field> in the same document. The canonical string form is stored in FieldEntry::tokenizer, so index metadata needs no new field; specs without stop_words render exactly as before.

Variants

Named(String)

A registered tokenizer name.

DynamicStem

Dynamic stemmer hinted by another field of the document.

Fields

by: String

Field whose text values supply the language hint.

default: Option<Language>

Language applied when the hint field is absent; None = simple.

stop_words: bool

Drop the routed language’s stop words (positions keep their gaps).

segmenter: Segmenter

Word segmentation; Simple is the historic behaviour.

Implementations

impl TokenizerSpec

pub fn parse(spec: &str) -> Result<TokenizerSpec, String>

Parse a tokenizer name or stem(...) spec.

pub fn hint_field(&self) -> Option<&str>

Field whose values hint the tokenizer, for dynamic specs.

pub fn dynamic_tokenizer(&self) -> Option<BoxedTokenizer>

Build the tokenizer described by a dynamic spec.

Trait Implementations

impl Clone for TokenizerSpec

fn clone(&self) -> TokenizerSpec

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenizerSpec

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for TokenizerSpec

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for TokenizerSpec

impl PartialEq for TokenizerSpec

fn eq(&self, other: &TokenizerSpec) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for TokenizerSpec