Struct TokenizerConfig 

pub struct TokenizerConfig {
    pub profile: Profile,
    pub segmenter: Segmenter,
    pub folding: FoldingForm,
    pub stopwords: StopwordList,
    pub stemmer: Stemmer,
    pub decompose_words: bool,
    pub emit_catenation: bool,
    pub number_words: bool,
    pub drop_single_char_parts: bool,
    pub vocabulary: Vocabulary,
}

The complete analysis configuration.

Every field is an epoch-digest input. Changing any one of them produces a different epoch and therefore a different index meaning.

Fields

profile: Profile

The named profile this configuration started from.

segmenter: Segmenter

Word segmentation strategy.

folding: FoldingForm

Character folding form.

stopwords: StopwordList

Stopword list.

stemmer: Stemmer

Stemming algorithm.

decompose_words: bool

Split words at joiners, case changes, and letter/digit transitions.

emit_catenation: bool

Emit the joiner-free catenation of a decomposed word.

number_words: bool

Emit digit and word variants for spelled numbers.

drop_single_char_parts: bool

Drop single-LETTER decomposition parts.

The k of 401k, the t of don't, the s of investor's: one-letter parts of a decomposed word are stop-level noise in linguistic text — they carry no identity the whole word and the catenation do not already carry, and they pollute document length and document frequency. One-DIGIT parts are kept: type-2 and SARS-CoV-2 are discriminated by exactly that digit. Dropping a part leaves its position spent, exactly as a removed stopword does, so phrase adjacency and the analyzed length are unchanged. Identifier-heavy profiles keep every part: the x of x_max is a real search target in code.

vocabulary: Vocabulary

User vocabulary and synonyms.

Implementations

impl TokenizerConfig

pub fn text_default() -> Self

The general-purpose text profile.

pub fn code() -> Self

The identifier-heavy profile for source code and tickets.

pub fn voice() -> Self

The spoken-text profile.

pub fn with_vocabulary(self, vocabulary: Vocabulary) -> Self

Replaces the user vocabulary.

pub fn epoch(&self) -> TokenizerEpoch

Computes the epoch digest for this configuration.

Trait Implementations

impl Clone for TokenizerConfig

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenizerConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for TokenizerConfig

impl PartialEq for TokenizerConfig

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for TokenizerConfig