Struct AnalyzerConfig 

pub struct AnalyzerConfig {
    pub base: BaseTokenizer,
    pub language: FtsLanguage,
    pub lower_case: bool,
    pub stem: bool,
    pub remove_stop_words: bool,
    pub custom_stop_words: Option<Vec<String>>,
    pub ascii_folding: bool,
    pub max_token_length: Option<u32>,
}

Full analyzer pipeline configuration for a full-text index.

This is a backend-agnostic, plainly serializable description of the token analysis chain. The uni-store Lance backend maps it onto the underlying InvertedIndexParams; this crate stays free of any Lance dependency.

Every field carries #[serde(default)] so schemas persisted before a field existed still deserialize.

Fields

base: BaseTokenizer

Base tokenizer that splits raw text into tokens.

language: FtsLanguage

Language used for stemming and built-in stop-word lists.

lower_case: bool

Whether to lowercase tokens.

stem: bool

Whether to apply language-specific stemming.

remove_stop_words: bool

Whether to drop stop words (built-in list, or custom_stop_words).

custom_stop_words: Option<Vec<String>>

Explicit stop-word list overriding the language’s built-in list.

ascii_folding: bool

Whether to fold accented characters to ASCII (é → e).

max_token_length: Option<u32>

Drop tokens longer than this many bytes (None keeps the backend default).

Trait Implementations

impl Clone for AnalyzerConfig

fn clone(&self) -> AnalyzerConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for AnalyzerConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for AnalyzerConfig

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for AnalyzerConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl PartialEq for AnalyzerConfig

fn eq(&self, other: &AnalyzerConfig) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl Serialize for AnalyzerConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for AnalyzerConfig