Enum TokenFilter 

pub enum TokenFilter {
    Lowercase,
    Stop {
        language: String,
        custom_words: Vec<String>,
    },
    PorterStem,
    ASCIIFolding,
    Synonym {
        synonyms: BTreeMap<String, Vec<String>>,
        synonyms_path: Option<PathBuf>,
    },
    Ngram {
        min_gram: usize,
        max_gram: usize,
        keep_short: bool,
    },
    EdgeNgram {
        min_gram: usize,
        max_gram: usize,
    },
    Length {
        min_length: usize,
        max_length: usize,
    },
}

Variants

Lowercase

Stop

Fields

language: String

custom_words: Vec<String>

PorterStem

ASCIIFolding

Synonym

Fields

synonyms: BTreeMap<String, Vec<String>>

Inline term -> [expansion, ...] mapping. Empty when the filter sources its mappings from synonyms_path instead.

synonyms_path: Option<PathBuf>

Path to a Solr / Elasticsearch-style synonym file. The file is parsed every time the filter runs so reload-on-edit is free; for production use cache the parsed map upstream. Optional path to a reloadable synonym map.

Ngram

Fields

min_gram: usize

max_gram: usize

keep_short: bool

EdgeNgram

Fields

min_gram: usize

max_gram: usize

Length

Fields

min_length: usize

max_length: usize

Implementations

impl TokenFilter

pub fn validate(&self) -> AnalysisResult<()>

Validate configuration without filtering tokens. File-backed synonym filters are read here so registration rejects missing/unreadable paths; Self::filter reads them again on every execution to detect later deletion, permission changes, and edits.

pub fn synonym_from_path<P: AsRef<Path>>( path: P, ) -> Result<Self, SynonymFileError>

Build a Synonym filter from a Solr or Elasticsearch synonym file. In this format, blank lines and # comments are skipped, a => b, c defines a one-way mapping, and a, b, c defines an equivalent group where every term expands to the other group members.

pub fn parse_synonym_file( path: &Path, ) -> Result<BTreeMap<String, Vec<String>>, SynonymFileError>

Parse a synonym file into the same shape Synonym::synonyms uses. Public so engines can pre-resolve a path to an inline map.

impl TokenFilter

pub fn filter(&self, tokens: Vec<String>) -> AnalysisResult<Vec<String>>

Trait Implementations

impl Clone for TokenFilter

fn clone(&self) -> TokenFilter

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenFilter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for TokenFilter

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for TokenFilter

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.