Enum TokenFilter 

pub enum TokenFilter {
    Lowercase,
    Stop {
        language: String,
        custom_words: Vec<String>,
    },
    PorterStem,
    ASCIIFolding,
    Synonym {
        synonyms: BTreeMap<String, Vec<String>>,
        synonyms_path: Option<PathBuf>,
    },
    Ngram {
        min_gram: usize,
        max_gram: usize,
        keep_short: bool,
    },
    EdgeNgram {
        min_gram: usize,
        max_gram: usize,
    },
    Length {
        min_length: usize,
        max_length: usize,
    },
}

Variants

Lowercase

Stop

Fields

language: String

custom_words: Vec<String>

PorterStem

ASCIIFolding

Synonym

Fields

synonyms: BTreeMap<String, Vec<String>>

Inline term -> [expansion, ...] mapping. Empty when the filter sources its mappings from synonyms_path instead.

synonyms_path: Option<PathBuf>

Path to a Solr / Elasticsearch-style synonym file. The file is parsed every time the filter runs so reload-on-edit is free; for production use cache the parsed map upstream. Optional path to a reloadable synonym map.

Ngram

Fields

min_gram: usize

max_gram: usize

keep_short: bool

EdgeNgram

Fields

min_gram: usize

max_gram: usize

Length

Fields

min_length: usize

max_length: usize

Implementations

impl TokenFilter

pub fn validate(&self) -> AnalysisResult<()>

Validate configuration without filtering tokens. File-backed synonym filters are read here so registration rejects missing/unreadable paths; Self::filter reads them again on every execution to detect later deletion, permission changes, and edits.

pub fn synonym_from_path<P: AsRef<Path>>( path: P, ) -> Result<Self, SynonymFileError>

Build a Synonym filter from a Solr or Elasticsearch synonym file. In this format, blank lines and # comments are skipped, a => b, c defines a one-way mapping, and a, b, c defines an equivalent group where every term expands to the other group members.

pub fn parse_synonym_file( path: &Path, ) -> Result<BTreeMap<String, Vec<String>>, SynonymFileError>

Parse a synonym file into the same shape Synonym::synonyms uses. Public so engines can pre-resolve a path to an inline map.

impl TokenFilter

pub fn filter(&self, tokens: Vec<String>) -> AnalysisResult<Vec<String>>

pub fn filter_analyzed( &self, input: AnalyzedText, ) -> AnalysisResult<AnalyzedText>

Transform tokens while retaining their source spans and graph end state.

pub fn filter_analyzed_budgeted( &self, input: Budgeted<AnalyzedText>, poll: impl FnMut() -> AnalysisResult<()>, ) -> AnalysisResult<Budgeted<AnalyzedText>>

Consume a reserved stream and retain its allowance through every common or Korean token filter.

The returned tokens retain their own terms, morphology, terminal state and vector reservations, and share existing source leases. Removed buffers release their reservations after destruction; replacements and copies reserve before allocation. Byte-limit and callback errors return no partial result. Immutable filter preparation and caller-owned configuration have separate ownership.

use uqa_analysis::{TokenFilter, Tokenizer};
use uqa_core::memory::MemoryBudget;
let budget = MemoryBudget::new(64 * 1024);
let tokens = Tokenizer::Whitespace.tokenize_with_offsets_budgeted(
    "UQA AND", &budget, || Ok(()),
)?;
let output = TokenFilter::Lowercase.filter_analyzed_budgeted(tokens, || Ok(()))?;
assert_eq!(output.tokens()[0].term(), "uqa");
drop(output);
assert_eq!(budget.used(), 0);

Trait Implementations

impl Clone for TokenFilter

fn clone(&self) -> TokenFilter

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenFilter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for TokenFilter

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for TokenFilter

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.