Struct MappingTokenFilter 

pub struct MappingTokenFilter { /* private fields */ }

Replace characters with the specified character mappings.

Implementations

impl MappingTokenFilter

pub fn new(mapping: HashMap<String, String>) -> LinderaResult<Self>

Create a new MappingTokenFilter from a surface-to-replacement mapping.

Arguments

• mapping - A map from surface text to its replacement text. Keys must be non-empty; an empty key would match at every byte position under the leftmost-longest search strategy used by apply, which is never a meaningful mapping and is therefore rejected.

Returns

A MappingTokenFilter, or an error if mapping contains an empty key or the underlying Aho-Corasick automaton fails to build.

pub fn from_config(config: &MappingTokenFilterConfig) -> LinderaResult<Self>

Trait Implementations

impl Clone for MappingTokenFilter

fn clone(&self) -> MappingTokenFilter

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl TokenFilter for MappingTokenFilter

fn apply(&self, tokens: &mut Vec<Token<'_>>) -> LinderaResult<()>

Apply the filter to each token’s surface, in place.

Performs a single leftmost-longest pass over each token’s surface with the underlying Aho-Corasick automaton: at each position the longest configured key wins, matches never overlap, and scanning resumes immediately after each match.

Arguments

• tokens - The tokens to filter; each token’s surface is replaced in place with the mapped text.

fn name(&self) -> &'static str