Struct DynamicStemmer 

pub struct DynamicStemmer { /* private fields */ }

Stemmer whose language is selected per call from the tokenizer hint.

The hint is a comma-separated list of language codes or names ("ru,en"). Each token is stemmed with the first hinted language whose script matches the token’s script; tokens of a script no hinted language covers are kept as cleaned (lowercased, punctuation stripped) text. Without a hint the default language applies, or plain cleaning when it is None.

Because Snowball stemmers are script-local, one field can hold mixed Cyrillic/Latin documents and still stem each part correctly.

Declared in SDL as text<stem(by: <field>, default: <language|simple>)>; see TokenizerSpec.

Implementations

impl DynamicStemmer

pub fn new(default: Option<Language>) -> Self

Create a dynamic stemmer; default applies when no hint is present.

pub fn default_language(&self) -> Option<Language>

Default language used when no hint is given.

pub fn parse_hint(hint: &str) -> Vec<Language>

Parse a hint into the ordered list of recognised languages.

Trait Implementations

impl Clone for DynamicStemmer

fn clone(&self) -> DynamicStemmer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DynamicStemmer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for DynamicStemmer

fn default() -> DynamicStemmer

Returns the “default value” for a type.

impl Tokenizer for DynamicStemmer

fn tokenize(&self, text: &str) -> Vec<Token>

Tokenize the input text into a vector of tokens

fn tokenize_hinted(&self, text: &str, hint: Option<&str>) -> Vec<Token>

Tokenize with an optional caller-supplied hint.