pub enum TsConfig {
Simple,
English,
Spanish,
French,
German,
}Expand description
v7.12.1 — supported tokeniser configs.
Variants§
Simple
simple / pg_catalog.simple — lowercase + split, no
stopword drop, no stem.
English
english / pg_catalog.english — lowercase + split +
stopword drop + Porter v1 stem.
Spanish
v7.38.18 — spanish: lowercase + split + Snowball’s 313-word
Spanish stopword list + the Snowball Spanish stem.
French
v7.38.18 — french: Snowball’s 155-word French stopword list +
the Snowball French stem.
German
v7.38.18 — german: Snowball’s 231-word German stopword list +
the Snowball German stem.
Implementations§
Source§impl TsConfig
impl TsConfig
Sourcepub const fn stems(self) -> bool
pub const fn stems(self) -> bool
v7.38.18 — does this configuration stem at all?
The four places that decide stopwords and stemming used to ask
config.stems(), a two-valued question.
Adding a language to an enum those read as a boolean would have
made it tokenise without dropping a stopword and without
stemming — silently, since every token still comes out.
Sourcepub fn stopwords(self) -> Option<&'static [&'static str]>
pub fn stopwords(self) -> Option<&'static [&'static str]>
This configuration’s stopword list, or None when it has none.
Sourcepub fn is_stopword(self, w: &str) -> bool
pub fn is_stopword(self, w: &str) -> bool
Is w a stopword under this configuration?