Struct nlprule::tokenizer::tag::Tagger

pub struct Tagger { /* fields omitted */ }

The lexical tagger.

Implementations

impl Tagger

pub fn from_dumps<S1: AsRef<Path>, S2: AsRef<Path>, S3: AsRef<str>>(
    paths: &[S1],
    remove_paths: &[S2],
    extra_tags: &[S3],
    common_words: &HashSet<String>
) -> Result<Self>

Creates a tagger from raw files.

Arguments

• paths: Paths to files where each line contains the word, lemma and tag, respectively, separated by tabs, to be added to the tagger.
• remove_paths: Paths to files where each line contains the word, lemma and tag, respectively, separated by tabs, to be removed from the tagger if present in the files from paths.

pub fn id_tag<'a>(&self, tag: &'a str) -> PosId<'a>

pub fn id_word<'t>(&'t self, text: Cow<'t, str>) -> WordId<'t>

pub fn get_tags(
    &self,
    word: &str,
    add_lower: bool,
    use_compound_split_heuristic: bool
) -> Vec<WordData<'_>>

Get the tags and lemmas (as WordData) for the given word.

Arguments

• word: The word to lookup data for.
• add_lower: Whether to add data for the lowercase variant of the word.
• use_compound_split_heuristic: Whether to use a heuristic to split compound words. If true, will attempt to find tags for words which are longer than some cutoff and unknown by looking up tags for substrings from left to right until tags are found or a minimum length reached.

pub fn get_group_members(&self, lemma: &str) -> Vec<&str>

Get the words with the same lemma as the given lemma.

Trait Implementations

impl Clone for Tagger

fn clone(&self) -> Tagger

Returns a copy of the value.

pub fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Default for Tagger

fn default() -> Tagger

Returns the "default value" for a type.

impl<'de> Deserialize<'de> for Tagger

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
    __D: Deserializer<'de>, 

Deserialize this value from the given Serde deserializer.

impl Serialize for Tagger

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error> where
    __S: Serializer, 

Serialize this value into the given Serde serializer.