Struct nlprule::tokenizer::tag::Tagger

pub struct Tagger { /* fields omitted */ }

The lexical tagger. Created from a dictionary that looks like this:

actualize   actualize   VB
actualize   actualize   VBP
actualized  actualize   VBD
actualized  actualize   VBN
actualizes  actualize   VBZ
actualizing actualize   VBG
actually    actually    RB

i.e. one word (left) associated with one or more pairs of lemma (middle) and POS (part-of-speech) tag (right). From this structure, the tagger must be able to look up:

1. lemma and pos by word: all lemmas and POS tags associated with a given word.

(1) is called extensively (at least once for every word) so it has to be as fast as possible.

Implementation

The tagger stores two bidirectional maps:

• A POS bimap: A bimap assigning each POS tag a 16-bit ID. POS tags are a closed set, so there is an entry for every tag in the bimap. This allows e.g. storing a set of IDs of matching tags for a regex instead of actually evaluating it in the matcher logic.
• A word bimap: A bimap assigning each known word a 32-bit ID. Words are not a closed set, so if an entry in this map does not exist it only means that the word is not known to nlprule. Still, the map can be used for optimizations similar to the POS bimap. The word bimap also stores lemmas since there is often a large overlap between known words and known lemmas.

These two maps can be used to relatively cheaply in terms of memory allow (1) while retaining fast lookup.

There is a tags map which associates a Word ID with multiple pairs of (lemma_id, pos_id) where the ID for the lemma is a regular 32-bit Word ID.

Implementations

impl Tagger

pub fn id_tag<'a>(&self, tag: &'a str) -> PosId<'a>

Tags the given string representation of a part-of-speech tag. Part-of-speech tags are treated as a closed set so each valid part-of-speech tag will get a numerical id.

pub fn id_word<'a>(&self, text: Cow<'a, str>) -> WordId<'a>

Tags the given text. Unknown words will not get a numerical id.

pub fn get_tags_with_options<'a>(
    &'a self,
    word: &'a str,
    add_lower: Option<bool>,
    use_compound_split_heuristic: Option<bool>
) -> TagIter<'a>

Get the tags and lemmas (as WordData) for the given word.

Arguments

• word: The word to lookup data for.
• add_lower: Whether to add data for the lowercase variant of the word. If None, will be set according to the language options.
• use_compound_split_heuristic: Whether to use a heuristic to split compound words. If None, will be set according to the language options. If true, will attempt to find tags for words which are longer than some cutoff and unknown by looking up tags for substrings from left to right until tags are found or a minimum length reached.

pub fn get_tags<'a>(&'a self, word: &'a str) -> TagIter<'a>

Get the tags and lemmas (as WordData) for the given word using the default options of the tagger.

Arguments

• word: The word to lookup data for.

Trait Implementations

impl Clone for Tagger

fn clone(&self) -> Tagger

Returns a copy of the value.

pub fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Default for Tagger

fn default() -> Tagger

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for Tagger

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
    __D: Deserializer<'de>, 

Deserialize this value from the given Serde deserializer.

impl Serialize for Tagger

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error> where
    __S: Serializer, 

Serialize this value into the given Serde serializer.