Crate english_pos_tagger 

Part-of-speech tagging and lemmatization of English text.

This crate assigns Penn-Treebank-style POS tags to tokens and adds base-form lemmas for nouns (excluding proper nouns), verbs, and adjectives. It follows a Brill-style transformation-based approach: a unigram pass proposes candidate tags, ordered context rules rewrite them from neighboring tokens, and a final pass lemmatizes and applies proper-noun and year fixups.

Every token gets a normal form, and lemmas are always produced for the parts of speech that carry them.

Example

use english_pos_tagger::Tagger;

let tagger = Tagger::new();
let tokens = tagger.tag_sentence("A bear just crossed the road.");
assert_eq!(tokens[1].value, "bear");
assert_eq!(tokens[1].pos.as_deref(), Some("NN"));
assert_eq!(tokens[1].lemma.as_deref(), Some("bear"));

Lexicon state

Tagger::update_lexicon adds or replaces word entries on that tagger instance. State stays local to the instance, not shared across taggers.

Structs

Tagger

A part-of-speech tagger with its own lexicon.

Token

A single token following the tokenizer standard.