Expand description
Brill-style English part-of-speech tagger.
The crate has two parts. Lexer splits text into tokens with a recursive
regex pipeline that keeps URLs, emails, IDs, times, and numbers whole. Tagger
assigns a part-of-speech tag to each token using a built-in lexicon of about
92,000 words followed by eight contextual rewrite rules.
§Example
use pos_tagger::{Lexer, Tagger};
let lexer = Lexer::new();
let tagger = Tagger::new();
let tokens = lexer.lex("They are wired.");
let tags = tagger.tag(&tokens);
assert_eq!(tags[0], ("They".to_string(), "PRP".to_string()));§Tag set
Tags follow Penn Treebank conventions (NN, NNS, VB, DT, JJ, RB, and
so on) plus a few extras the rules produce, such as URL and CD. The lexicon
also contains ambiguity tags joined with |, but tagging only ever uses the
first tag of each entry.
§Accuracy
Tagging is coarse and best-effort. The rules run after the lexicon lookup and
several fire on word shape alone, so they can override a correct lexicon tag.
Any word ending in ly becomes an adverb, so Italy, apply, family, and
supply tag as RB. Nouns ending in ing tag as gerunds, so string and
king tag as VBG. For high accuracy on arbitrary text, use a statistical or
neural tagger instead.