pos-tagger

A Brill-style English part-of-speech tagger. It has two parts: a tokenizer that splits text into words and punctuation, and a tagger that labels each token with a Penn Treebank tag.

The tagger looks each word up in a built-in lexicon of about 92,000 entries, then applies eight contextual rules that fix up nouns, numbers, URLs, adverbs, and verb forms based on word shape and neighbors.

Accuracy

Tagging is coarse and best-effort. The rules run after the lexicon lookup and several fire on word shape alone, so they can override a correct lexicon tag. Any word ending in "ly" becomes an adverb, so Italy, apply, family, and supply tag as RB. Nouns ending in "ing" tag as gerunds, so string and king tag as VBG. If you need high accuracy on arbitrary text, use a statistical or neural tagger instead.

Installation

[dependencies]
pos-tagger = "0.1"

Usage

use pos_tagger::{Lexer, Tagger};

let lexer = Lexer::new();
let tagger = Tagger::new();

let tokens = lexer.lex("I made $5.42 today.");
assert_eq!(tokens, vec!["I", "made", "$", "5.42", "today", "."]);

let tagged = tagger.tag(&tokens);
assert_eq!(tagged[3], ("5.42".to_string(), "CD".to_string()));

Lexer::lex returns tokens in order. URLs, emails, long IDs, times, and numbers stay whole. Single punctuation marks split off as their own tokens.

Tagger::tag returns one (word, tag) pair per token. The word is never changed.

Extending the lexicon

Add words the tagger does not know. Existing entries are never overwritten.

use pos_tagger::Tagger;
use std::collections::HashMap;

let mut tagger = Tagger::new();
let mut entries = HashMap::new();
entries.insert("Obama".to_string(), vec!["NNP".to_string()]);
tagger.extend_lexicon(entries);

assert_eq!(
    tagger.tag(&["Mr", "Obama"]),
    vec![
        ("Mr".to_string(), "NNP".to_string()),
        ("Obama".to_string(), "NNP".to_string()),
    ]
);

Each tagger owns its lexicon, so an extension on one instance does not affect another.

Tags

Tags follow Penn Treebank conventions: NN noun, NNS plural noun, NNP proper noun, VB verb, VBD past tense, VBN past participle, VBG gerund, DT determiner, JJ adjective, RB adverb, IN preposition, and more. The rules also produce CD for numbers and URL for dotted tokens with letters.

License

Licensed under the GNU GPL version 3 or later. The tagger builds on GPL and LGPL work by Hugo W.L. ter Doest and Percy Wegmann and embeds Eric Brill's English lexicon. See NOTICE for attribution.