pos-tagger 0.2.0

Brill-style English part-of-speech tagger (lexicon lookup + 8 contextual rewrite rules)
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82

# pos-tagger

A Brill-style English part-of-speech tagger. It has two parts: a tokenizer that
splits text into words and punctuation, and a tagger that labels each token with
a Penn Treebank tag.

The tagger looks each word up in a built-in lexicon of about 92,000 entries, then
applies eight contextual rules that fix up nouns, numbers, URLs, adverbs, and
verb forms based on word shape and neighbors.

## Accuracy

Tagging is coarse and best-effort. The rules run after the lexicon lookup and
several fire on word shape alone, so they can override a correct lexicon tag. Any
word ending in "ly" becomes an adverb, so Italy, apply, family, and supply tag as
RB. Nouns ending in "ing" tag as gerunds, so string and king tag as VBG. If you
need high accuracy on arbitrary text, use a statistical or neural tagger instead.

## Installation

```toml
[dependencies]
pos-tagger = "0.1"
```

## Usage

```rust
use pos_tagger::{Lexer, Tagger};

let lexer = Lexer::new();
let tagger = Tagger::new();

let tokens = lexer.lex("I made $5.42 today.");
assert_eq!(tokens, vec!["I", "made", "$", "5.42", "today", "."]);

let tagged = tagger.tag(&tokens);
assert_eq!(tagged[3], ("5.42".to_string(), "CD".to_string()));
```

`Lexer::lex` returns tokens in order. URLs, emails, long IDs, times, and numbers
stay whole. Single punctuation marks split off as their own tokens.

`Tagger::tag` returns one `(word, tag)` pair per token. The word is never changed.

## Extending the lexicon

Add words the tagger does not know. Existing entries are never overwritten.

```rust
use pos_tagger::Tagger;
use std::collections::HashMap;

let mut tagger = Tagger::new();
let mut entries = HashMap::new();
entries.insert("Obama".to_string(), vec!["NNP".to_string()]);
tagger.extend_lexicon(entries);

assert_eq!(
    tagger.tag(&["Mr", "Obama"]),
    vec![
        ("Mr".to_string(), "NNP".to_string()),
        ("Obama".to_string(), "NNP".to_string()),
    ]
);
```

Each tagger owns its lexicon, so an extension on one instance does not affect
another.

## Tags

Tags follow Penn Treebank conventions: `NN` noun, `NNS` plural noun, `NNP` proper
noun, `VB` verb, `VBD` past tense, `VBN` past participle, `VBG` gerund, `DT`
determiner, `JJ` adjective, `RB` adverb, `IN` preposition, and more. The rules
also produce `CD` for numbers and `URL` for dotted tokens with letters.

## License

Licensed under the [GNU GPL version 3 or later](LICENSE). The tagger builds on
GPL and LGPL work by Hugo W.L. ter Doest and Percy Wegmann and embeds Eric
Brill's English lexicon. See [NOTICE](NOTICE) for attribution.