1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
//! Brill-style English part-of-speech tagger.
//!
//! The crate has two parts. [`Lexer`] splits text into tokens with a recursive
//! regex pipeline that keeps URLs, emails, IDs, times, and numbers whole. [`Tagger`]
//! assigns a part-of-speech tag to each token using a built-in lexicon of about
//! 92,000 words followed by eight contextual rewrite rules.
//!
//! # Example
//!
//! ```
//! use pos_tagger::{Lexer, Tagger};
//!
//! let lexer = Lexer::new();
//! let tagger = Tagger::new();
//!
//! let tokens = lexer.lex("They are wired.");
//! let tags = tagger.tag(&tokens);
//!
//! assert_eq!(tags[0], ("They".to_string(), "PRP".to_string()));
//! ```
//!
//! # Tag set
//!
//! Tags follow Penn Treebank conventions (`NN`, `NNS`, `VB`, `DT`, `JJ`, `RB`, and
//! so on) plus a few extras the rules produce, such as `URL` and `CD`. The lexicon
//! also contains ambiguity tags joined with `|`, but tagging only ever uses the
//! first tag of each entry.
//!
//! # Accuracy
//!
//! Tagging is coarse and best-effort. The rules run after the lexicon lookup and
//! several fire on word shape alone, so they can override a correct lexicon tag.
//! Any word ending in `ly` becomes an adverb, so `Italy`, `apply`, `family`, and
//! `supply` tag as `RB`. Nouns ending in `ing` tag as gerunds, so `string` and
//! `king` tag as `VBG`. For high accuracy on arbitrary text, use a statistical or
//! neural tagger instead.
pub use Lexer;
pub use Tagger;