pos_tagger/lib.rs
1//! Brill-style English part-of-speech tagger.
2//!
3//! The crate has two parts. [`Lexer`] splits text into tokens with a recursive
4//! regex pipeline that keeps URLs, emails, IDs, times, and numbers whole. [`Tagger`]
5//! assigns a part-of-speech tag to each token using a built-in lexicon of about
6//! 92,000 words followed by eight contextual rewrite rules.
7//!
8//! # Example
9//!
10//! ```
11//! use pos_tagger::{Lexer, Tagger};
12//!
13//! let lexer = Lexer::new();
14//! let tagger = Tagger::new();
15//!
16//! let tokens = lexer.lex("They are wired.");
17//! let tags = tagger.tag(&tokens);
18//!
19//! assert_eq!(tags[0], ("They".to_string(), "PRP".to_string()));
20//! ```
21//!
22//! # Tag set
23//!
24//! Tags follow Penn Treebank conventions (`NN`, `NNS`, `VB`, `DT`, `JJ`, `RB`, and
25//! so on) plus a few extras the rules produce, such as `URL` and `CD`. The lexicon
26//! also contains ambiguity tags joined with `|`, but tagging only ever uses the
27//! first tag of each entry.
28//!
29//! # Accuracy
30//!
31//! Tagging is coarse and best-effort. The rules run after the lexicon lookup and
32//! several fire on word shape alone, so they can override a correct lexicon tag.
33//! Any word ending in `ly` becomes an adverb, so `Italy`, `apply`, `family`, and
34//! `supply` tag as `RB`. Nouns ending in `ing` tag as gerunds, so `string` and
35//! `king` tag as `VBG`. For high accuracy on arbitrary text, use a statistical or
36//! neural tagger instead.
37
38#![forbid(unsafe_code)]
39#![warn(missing_docs)]
40
41mod lexer;
42mod lexicon;
43mod rules;
44mod tagger;
45
46pub use lexer::Lexer;
47pub use tagger::Tagger;