Crate english_tokenizer 

Regex-driven tokenizer that tags each token with its type.

Give it a sentence and it returns an ordered list of tokens. Each token has a value and a tag. Tags name the token type: word, number, ordinal, email, url, mention, hashtag, emoji, emoticon, time, currency, quoted_phrase, punctuation, symbol, plus alien for text that no active rule matched.

The tokenizer covers Latin-1 and Devanagari scripts. It splits common English contractions and possessives into separate word tokens. For example I'll becomes I and 'll, and dog's becomes dog and 's.

Example

use english_tokenizer::Tokenizer;

let mut tk = Tokenizer::new();
let tokens = tk.tokenize("feeling good #FunTime");
assert_eq!(tokens.len(), 3);
assert_eq!(tokens[2].value, "#FunTime");
assert_eq!(tokens[2].tag, "hashtag");

Configuration

Tokenizer::new enables every token type except quoted_phrase. Use Tokenizer::define_config to turn types on or off. An empty config splits on spaces and tags everything alien.

Tokenizer::add_regex injects a custom rule that wins over the built-ins. Tokenizer::add_tag registers a new tag with a fingerprint code. Tokenizer::get_tokens_fp returns a one character per token fingerprint of the last tokenize call.

Modules

tags

Built-in token tag names.

Structs

Token

A single token: its text and its type tag.

Tokenizer

A regex tokenizer instance.

Enums

AddError

Error returned when a tag or regex cannot be added.

Constants

CONTRACTION_COUNT

Number of contraction keys in the built-in table.