Struct Tokenizer

Source

pub struct Tokenizer { /* private fields */ }

Expand description

A regex tokenizer instance.

Each instance holds its own rule list, fingerprint codes, and the result of the last tokenize call. Fingerprint codes are per instance. A define_config call resets them. Create an instance with Tokenizer::new.

Implementations§

Source §

impl Tokenizer

Source

pub fn new() -> Self

Creates a tokenizer with every token type enabled except quoted_phrase.

§Example

use english_tokenizer::Tokenizer;
let mut tk = Tokenizer::new();
assert!(tk.tokenize("").is_empty());

Source

pub fn tokenize(&mut self, sentence: &str) -> Vec<Token>

Tokenizes sentence and returns the tokens in left to right order.

The result is also kept for the next get_tokens_fp call.

§Example

use english_tokenizer::Tokenizer;
let mut tk = Tokenizer::new();
let tokens = tk.tokenize("4pm");
assert_eq!(tokens[0].tag, "time");

Source

pub fn define_config(&mut self, config: &[(&str, bool)]) -> usize

Sets which token types are active and returns the count of unique active categories.

Each pair is a category name and a flag. A true flag enables the category. A false flag disables it. Categories not listed stay enabled. An empty list disables everything, so tokenize splits on spaces and tags every piece alien.

This also resets any rules added with add_regex and resets the fingerprint codes.

Categories that use more than one rule, like hashtag and number, count once. The full default returns 13 because quoted_phrase is off. Enabling quoted_phrase returns 14.

§Example

use english_tokenizer::Tokenizer;
let mut tk = Tokenizer::new();
assert_eq!(tk.define_config(&[("hashtag", false)]), 13);
assert_eq!(tk.define_config(&[]), 0);
assert_eq!(tk.define_config(&[("quoted_phrase", true)]), 14);

Source

pub fn get_tokens_fp(&self) -> String

Returns the fingerprint of the tokens from the last tokenize call.

For each token, if its tag has a fingerprint code, that single character is appended. Otherwise the token’s literal value is appended. Tokens tagged punctuation or symbol have no code, so their value passes through. Before any tokenize call this returns an empty string.

§Example

use english_tokenizer::Tokenizer;
let mut tk = Tokenizer::new();
tk.tokenize("feeling good #FunTime");
assert_eq!(tk.get_tokens_fp(), "wwh");

Source

pub fn add_regex( &mut self, regex: Regex, tag: &str, fingerprint_code: Option<&str>, ) -> Result<(), AddError>

Adds a regex rule that wins over the built-ins.

The rule is tried before every built-in rule. If tag is already known, fingerprint_code is ignored. If tag is new, fingerprint_code must be a non-empty string so the tag has a fingerprint. A later define_config call removes added rules.

Write the pattern with non-capturing groups (?:...). Matching takes the whole match and the text between matches. A capturing group does not inject its capture as an extra token.

§Errors

Returns AddError::MissingFingerprint when tag is new and the fingerprint code is missing or empty.

§Example

use english_tokenizer::Tokenizer;
use regex::Regex;
let mut tk = Tokenizer::new();
tk.add_regex(Regex::new(r"(?i)\(oo\)").unwrap(), "emoticon", None).unwrap();
let tokens = tk.tokenize("(oo) Hi!");
assert_eq!(tokens[0].tag, "emoticon");