use-token

Composable tokenization primitives for RustUse.

use-token keeps tokenization explicit and small. It handles whitespace splitting, conservative word tokenization, lightweight sentence boundaries, and character spans without claiming to be a full NLP parser.

Included primitives

• tokenize_whitespace
• tokenize_words
• tokenize_sentences
• tokenize_chars
• token_count

Example

use use_token::{token_count, tokenize_sentences, tokenize_words};

assert_eq!(token_count("Hello, world!"), 2);
assert_eq!(tokenize_words("don't stop").len(), 2);
assert_eq!(tokenize_sentences("One. Two!").len(), 2);