Skip to main content

Crate text_stats

Crate text_stats 

Source
Expand description

§text-stats

Word and character counts for text in a lot of languages.

let counts = text_stats::count("Don't panic, it's 3.14 o'clock!");

assert_eq!(counts.words, 5);
assert_eq!(counts.characters, 31);
assert_eq!(counts.characters_no_whitespace, 27);
  • Words are the word-like segments of the ICU word segmenter: letters, numbers and CJK ideographs/kana, never whitespace, punctuation or emoji. don't and 3.14 are one word, well-known is two.
  • Characters are extended grapheme clusters (a letter with its accent, a flag or a family emoji is one character), spaces included. characters_no_whitespace leaves out the clusters that start with whitespace.
  • No language has to be given; mixed-script text works.

§Features

FeatureDefaultWhat it does
dictionaryyesSplits Chinese, Japanese, Thai, Lao, Khmer and Burmese into words with dictionaries. Without it a run of such text counts as one word.
lstmnoAdds LSTM models for Thai, Lao, Khmer and Burmese (better than dictionaries there); Chinese and Japanese keep using the dictionary.

Both add data to the binary, which is why they can be turned off for projects that only count space-delimited languages.

§Reusing the segmenters

count uses a shared Counter. Build your own with Counter::new() if you want to control when the data is loaded; a Counter is Send + Sync and cheap to call.

Counts of several texts add up (Counts implements Add and Sum). Word and character counts for text in any language.

Counting is done the way translation tools do it, but with real Unicode segmentation instead of splitting on spaces, so it also works for languages that do not use spaces:

  • words are the word-like segments of ICU4X’s word segmenter: letters, numbers and CJK ideographs/kana, but not whitespace, punctuation or emoji. don't and 3.14 are one word, well-known is two. With the dictionary feature (on by default) Chinese, Japanese, Thai, Lao, Khmer and Burmese are split into real words; without it a run of such text is not split and counts as one word.
  • characters are extended grapheme clusters, i.e. what a reader sees as one character (e plus a combining accent, or a family emoji, is one), not bytes or chars.
  • characters without whitespace are the same clusters, minus the ones that start with whitespace.

No language needs to be given: the segmenter works on mixed-script text directly.

use text_stats::count;

let counts = count("Don't panic, it's 3.14 o'clock!");
assert_eq!(counts.words, 5);
assert_eq!(counts.characters, 31);
assert_eq!(counts.characters_no_whitespace, 27);

Structs§

Counter
Counts words and characters. Building a Counter loads the segmentation data, so build one and reuse it; count does that for you with a shared instance. A Counter can be used from many threads at once.
Counts
Word and character counts of a text. Counts of several texts can be added.

Functions§

count
Counts the words and characters of text with a shared Counter.