text-stats 0.1.0

Word and character counts for a lot of languages, with proper segmentation for Chinese, Japanese and Thai.
Documentation
  • Coverage
  • 88.89%
    8 out of 9 items documented1 out of 6 items with examples
  • Size
  • Source code size: 22.0 kB This is the summed size of all the files inside the crates.io package for this release.
  • Documentation size: 132.7 kB This is the summed size of all files generated by rustdoc for all configured targets
  • Ø build duration
  • this release: 7s Average build duration of successful builds.
  • all releases: 7s Average build duration of successful builds in releases after 2024-10-23.
  • Links
  • Homepage
  • savannstm/text-stats
    0 0 0
  • crates.io
  • Dependencies
  • Versions
  • Owners
  • savannstm

text-stats

Word and character counts for text in a lot of languages.

let counts = text_stats::count("Don't panic, it's 3.14 o'clock!");

assert_eq!(counts.words, 5);
assert_eq!(counts.characters, 31);
assert_eq!(counts.characters_no_whitespace, 27);
  • Words are the word-like segments of the ICU word segmenter: letters, numbers and CJK ideographs/kana, never whitespace, punctuation or emoji. don't and 3.14 are one word, well-known is two.
  • Characters are extended grapheme clusters (a letter with its accent, a flag or a family emoji is one character), spaces included. characters_no_whitespace leaves out the clusters that start with whitespace.
  • No language has to be given; mixed-script text works.

Features

Feature Default What it does
dictionary yes Splits Chinese, Japanese, Thai, Lao, Khmer and Burmese into words with dictionaries. Without it a run of such text counts as one word.
lstm no Adds LSTM models for Thai, Lao, Khmer and Burmese (better than dictionaries there); Chinese and Japanese keep using the dictionary.

Both add data to the binary, which is why they can be turned off for projects that only count space-delimited languages.

Reusing the segmenters

count uses a shared Counter. Build your own with Counter::new() if you want to control when the data is loaded; a Counter is Send + Sync and cheap to call.

Counts of several texts add up (Counts implements Add and Sum).