Crate wordfreq

wordfreq

This crate is a yet another Rust port of Python’s wordfreq, allowing you to look up the frequencies of words in many languages.

Note that this crate provides only the algorithms (including hardcoded standardization) and does not contain the models. Use wordfreq-model to easily load distributed models. We recommend to see the documentation for quick start.

How to create instances without wordfreq-model

If you do not desire automatic model downloads using wordfreq-model, you can create instances directly from the actual model files placed here. The model files describe words and their frequencies in the text format:

<word1> <freq1>
<word2> <freq2>
<word3> <freq3>
...

You can create instances as follows:

use approx::assert_relative_eq;
use wordfreq::WordFreq;

let word_weight_text = "las 10\nvegas 30\n";
let word_weights = wordfreq::word_weights_from_text(word_weight_text.as_bytes())?;

let wf = WordFreq::new(word_weights);
assert_relative_eq!(wf.word_frequency("las"), 0.25);
assert_relative_eq!(wf.word_frequency("vegas"), 0.75);
assert_relative_eq!(wf.word_frequency("Las"), 0.00);

Standardization

If you want to standardize words before looking up their frequencies, set up an instance of Standardizer.

use approx::assert_relative_eq;
use wordfreq::WordFreq;
use wordfreq::Standardizer;

let word_weight_text = "las 10\nvegas 30\n";
let word_weights = wordfreq::word_weights_from_text(word_weight_text.as_bytes())?;

let standardizer = Standardizer::new("en")?;
let wf = WordFreq::new(word_weights).standardizer(standardizer);
assert_relative_eq!(wf.word_frequency("Las"), 0.25); // Standardized

Precision errors

Even if the algorithms are the same, the results may differ slightly from the original implementation due to floating point precision.

Unprovided features

This crate is a straightforward port of Python’s wordfreq, although some features are not provided:

• Tokenization
• Additional functions

Re-exports

pub use preprocessers::Standardizer;

Modules

language

Language utilities.

preprocessers

Preprocessers in multiple languages.

Structs

WordFreq

Implementation of wordfreq.

Functions

word_weights_from_text

Parses pairs of a word and its weight from a text file, where each line has a word and its weight sparated by the ASCII whitespace.

Type Aliases

Float

Common type of floating numbers.