Expand description
wordfreq
This crate is a yet another Rust port of Python’s wordfreq, allowing you to look up the frequencies of words in many languages.
Note that this crate provides only the algorithms (including hardcoded standardization) and does not contain the models. Use wordfreq-model to easily load distributed models. We recommend to see the documentation for quick start.
How to create instances without wordfreq-model
If you do not desire automatic model downloads using wordfreq-model, you can create instances directly from the actual model files placed here. The model files describe words and their frequencies in the text format:
<word1> <freq1>
<word2> <freq2>
<word3> <freq3>
...
You can create instances as follows:
use approx::assert_relative_eq;
use wordfreq::WordFreq;
let word_weight_text = "las 10\nvegas 30\n";
let word_weights = wordfreq::word_weights_from_text(word_weight_text.as_bytes())?;
let wf = WordFreq::new(word_weights);
assert_relative_eq!(wf.word_frequency("las"), 0.25);
assert_relative_eq!(wf.word_frequency("vegas"), 0.75);
assert_relative_eq!(wf.word_frequency("Las"), 0.00);
Standardization
If you want to standardize words before looking up their frequencies,
set up an instance of Standardizer
.
use approx::assert_relative_eq;
use wordfreq::WordFreq;
use wordfreq::Standardizer;
let word_weight_text = "las 10\nvegas 30\n";
let word_weights = wordfreq::word_weights_from_text(word_weight_text.as_bytes())?;
let standardizer = Standardizer::new("en")?;
let wf = WordFreq::new(word_weights).standardizer(standardizer);
assert_relative_eq!(wf.word_frequency("Las"), 0.25); // Standardized
Precision errors
Even if the algorithms are the same, the results may differ slightly from the original implementation due to floating point precision.
Unprovided features
This crate is a straightforward port of Python’s wordfreq, although some features are not provided:
Re-exports
pub use preprocessers::Standardizer;
Modules
- Language utilities.
- Preprocessers in multiple languages.
Structs
- Implementation of wordfreq.
Functions
- Parses pairs of a word and its weight from a text file, where each line has a word and its weight sparated by the ASCII whitespace.
Type Definitions
- Common type of floating numbers.