Expand description
VADER sentiment scoring with exact output parity against Python vaderSentiment 3.3.2.
VADER (Valence Aware Dictionary and sEntiment Reasoner) is the rule-based
sentiment model of Hutto & Gilbert (ICWSM-14). This crate reimplements
that package’s whole rule set over that package’s own data files,
vendored byte-identical from the 3.3.2 sdist: lexicon scoring, booster
and dampener modifiers with the casing increment, negation with the
“least” special case, contrastive “but” reweighting, exclamation and
question mark emphasis, the special-case idioms, emoji translation, and
the alpha-15 normalization to compound. For every input, all four scores
equal what the Python package returns at its published precision: three
decimals for the proportions, four for the compound. The frozen oracle
vectors in tests/vectors/frozen.tsv replay on every cargo test, and a
release-time differential corpus against the pinned package holds that
parity with zero tolerated divergence. Casing is pinned with the rest:
str.isupper() and str.lower() answer from tables vendored out of
CPython 3.14.6 with Unicode data 16.0.0, checked across every codepoint,
so the scores hold still when the Rust standard library adopts a newer
Unicode version than the reference interpreter’s.
use vader_parity::SentimentIntensityAnalyzer;
let analyzer = SentimentIntensityAnalyzer::new();
let scores = analyzer.polarity_scores("VADER is smart, handsome, and funny.");
assert_eq!(scores.neg, 0.0);
assert_eq!(scores.neu, 0.254);
assert_eq!(scores.pos, 0.746);
assert_eq!(scores.compound, 0.8316);§Scoring many texts
SentimentIntensityAnalyzer::new parses the vendored lexicons, about
11,000 data lines, once. Build one analyzer and score many texts. Each
call is a bounded linear pass, and the example below is the crate’s own
regression tripwire for that claim.
use vader_parity::SentimentIntensityAnalyzer;
let analyzer = SentimentIntensityAnalyzer::new();
let started = std::time::Instant::now();
for _ in 0..10_000 {
analyzer.polarity_scores("Today only kinda sux! But I'll get by, lol");
}
assert!(started.elapsed() < std::time::Duration::from_secs(30));§Scope
The reference is English-only and so is its lexicon. This crate is the
scoring engine only, pinned to reference version 3.3.2, std-only, with
zero runtime dependencies. Sentence splitting, custom lexicons, and other
reference versions are out of scope.
§Attribution
The VADER algorithm design, the sentiment lexicon, and the emoji lexicon
are the MIT-licensed work of C.J. Hutto and E.E. Gilbert. The vendored
files in data/ ship with the authors’ license text and their recorded
sha256 digests, and the Rust implementation and its tests are this
crate’s. If you use VADER, cite: Hutto, C.J. & Gilbert, E.E. (2014).
VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social
Media Text. Eighth International Conference on Weblogs and Social Media
(ICWSM-14). Ann Arbor, MI.
Structs§
- Sentiment
Intensity Analyzer - The analyzer, holding the loaded lexicons.
- Sentiment
Scores - The four scores for one text. Field values match Python vaderSentiment 3.3.2, at that package’s published precision.