ReadSight (Rust) — Multilingual Readability Engine

readsight measures text readability across 86 languages. It implements 17 readability formulas with language-specific coefficients and uses the Frank M. Liang (TeX) hyphenation algorithm for syllable counting. All language data and hyphenation patterns are embedded in the crate — no filesystem or network access is required at runtime.

This is a byte-accurate Rust port of the canonical PHP library and its Python port:

• PHP (canonical): https://github.com/MADEVAL/ReadSight
• Python port: https://github.com/MADEVAL/ReadSightPy

Output parity with the reference implementation is verified with golden vectors generated from the PHP library (see tests/golden).

See It in Action

Two texts of almost equal length — a plain sentence and a chunk of legal boilerplate:

let plain = "We made an app that reads your text. It tells you how easy it is to read. You get a score in one second.";
let legal = "The parties acknowledge that any unauthorized disclosure of confidential information may cause irreparable harm. In such an event, the affected party shall be entitled to seek injunctive relief.";

There is no "score everything" call — you loop over the formulas the language supports and call score() for each:

use readsight::ReadSight;
# let legal = "The parties acknowledge that any unauthorized disclosure of confidential information may cause irreparable harm. In such an event, the affected party shall be entitled to seek injunctive relief.";

let rs = ReadSight::new("en-us")?;

for formula in rs.supported_formulas() {
    let result = rs.score(&formula, legal)?;
    // result.score, result.grade_level, result.interpretation
    // ...
}
# Ok::<(), readsight::Error>(())

For both texts that produces (verbatim output of cargo run --example demo):

READABILITY FORMULA          | Plain text               | Legalese
------------------------------------------------------------------------------------
ari                          | -2.1  g0.0 Kindergarten  | 13.2  g13.2 College
coleman_liau                 | -0.4  g0.0 Kindergarten  | 16.5  g16.5 Graduate
dale_chall                   | 5.3  5th-6th grade       | 12.2  Graduate
flesch_kincaid_grade_level   | 0.3  g0.3 1st Grade      | 13.5  g13.5 College
flesch_reading_ease          | 107.1  Very Easy         | 23.4  Very Hard
gunning_fog                  | 3.2  g3.2 Very Easy      | 18.5  g18.5 Extremely Hard
lix                          | 8  Children's Books      | 49.71  Factual Information
smog                         | 3.1  g3.1 3rd Grade      | 15.2  g15.2 College
spache                       | 2.3  g2.3 2nd Grade      | 6.5  g5.0 Above 4th Grade

All 9 formulas for en-us agree the second text is far harder. The bundled example prints this grid plus syllable breakdowns and text statistics:

cargo run --example demo
cargo run --example multilingual

17 formulas, 86 languages, one consistent API. Five of the formulas are truly universal — Gunning Fog, SMOG, Coleman-Liau, ARI and LIX score text in every one of the 86 languages. The remaining 12 are language-aware, each carrying its own coefficients: Flesch Reading Ease and Flesch-Kincaid span 12 languages, the Wiener Sachtextformel speaks German, Gulpease speaks Italian, OSMAN speaks Arabic, and the Fernández-Huerta · Szigriszt-Pazos · Gutiérrez-Polini · Crawford family handles Spanish. supported_formulas() then hands each language exactly the slice that fits it — 9 formulas for en-us, 11 for es, 8 for de-1996, down to the 5 universal ones for a language like th — so an English-only metric never lands on a Thai sentence by mistake.

Table of Contents

• Installation
• See It in Action
• Quick Start
• Syllable Counting Modes
• Examples
• Supported Languages
• Readability Formulas
• FormulaResult
• API Reference
• Data Source
• Architecture
• Development
• License

Installation

[dependencies]
readsight = "1.0"

Requirements:

• Rust >= 1.74 (edition 2021)

The crate depends on regex, fancy-regex, serde, serde_json, include_dir, and indexmap. All language and hyphenation data is bundled in the crate, so there is no runtime filesystem or network access.

Quick Start

use readsight::ReadSight;

let engine = ReadSight::new("en-us")?;

// Syllable counting
assert_eq!(engine.syllable_count("banana"), 3);          // 3
assert_eq!(engine.split_syllables("hyphenation"),        // ["hyp", "hen", "ati", "on"] (heuristic split)
           vec!["hyp", "hen", "ati", "on"]);
assert_eq!(engine.split_word("hyphenation"),             // ["hy", "phen", "ation"] (TeX hyphenation points)
           vec!["hy", "phen", "ation"]);

// Text analysis
let stats = engine.analyze("The quick brown fox jumps over the lazy dog.")?;
println!("Words: {}, Syllables: {}", stats.word_count, stats.syllable_count);

// Readability formulas
let fre = engine.flesch_reading_ease("The quick brown fox jumps over the lazy dog.")?;
println!("Flesch Reading Ease: {} - {}", fre.score, fre.interpretation);

let fog = engine.gunning_fog("The quick brown fox jumps over the lazy dog.")?;
println!("Gunning Fog: {} (grade {:?})", fog.score, fog.grade_level);
# Ok::<(), readsight::Error>(())

All formula methods return Result<FormulaResult, Error>. analyze (and hence every formula) returns [Error::EmptyText] for empty input.

Syllable Counting Modes

ReadSight has three syllable counting modes, configured per language via syllableMode in data/languages/*.json:

80 languages use tex, 2 use composite (en-us, en-gb), 4 use heuristic (ru, uk, be, bg). The default mode is tex.

Why tex count is approximate: TeX hyphenation patterns are optimised for line-breaking, not phonetic syllabification. They respect hyphenMins and avoid awkward break points, so the number of pieces can differ from the true syllable count. For scripts where one syllable = one vowel (e.g. Cyrillic Slavic languages), TeX under- or over-counts. Those languages use heuristic mode with a per-language vowel pattern and "vowelMode": "individual" so each vowel is counted as a syllable. split_word() keeps using the exact TeX hyphenator regardless of mode.

Vowel counting: vowelMode

The syllableHeuristics block accepts a vowelMode field:

use readsight::ReadSight;

// Russian uses heuristic + vowelMode "individual"
let ru = ReadSight::new("ru")?;
assert_eq!(ru.syllable_count("дыхание"), 4);
# Ok::<(), readsight::Error>(())

Example: "hyphenation" in each mode

use readsight::ReadSight;

let en = ReadSight::new("en-us")?;           // composite mode - heuristic wins
assert_eq!(en.syllable_count("hyphenation"), 4);
assert_eq!(en.split_syllables("hyphenation"), vec!["hyp", "hen", "ati", "on"]); // heuristic
assert_eq!(en.split_word("hyphenation"), vec!["hy", "phen", "ation"]);          // TeX

let de = ReadSight::new("de-1996")?;          // tex mode
assert_eq!(de.syllable_count("hyphenation"), 4);
assert_eq!(de.split_syllables("hyphenation"), vec!["hy", "phena", "ti", "on"]); // TeX
assert_eq!(de.split_word("hyphenation"), vec!["hy", "phena", "ti", "on"]);      // same
# Ok::<(), readsight::Error>(())

Tip: split_word() always uses the TeX hyphenator (exact). split_syllables() may use the heuristic split (approximate) in composite/heuristic modes. Syllable counts are exact in heuristic/composite mode; in tex mode they follow TeX break points and are approximate (see note above).

Note: add_hyphenations() adds overrides to the TeX hyphenator. These affect split_word() but NOT split_syllables() in composite/heuristic modes (the heuristic counter doesn't see them).

Examples

Run the bundled demo to see ReadSight in action:

cargo run --example demo

This scores the plain/legal texts side by side and outputs:

• Readability grid with every applicable formula, its score, grade level and interpretation
• Syllable breakdown with heuristic split and TeX hyphenation points
• Text statistics — letters, words, sentences, syllables, and a syllable histogram

Compare a short sample across several languages:

cargo run --example multilingual

Supported Languages

86 languages across 19 writing systems: Latin, Cyrillic, Arabic, Hebrew, Devanagari, Bengali, Tamil, Thai, Greek, Armenian, Georgian, Gujarati, Gurmukhi, Kannada, Malayalam, Odia, Telugu, Ethiopic, Coptic.

use readsight::ReadSight;

let _ru = ReadSight::new("ru")?;       // Russian
let _de = ReadSight::new("de-1996")?;  // German (1996 reform)
let _es = ReadSight::new("es")?;       // Spanish
let _th = ReadSight::new("th")?;       // Thai

// List all supported languages (sorted)
let langs = ReadSight::supported_languages(None);
assert_eq!(langs.len(), 86);
# Ok::<(), readsight::Error>(())

ReadSight::supported_languages(None) returns all 86 codes, sorted, mirroring the JSON files under data/languages exactly:

af, ar, as, be, bg, bn, ca, cop, cs, cu, cy, da, de-1901, de-1996, de-ch-1901,
el-monoton, el-polyton, en-gb, en-us, eo, es, et, eu, fa, fi, fi-x-school, fr,
fur, ga, gl, grc, gu, he, hi, hr, hsb, hu, hy, ia, id, is, it, ka, kk, kmr, kn,
la, la-x-classic, la-x-liturgic, lt, lv, mk, ml, mn-cyrl, mn-cyrl-x-lmc, mr,
mul-ethi, nb, nl, nn, oc, or, pa, pi, pl, pms, pt, rm, ro, ru, sa, sh-cyrl,
sh-latn, sk, sl, sq, sr-cyrl, sv, ta, te, th, tk, tr, uk, vi, zh-latn-pinyin

Readability Formulas

Universal (all 86 languages)

Language-Specific

* Note: Dale-Chall and Spache use a syllable-based heuristic to estimate difficult words (1-syllable ≈ easy). This is a simplified estimation, not the original Dale/Spache word lists.

Generic dispatching by name:

use readsight::ReadSight;

let rs = ReadSight::new("de-1996")?;
let r = rs.score("gunning_fog", "Ein einfacher deutscher Satz. Und noch einer.")?;
assert_eq!(r.formula_name, "gunning_fog");

// Wiener Sachtextformel supports variants 1..=4
let w = rs.wiener_sachtextformel("Ein einfacher deutscher Satz. Und noch einer.", 1)?;
assert_eq!(w.formula_name, "wiener_sachtextformel_1");
# Ok::<(), readsight::Error>(())

FormulaResult

# use readsight::FormulaResult;
# fn _doc(result: FormulaResult) {
result.score;           // f64 - raw (rounded) formula score
result.grade_level;     // Option<f64> - normalized grade level (FKGL, GF, SMOG, CL, ARI, Spache)
result.interpretation;  // String - qualitative interpretation ("Easy", "Hard", ...)
result.formula_name;    // String - formula key
result.language_code;   // String - language code used
result.inputs;          // BTreeMap<String, f64> - intermediate values for debugging
# }

API Reference

ReadSight (aliased as Engine) is the entry point.

Text / syllable methods

engine.syllable_count(word: &str) -> i64
engine.split_word(word: &str) -> Vec<String>
engine.split_syllables(word: &str) -> Vec<String>
engine.word_count(text: &str) -> i64
engine.sentence_count(text: &str) -> i64
engine.letter_count(text: &str) -> i64
engine.total_syllables(text: &str) -> i64
engine.average_syllables_per_word(text: &str) -> f64
engine.average_words_per_sentence(text: &str) -> f64
engine.polysyllable_count(text: &str, count_proper_nouns: bool) -> i64
engine.words_with_more_than_n_syllables(text: &str, n: i64, count_proper_nouns: bool) -> i64
engine.histogram_syllables(text: &str) -> BTreeMap<i64, i64>
engine.analyze(text: &str) -> Result<TextStatistics>
engine.add_hyphenations(iter)   // (word, "hy-phen-a-ted") overrides

Formula methods

engine.score(name: &str, text: &str) -> Result<FormulaResult>
engine.flesch_reading_ease(text) / flesch_kincaid_grade_level(text)
engine.gunning_fog(text) / smog_index(text) / coleman_liau(text)
engine.automated_readability_index(text) / lix(text)
engine.gulpease(text) / fernandez_huerta(text) / szigriszt_pazos(text)
engine.gutierrez_polini(text) / crawford(text) / fog_pl(text)
engine.dale_chall(text) / spache(text) / osman(text)
engine.wiener_sachtextformel(text, variant: i32)   // variant 1..=4

Static

ReadSight::supported_languages(config: Option<&Config>) -> Vec<String>

Data Source

By default all data is embedded at compile time (via include_dir!): 86 language JSON files and 86 hyph-*.tex pattern files under data/. To load from the filesystem instead:

use readsight::{Config, ReadSight};

let config = Config::from_dirs("data/patterns", "data/languages");
let rs = ReadSight::with_config("en-us", config)?;
# Ok::<(), readsight::Error>(())

Custom hyphenation overrides affect split_word (not split_syllables in composite/heuristic modes):

use readsight::ReadSight;

let engine = ReadSight::new("en-us")?;
engine.add_hyphenations([("customword", "cus-tom-word")]);
assert_eq!(engine.split_word("customword"), vec!["cus", "tom", "word"]);
# Ok::<(), readsight::Error>(())

Architecture

ReadSight (facade, aliased as Engine)
  ├── TextAnalyzer (syllable counting, text metrics)
  │   ├── SyllableCounterKind (tex | heuristic | composite)
  │   │   ├── CompositeSyllableCounter (heuristic problem words → heuristic, rest → TeX)
  │   │   ├── HeuristicSyllableCounter (vowel patterns + word list, vowelMode)
  │   │   └── TexSyllableCounter → LiangHyphenator (TeX hyphenation)
  │   ├── LiangHyphenator
  │   │   ├── parse_tex / TexSource (parses .tex from hyph-utf8)
  │   │   ├── PatternsCollection (pattern data)
  │   │   └── HyphenationExceptionsCollection (word overrides)
  │   └── TextSplitter (word/sentence/letter counting)
  ├── Language (JSON config per language, syllableMode + formula configs)
  └── FormulaRegistry (17 formulas)
      ├── FleschReadingEase / FleschKincaidGradeLevel (lang-specific coefficients)
      ├── GunningFog, SmogIndex, ColemanLiau, ARI, LIX (universal)
      └── WienerSachtextformel, Gulpease, FernandezHuerta, ... (lang-specific)

Development

cargo test                                  # unit + golden parity + smoke tests
cargo clippy --all-targets -- -D warnings   # lints (clean)
cargo fmt --check                           # formatting (clean)

The suite currently runs 90 integration tests, plus the crate's doc tests (including the runnable Rust examples in this README, which are compiled and executed by cargo test):

• Golden parity against the PHP reference: supported_formulas, analyze, and the applicable formulas over all 86 languages, plus per-word syllable vectors for a 20-language subset (tests/golden).
• Ported unit tests from the PHP/Python suites (hyphenation, syllables, text splitting, formulas, grade-level interpretation).
• Full-language smoke test that builds every language and runs every supported formula.

The library is #![forbid(unsafe_code)].

License

MIT — see LICENSE. Author of the original library: Yevhen Leonidov. Readability data and hyphenation patterns originate from the canonical PHP project and the hyph-utf8 package; the TeX pattern files are packaged under their original licenses (see individual file headers).