# ReadSight (Rust) — Multilingual Readability Engine
[](LICENSE)
[](#supported-languages)
[](#readability-formulas)
`readsight` measures text readability across **86 languages**. It implements
**17 readability formulas** with language-specific coefficients and uses the
Frank M. Liang (TeX) hyphenation algorithm for syllable counting. All language
data and hyphenation patterns are **embedded in the crate** — no filesystem or
network access is required at runtime.
This is a **byte-accurate Rust port** of the canonical PHP library and its Python port:
- **PHP (canonical):** <https://github.com/MADEVAL/ReadSight>
- **Python port:** <https://github.com/MADEVAL/ReadSightPy>
Output parity with the reference implementation is verified with golden vectors
generated from the PHP library (see [`tests/golden`](tests/golden)).
## See It in Action
Two texts of almost equal length — a plain sentence and a chunk of legal boilerplate:
```rust
let plain = "We made an app that reads your text. It tells you how easy it is to read. You get a score in one second.";
let legal = "The parties acknowledge that any unauthorized disclosure of confidential information may cause irreparable harm. In such an event, the affected party shall be entitled to seek injunctive relief.";
```
There is no "score everything" call — you loop over the formulas the language
supports and call `score()` for each:
```rust
use readsight::ReadSight;
# let legal = "The parties acknowledge that any unauthorized disclosure of confidential information may cause irreparable harm. In such an event, the affected party shall be entitled to seek injunctive relief.";
let rs = ReadSight::new("en-us")?;
for formula in rs.supported_formulas() {
let result = rs.score(&formula, legal)?;
// result.score, result.grade_level, result.interpretation
// ...
}
# Ok::<(), readsight::Error>(())
```
For both texts that produces (verbatim output of `cargo run --example demo`):
```text
ari | -2.1 g0.0 Kindergarten | 13.2 g13.2 College
coleman_liau | -0.4 g0.0 Kindergarten | 16.5 g16.5 Graduate
dale_chall | 5.3 5th-6th grade | 12.2 Graduate
flesch_kincaid_grade_level | 0.3 g0.3 1st Grade | 13.5 g13.5 College
flesch_reading_ease | 107.1 Very Easy | 23.4 Very Hard
gunning_fog | 3.2 g3.2 Very Easy | 18.5 g18.5 Extremely Hard
lix | 8 Children's Books | 49.71 Factual Information
smog | 3.1 g3.1 3rd Grade | 15.2 g15.2 College
spache | 2.3 g2.3 2nd Grade | 6.5 g5.0 Above 4th Grade
```
All 9 formulas for `en-us` agree the second text is far harder. The bundled
example prints this grid plus syllable breakdowns and text statistics:
```sh
cargo run --example demo
cargo run --example multilingual
```
**17 formulas, 86 languages, one consistent API.** Five of the formulas are
truly universal — **Gunning Fog, SMOG, Coleman-Liau, ARI and LIX** score text in
*every* one of the 86 languages. The remaining **12 are language-aware**, each
carrying its own coefficients: Flesch Reading Ease and Flesch-Kincaid span 12
languages, the Wiener Sachtextformel speaks German, Gulpease speaks Italian,
OSMAN speaks Arabic, and the Fernández-Huerta · Szigriszt-Pazos ·
Gutiérrez-Polini · Crawford family handles Spanish. `supported_formulas()` then
hands each language exactly the slice that fits it — **9** formulas for `en-us`,
**11** for `es`, **8** for `de-1996`, down to the **5** universal ones for a
language like `th` — so an English-only metric never lands on a Thai sentence by
mistake.
## Table of Contents
- [Installation](#installation)
- [See It in Action](#see-it-in-action)
- [Quick Start](#quick-start)
- [Syllable Counting Modes](#syllable-counting-modes)
- [Examples](#examples)
- [Supported Languages](#supported-languages)
- [Readability Formulas](#readability-formulas)
- [FormulaResult](#formularesult)
- [API Reference](#api-reference)
- [Data Source](#data-source)
- [Architecture](#architecture)
- [Development](#development)
- [License](#license)
## Installation
```toml
[dependencies]
readsight = "1.0"
```
**Requirements:**
- Rust >= 1.74 (edition 2021)
The crate depends on `regex`, `fancy-regex`, `serde`, `serde_json`,
`include_dir`, and `indexmap`. All language and hyphenation data is bundled in
the crate, so there is no runtime filesystem or network access.
## Quick Start
```rust
use readsight::ReadSight;
let engine = ReadSight::new("en-us")?;
// Syllable counting
assert_eq!(engine.syllable_count("banana"), 3); // 3
assert_eq!(engine.split_syllables("hyphenation"), // ["hyp", "hen", "ati", "on"] (heuristic split)
vec!["hyp", "hen", "ati", "on"]);
assert_eq!(engine.split_word("hyphenation"), // ["hy", "phen", "ation"] (TeX hyphenation points)
vec!["hy", "phen", "ation"]);
// Text analysis
let stats = engine.analyze("The quick brown fox jumps over the lazy dog.")?;
println!("Words: {}, Syllables: {}", stats.word_count, stats.syllable_count);
// Readability formulas
let fre = engine.flesch_reading_ease("The quick brown fox jumps over the lazy dog.")?;
println!("Flesch Reading Ease: {} - {}", fre.score, fre.interpretation);
let fog = engine.gunning_fog("The quick brown fox jumps over the lazy dog.")?;
println!("Gunning Fog: {} (grade {:?})", fog.score, fog.grade_level);
# Ok::<(), readsight::Error>(())
```
All formula methods return `Result<FormulaResult, Error>`. `analyze` (and hence
every formula) returns [`Error::EmptyText`] for empty input.
## Syllable Counting Modes
ReadSight has three syllable counting modes, configured per language via
`syllableMode` in `data/languages/*.json`:
| **`heuristic`** | Vowel patterns + word list + prefix/suffix rules | exact | ≈ approximate |
| **`tex`** | Frank M. Liang hyphenation algorithm (TeX `.tex` patterns) | ≈ approximate | exact |
| **`composite`** | Heuristic first, TeX as fallback | exact | ≈ approximate (uses heuristic split) |
**80 languages use `tex`**, **2 use `composite`** (`en-us`, `en-gb`), **4 use
`heuristic`** (`ru`, `uk`, `be`, `bg`). The default mode is **`tex`**.
> **Why `tex` count is approximate:** TeX hyphenation patterns are optimised for
> *line-breaking*, not phonetic syllabification. They respect `hyphenMins` and
> avoid awkward break points, so the number of pieces can differ from the true
> syllable count. For scripts where **one syllable = one vowel** (e.g. Cyrillic
> Slavic languages), TeX under- or over-counts. Those languages use `heuristic`
> mode with a per-language vowel pattern and `"vowelMode": "individual"` so each
> vowel is counted as a syllable. `split_word()` keeps using the exact TeX
> hyphenator regardless of mode.
### Vowel counting: `vowelMode`
The `syllableHeuristics` block accepts a `vowelMode` field:
| `"cluster"` *(default)* | Each run of consecutive vowels = 1 syllable |
| `"individual"` | Each vowel letter = 1 syllable (Slavic Cyrillic) |
```rust
use readsight::ReadSight;
// Russian uses heuristic + vowelMode "individual"
let ru = ReadSight::new("ru")?;
assert_eq!(ru.syllable_count("дыхание"), 4);
# Ok::<(), readsight::Error>(())
```
### Example: "hyphenation" in each mode
```rust
use readsight::ReadSight;
let en = ReadSight::new("en-us")?; // composite mode - heuristic wins
assert_eq!(en.syllable_count("hyphenation"), 4);
assert_eq!(en.split_syllables("hyphenation"), vec!["hyp", "hen", "ati", "on"]); // heuristic
assert_eq!(en.split_word("hyphenation"), vec!["hy", "phen", "ation"]); // TeX
let de = ReadSight::new("de-1996")?; // tex mode
assert_eq!(de.syllable_count("hyphenation"), 4);
assert_eq!(de.split_syllables("hyphenation"), vec!["hy", "phena", "ti", "on"]); // TeX
assert_eq!(de.split_word("hyphenation"), vec!["hy", "phena", "ti", "on"]); // same
# Ok::<(), readsight::Error>(())
```
> **Tip:** `split_word()` always uses the TeX hyphenator (exact).
> `split_syllables()` may use the heuristic split (approximate) in
> `composite`/`heuristic` modes. Syllable *counts* are exact in
> `heuristic`/`composite` mode; in `tex` mode they follow TeX break points and
> are approximate (see note above).
> **Note:** `add_hyphenations()` adds overrides to the TeX hyphenator. These
> affect `split_word()` but NOT `split_syllables()` in `composite`/`heuristic`
> modes (the heuristic counter doesn't see them).
## Examples
Run the bundled demo to see ReadSight in action:
```sh
cargo run --example demo
```
This scores the plain/legal texts side by side and outputs:
- **Readability grid** with every applicable formula, its score, grade level and interpretation
- **Syllable breakdown** with heuristic split and TeX hyphenation points
- **Text statistics** — letters, words, sentences, syllables, and a syllable histogram
Compare a short sample across several languages:
```sh
cargo run --example multilingual
```
## Supported Languages
86 languages across 19 writing systems: Latin, Cyrillic, Arabic, Hebrew,
Devanagari, Bengali, Tamil, Thai, Greek, Armenian, Georgian, Gujarati, Gurmukhi,
Kannada, Malayalam, Odia, Telugu, Ethiopic, Coptic.
```rust
use readsight::ReadSight;
let _ru = ReadSight::new("ru")?; // Russian
let _de = ReadSight::new("de-1996")?; // German (1996 reform)
let _es = ReadSight::new("es")?; // Spanish
let _th = ReadSight::new("th")?; // Thai
// List all supported languages (sorted)
let langs = ReadSight::supported_languages(None);
assert_eq!(langs.len(), 86);
# Ok::<(), readsight::Error>(())
```
`ReadSight::supported_languages(None)` returns all 86 codes, sorted, mirroring
the JSON files under [`data/languages`](data/languages) exactly:
```text
af, ar, as, be, bg, bn, ca, cop, cs, cu, cy, da, de-1901, de-1996, de-ch-1901,
el-monoton, el-polyton, en-gb, en-us, eo, es, et, eu, fa, fi, fi-x-school, fr,
fur, ga, gl, grc, gu, he, hi, hr, hsb, hu, hy, ia, id, is, it, ka, kk, kmr, kn,
la, la-x-classic, la-x-liturgic, lt, lv, mk, ml, mn-cyrl, mn-cyrl-x-lmc, mr,
mul-ethi, nb, nl, nn, oc, or, pa, pi, pl, pms, pt, rm, ro, ru, sa, sh-cyrl,
sh-latn, sk, sl, sq, sr-cyrl, sv, ta, te, th, tk, tr, uk, vi, zh-latn-pinyin
```
## Readability Formulas
### Universal (all 86 languages)
| Gunning Fog | `gunning_fog` | `gunning_fog()` | Syllable-based |
| SMOG Index | `smog` | `smog_index()` | Syllable-based |
| Coleman-Liau | `coleman_liau` | `coleman_liau()` | Letter-based |
| ARI | `ari` | `automated_readability_index()` | Letter-based |
| LIX | `lix` | `lix()` | Letter-based |
### Language-Specific
| `en-us`, `en-gb`, `de-*`, `ru`, `es`, `it`, `fr`, `nl`, `pt`, `tr` (12 codes) | Flesch Reading Ease, Flesch-Kincaid Grade Level |
| English (`en-us`, `en-gb`) | Dale-Chall\*, Spache\* |
| German (`de-1996`, `de-1901`, `de-ch-1901`) | Wiener Sachtextformel (4 variants) |
| Spanish (`es`) | Fernández-Huerta, Szigriszt-Pazos, Gutiérrez-Polini, Crawford |
| Italian (`it`) | Gulpease |
| Polish (`pl`) | FOG-PL |
| Arabic (`ar`) | OSMAN |
> \* **Note:** Dale-Chall and Spache use a syllable-based heuristic to estimate
> difficult words (1-syllable ≈ easy). This is a simplified estimation, not the
> original Dale/Spache word lists.
Generic dispatching by name:
```rust
use readsight::ReadSight;
let rs = ReadSight::new("de-1996")?;
let r = rs.score("gunning_fog", "Ein einfacher deutscher Satz. Und noch einer.")?;
assert_eq!(r.formula_name, "gunning_fog");
// Wiener Sachtextformel supports variants 1..=4
let w = rs.wiener_sachtextformel("Ein einfacher deutscher Satz. Und noch einer.", 1)?;
assert_eq!(w.formula_name, "wiener_sachtextformel_1");
# Ok::<(), readsight::Error>(())
```
## FormulaResult
```rust
# use readsight::FormulaResult;
# fn _doc(result: FormulaResult) {
result.score; // f64 - raw (rounded) formula score
result.grade_level; // Option<f64> - normalized grade level (FKGL, GF, SMOG, CL, ARI, Spache)
result.interpretation; // String - qualitative interpretation ("Easy", "Hard", ...)
result.formula_name; // String - formula key
result.language_code; // String - language code used
result.inputs; // BTreeMap<String, f64> - intermediate values for debugging
# }
```
## API Reference
`ReadSight` (aliased as `Engine`) is the entry point.
### Text / syllable methods
```rust,ignore
engine.syllable_count(word: &str) -> i64
engine.split_word(word: &str) -> Vec<String>
engine.split_syllables(word: &str) -> Vec<String>
engine.word_count(text: &str) -> i64
engine.sentence_count(text: &str) -> i64
engine.letter_count(text: &str) -> i64
engine.total_syllables(text: &str) -> i64
engine.average_syllables_per_word(text: &str) -> f64
engine.average_words_per_sentence(text: &str) -> f64
engine.polysyllable_count(text: &str, count_proper_nouns: bool) -> i64
engine.words_with_more_than_n_syllables(text: &str, n: i64, count_proper_nouns: bool) -> i64
engine.histogram_syllables(text: &str) -> BTreeMap<i64, i64>
engine.analyze(text: &str) -> Result<TextStatistics>
engine.add_hyphenations(iter) // (word, "hy-phen-a-ted") overrides
```
### Formula methods
```rust,ignore
engine.score(name: &str, text: &str) -> Result<FormulaResult>
engine.flesch_reading_ease(text) / flesch_kincaid_grade_level(text)
engine.gunning_fog(text) / smog_index(text) / coleman_liau(text)
engine.automated_readability_index(text) / lix(text)
engine.gulpease(text) / fernandez_huerta(text) / szigriszt_pazos(text)
engine.gutierrez_polini(text) / crawford(text) / fog_pl(text)
engine.dale_chall(text) / spache(text) / osman(text)
engine.wiener_sachtextformel(text, variant: i32) // variant 1..=4
```
### Static
```rust,ignore
ReadSight::supported_languages(config: Option<&Config>) -> Vec<String>
```
## Data Source
By default all data is embedded at compile time (via `include_dir!`): 86 language
JSON files and 86 `hyph-*.tex` pattern files under [`data/`](data). To load from
the filesystem instead:
```rust,no_run
use readsight::{Config, ReadSight};
let config = Config::from_dirs("data/patterns", "data/languages");
let rs = ReadSight::with_config("en-us", config)?;
# Ok::<(), readsight::Error>(())
```
Custom hyphenation overrides affect `split_word` (not `split_syllables` in
`composite`/`heuristic` modes):
```rust
use readsight::ReadSight;
let engine = ReadSight::new("en-us")?;
engine.add_hyphenations([("customword", "cus-tom-word")]);
assert_eq!(engine.split_word("customword"), vec!["cus", "tom", "word"]);
# Ok::<(), readsight::Error>(())
```
## Architecture
```text
ReadSight (facade, aliased as Engine)
├── TextAnalyzer (syllable counting, text metrics)
│ │ ├── HeuristicSyllableCounter (vowel patterns + word list, vowelMode)
│ │ └── TexSyllableCounter → LiangHyphenator (TeX hyphenation)
│ ├── LiangHyphenator
│ │ ├── parse_tex / TexSource (parses .tex from hyph-utf8)
│ │ ├── PatternsCollection (pattern data)
│ │ └── HyphenationExceptionsCollection (word overrides)
│ └── TextSplitter (word/sentence/letter counting)
├── Language (JSON config per language, syllableMode + formula configs)
└── FormulaRegistry (17 formulas)
├── FleschReadingEase / FleschKincaidGradeLevel (lang-specific coefficients)
├── GunningFog, SmogIndex, ColemanLiau, ARI, LIX (universal)
└── WienerSachtextformel, Gulpease, FernandezHuerta, ... (lang-specific)
```
## Development
```sh
cargo test # unit + golden parity + smoke tests
cargo clippy --all-targets -- -D warnings # lints (clean)
cargo fmt --check # formatting (clean)
```
The suite currently runs **90 integration tests**, plus the crate's doc tests
(including the runnable Rust examples in this README, which are compiled and
executed by `cargo test`):
- **Golden parity** against the PHP reference: `supported_formulas`, `analyze`,
and the applicable formulas over all 86 languages, plus per-word syllable
vectors for a 20-language subset ([`tests/golden`](tests/golden)).
- **Ported unit tests** from the PHP/Python suites (hyphenation, syllables, text
splitting, formulas, grade-level interpretation).
- **Full-language smoke test** that builds every language and runs every
supported formula.
The library is `#![forbid(unsafe_code)]`.
## License
MIT — see [LICENSE](LICENSE). Author of the original library: Yevhen Leonidov.
Readability data and hyphenation patterns originate from the canonical PHP
project and the [`hyph-utf8`](https://ctan.org/pkg/hyph-utf8) package; the TeX
pattern files are packaged under their original licenses (see individual file
headers).