# ReadSight → Rust Port Guide (`readsight-rs`)
> An exhaustive, implementation-ready specification for building a **complete and
> byte-accurate Rust port** of ReadSight. This document is written to be handed
> directly to an AI coding agent (or a human) and followed top-to-bottom. Every
> algorithm, constant, coefficient, rounding rule, and edge case that matters for
> output parity is spelled out.
## 0. Source repositories (the ground truth)
You are porting an existing, tested library. **Always cross-check against the
reference implementations** — do not invent behaviour.
| **PHP (canonical)** | https://github.com/MADEVAL/ReadSight | Primary source of truth. Class/algorithm layout below mirrors it 1:1. |
| **Python port** | https://github.com/MADEVAL/ReadSightPy | Second reference; a proven, idiomatic port of the same logic. Use it to disambiguate anything unclear in PHP. |
| Python on PyPI | https://pypi.org/project/readsight/ | Installable reference (`pip install readsight`) for generating golden vectors. |
| TeX hyphenation data | https://ctan.org/pkg/hyph-utf8 | Origin of the `.tex` pattern files (version 2026-02-21). |
Key raw files you will re-read constantly:
- PHP source tree: `https://github.com/MADEVAL/ReadSight/tree/main/src`
- PHP data: `https://github.com/MADEVAL/ReadSight/tree/main/data` (`languages/*.json`, `patterns/hyph-*.tex`)
- PHP tests: `https://github.com/MADEVAL/ReadSight/tree/main/tests`
- Python source: `https://github.com/MADEVAL/ReadSightPy/tree/main/src/readsight`
- Python data (identical JSON + `.tex`): `https://github.com/MADEVAL/ReadSightPy/tree/main/src/readsight/data`
**Rule of thumb:** if PHP and Python ever disagree, PHP wins (it is canonical),
but the disagreement is a red flag — investigate before continuing.
---
## 1. What ReadSight does (mental model)
ReadSight measures **text readability** across **86 languages** using **17
readability formulas**, plus a **syllable counter** and **word hyphenator** based
on the **Frank M. Liang (TeX) hyphenation algorithm**. Zero network, zero heavy
deps; all data ships in the package (`data/languages/*.json`,
`data/patterns/hyph-*.tex`).
The pipeline for a single call is:
```
text ──► TextSplitter (words / sentences / letters via language regexes)
──► SyllableCounter (per word: tex | heuristic | composite)
──► TextStatistics (aggregate metrics)
──► Formula.calculate(stats, language) ──► FormulaResult
```
Syllable counting has two independent outputs:
- **count** (`syllable_count`) — a number.
- **split** (`split_word` = exact TeX hyphenation points; `split_syllables` =
possibly-approximate split depending on mode).
---
## 2. Target crate layout
Create a library crate `readsight` (binary demo optional). Recommended layout —
it mirrors the PHP namespaces so cross-referencing is trivial:
```
readsight-rs/
├── Cargo.toml
├── build.rs # optional: embed / precompile data
├── data/ # COPY verbatim from PHP repo
│ ├── languages/*.json # 86 files
│ └── patterns/hyph-*.tex # 86 files
├── src/
│ ├── lib.rs # re-exports; crate docs
│ ├── engine.rs # ReadSight facade (PHP Engine.php)
│ ├── config.rs # Config (paths / data source)
│ ├── error.rs # Error enum (PHP Exception/*)
│ ├── language/
│ │ ├── mod.rs
│ │ ├── language.rs # Language struct (parsed JSON)
│ │ ├── code.rs # LanguageCode::normalize
│ │ ├── script.rs # Script enum
│ │ └── repository.rs # JsonLanguageRepository
│ ├── hyphenation/
│ │ ├── mod.rs
│ │ ├── hyphenator.rs # trait Hyphenator + LiangHyphenator
│ │ ├── pattern.rs # Pattern + PatternsCollection
│ │ ├── exceptions.rs # HyphenationOverride + collection
│ │ ├── tex_source.rs # TeX .tex parser
│ │ └── cache.rs # optional JSON pattern cache
│ ├── syllable/
│ │ ├── mod.rs
│ │ ├── counter.rs # trait SyllableCounter
│ │ ├── tex.rs # TexSyllableCounter
│ │ ├── heuristic.rs # HeuristicSyllableCounter (+ vowelMode)
│ │ └── composite.rs # CompositeSyllableCounter
│ ├── text/
│ │ ├── mod.rs
│ │ ├── splitter.rs # TextSplitter
│ │ ├── analyzer.rs # TextAnalyzer
│ │ └── statistics.rs # TextStatistics
│ └── formula/
│ ├── mod.rs # trait Formula + FormulaResult + registry
│ ├── registry.rs
│ ├── grade_level.rs # GradeLevelInterpretation
│ ├── helper.rs # TextStatisticsHelper
│ └── impls/ # one file per formula (17)
└── tests/
├── golden/ # golden vectors dumped from PHP/Python
├── hyphenation.rs
├── syllable.rs
├── formula.rs
└── multilingual.rs
```
### 2.1 `Cargo.toml`
```toml
[package]
name = "readsight"
version = "1.0.0"
edition = "2021"
rust-version = "1.74"
license = "MIT"
description = "Multilingual readability library — 86 languages, 17 formulas, TeX-based syllable counting via the Frank M. Liang algorithm."
[dependencies]
regex = "1" # Unicode \p{L}; NOTE: no look-around (not needed)
serde = { version = "1", features = ["derive"] }
serde_json = "1"
include_dir = "0.7" # embed data/ into the binary (recommended)
[dev-dependencies]
serde_json = "1"
```
**Do not** pull in an ICU crate for the hyphenation/formula logic. PHP uses
`mbstring` + PCRE; Python uses `regex`. The `regex` crate with default Unicode
features is the correct analogue. See §12 for the exact string-op mapping.
---
## 3. Data embedding
PHP resolves data via `Config { patternsDir, languagesDir, cacheDir }` (see
`src/Config.php`). Mirror that, **and** allow embedded data as the default so the
crate works with no filesystem.
```rust
// config.rs
pub struct Config {
pub source: DataSource,
pub cache_dir: Option<PathBuf>, // caching is OPTIONAL in Rust (see §7.4)
}
pub enum DataSource {
Embedded, // default: include_dir!("$CARGO_MANIFEST_DIR/data")
Filesystem { patterns_dir: PathBuf, languages_dir: PathBuf },
}
```
- Copy the **entire** `data/` directory from the PHP repo verbatim (86 JSON + 86
`.tex`). Do not regenerate or "clean up" — file contents (including the Cyrillic
vowel-mode configs) are part of the contract.
- Embed with `include_dir!` (or `rust-embed`). Look-ups: languages by
`"{code}.json"`, patterns by `"hyph-{code}.tex"`.
---
## 4. Language layer
### 4.1 `LanguageCode::normalize` (`src/Language/LanguageCode.php`)
```rust
pub fn normalize(code: &str) -> String { code.trim().to_lowercase() }
```
Unicode-aware lowercase (PHP `mb_strtolower`). Rust `str::to_lowercase` matches.
### 4.2 `Script` enum (`src/Language/Script.php`)
String-backed enum, exact variants (value == name):
`Latin, Cyrillic, Arabic, Hebrew, Devanagari, Bengali, Greek, Armenian, Georgian,
Thai, Tamil, Telugu, Kannada, Malayalam, Gujarati, Gurmukhi, Odia, Ethiopic,
Coptic, CJK, Other`. Deserialize from the JSON `script` string.
### 4.3 `Language` struct (`src/Language/Language.php`)
Parsed from a language JSON file. Fields (defaults in parentheses):
| `code` | `code` | String | — |
| `name` | `name` | String | — |
| `native_name` | `nativeName` | String | — |
| `script` | `script` | Script | — |
| `min_hyphen_left` | `hyphenMins.left` | i32 | — |
| `min_hyphen_right` | `hyphenMins.right` | i32 | — |
| `letter_pattern` | `letterPattern` | String (regex body) | — |
| `word_split_pattern` | `wordSplitPattern` | String (regex body) | — |
| `sentence_boundary_pattern` | `sentenceBoundaryPattern` | String (regex body) | — |
| `formula_configs` | `formulas` | Map<String, Map<String, Value>> | `{}` |
| `syllable_heuristics` | `syllableHeuristics` | Option<Map<String, Value>> | `None` |
| `syllable_mode` | `syllableMode` | String | `"tex"` |
Methods: `supports_formula(name) -> formula_configs.contains_key(name)`,
`get_formula_config(name) -> Option<&Map>`, `get_supported_formulas() ->
Vec<String>` (= keys of `formula_configs`).
> ⚠️ **Critical distinction (do not conflate):** `Language::get_supported_formulas`
> (JSON keys) is **NOT** the list used by `Engine::get_supported_formulas`. The
> Engine list comes from each `Formula`'s own `supported_languages()` (see §9.2).
> The JSON `formulas` block only supplies **coefficients** (FRE `base/aslMult/
> aswMult`) and the **LIX `longWordThreshold`**.
### 4.4 `JsonLanguageRepository` (`src/Language/JsonLanguageRepository.php`)
- `find(code)`: normalize → look up `"{code}.json"` in the data source → parse →
memoize in an internal cache map. Missing file ⇒ `UnsupportedLanguage` error.
- `list_codes()`: enumerate `*.json`, strip extension, **sorted ascending**.
- `exists(code)`: normalize then check presence.
---
## 5. Hyphenation layer (the Liang algorithm) — port with surgical precision
This is the highest-risk area for divergence. Port it **line by line** from
`src/Hyphenation/`, then validate against golden vectors (§10).
### 5.1 `Pattern` (`Pattern.php`)
```rust
pub struct Pattern { pub chars: Vec<String>, pub weights: Vec<i32>, pub length: usize }
// length = chars.len()
```
`chars` are Unicode "characters" (scalar values as 1-char strings); `weights.len()
== chars.len() + 1`.
### 5.2 `PatternsCollection` (`PatternsCollection.php`)
Stores a map **key → weights-as-digit-string**:
- `add(pattern)`: `key = pattern.chars.concat()`, `value = pattern.weights.map(to_string).concat()`; track `max_pattern_length = max(len, chars.len())`.
- `get_weights(subword) -> Option<&str>`.
- `max_length() -> usize`.
In Rust prefer `HashMap<String, Vec<u8>>` (weights as small ints) but keep the
**digit-string semantics** in mind when reading PHP: `weights[offset]` indexes a
byte/char that is a single ASCII digit.
### 5.3 `HyphenationOverride` + `HyphenationExceptionsCollection`
DTO `{ word, hyphenated }`. Collection is a `HashMap<String,String>`:
key = word (lowercased, hyphens removed), value = hyphenated form (with `-`).
Methods `add/has/get/all/count/is_empty`.
### 5.4 TeX parser `TexSource` (`Source/TexSource.php`)
Parses a hyph-utf8 `.tex` file. Read the file as UTF-8 **line by line**. Maintain
state `command: Option<String>` and `in_braces: bool`. For each line iterate
**byte offset** `offset` while `offset < line.len()` (bytes):
1. `%` and not `in_braces` ⇒ rest of line is a comment; `break` to next line.
2. `\` and not `in_braces` ⇒ match regex `^\\([a-zA-Z]+)` on the remaining slice;
if it matches, set `command = group1`, advance by full match length; else
advance 1.
3. `{` ⇒ if `command.is_some()` set `in_braces = true`; advance 1.
4. `}` and `in_braces` ⇒ `in_braces=false; command=None`; advance 1.
5. If `in_braces`:
- command `"patterns"`: match `^(\S+)` (Unicode) on remaining slice ⇒ token;
`parse_pattern_token(token)`; if `Some`, add to patterns; advance by match
byte-length.
- command `"hyphenation"`: match `^(\S+)` ⇒ token; `word = token.replace('-',"")`
lowercased; `hyphenated = token.lowercased()`; add override; advance.
6. Otherwise advance 1.
Return `{ patterns, exceptions, max_pattern_length }`.
Only the `\patterns{…}` and `\hyphenation{…}` blocks matter; everything else
(comments, `\message`, `\endinput`, etc.) is ignored by the state machine above.
**`parse_pattern_token(token)`** (the trickiest bit — reproduce exactly):
- Split `token` into Unicode characters.
- `numbers = String::new()`, `chars: Vec<String>`, `expect_number = true`,
`has_digit = false`.
- For each `ch`:
- if `ch` is a digit (PHP `is_numeric` on a single char): `numbers.push(ch)`,
`has_digit = true`, `expect_number = false`.
- else: if `expect_number` then `numbers.push('0')`; `chars.push(ch)`;
`expect_number = true`.
- After loop: if `expect_number` then `numbers.push('0')`.
- If `chars.is_empty() || !has_digit` ⇒ return `None`.
- `weights = numbers.chars().map(|d| d.to_digit(10))` as `Vec<i32>`.
- Return `Some(Pattern { chars, weights })`.
Worked example: token `a2ch` → chars `[a,c,h]`, numbers `"0200"` →
weights `[0,2,0,0]`, key `"ach"`, weight-string `"0200"`.
Token `.ach4` → chars `[.,a,c,h]`, weights `[0,0,0,0,4]`.
### 5.5 `LiangHyphenator` (`LiangHyphenator.php`)
Constructor: `patterns`, `exceptions`, `min_hyphen_left` (default 2),
`min_hyphen_right` (default 2), plus a mutable `user_hyphenations: HashMap<String,String>`.
`add_hyphenations(map)`: insert each `(word.to_lowercase(), hyphenated.to_lowercase())`.
**`hyphenate(word) -> Vec<String>`** — operate on Unicode chars (`mb_strlen`/
`mb_substr` == iterate `chars()`):
```
let n = word.chars().count();
if n == 0 { return vec![]; }
if n < min_hyphen_left + min_hyphen_right { return vec![word.to_string()]; }
let lower = word.to_lowercase();
if let Some(h) = user_hyphenations.get(&lower) { return split_by_hyphenation(h, word); }
if let Some(h) = exceptions.get(&lower) { return split_by_hyphenation(h, word); }
split_by_patterns(word, n, &lower)
```
**`count_syllables(word)`** = `let p = hyphenate(word); if p.is_empty() {0} else {p.len()}`.
**`split_by_hyphenation(hyphenated, original)`**: walk `hyphenated` char by char;
on `-` push the accumulated part and reset; otherwise append the **original**
word's next char (preserving original case). Push trailing non-empty part. (This
maps hyphen positions from the lowercased override onto the original-cased word.)
**`split_by_patterns(word, word_len, word_lower)`** — reproduce indices EXACTLY:
```
text = format!(".{}.", word_lower); // note leading & trailing '.'
let text_chars: Vec<char> = text.chars().collect();
let text_length = word_len + 2;
let mut pattern_length = patterns.max_length();
if pattern_length > text_length { pattern_length = text_length; }
let mut scores: HashMap<usize,i32> = {}; // (PHP uses a sparse array)
let end = text_length - min_hyphen_right;
for start in 0..end {
let max_len = min(pattern_length, text_length - start);
for len in 1..=max_len {
let subword: String = text_chars[start..start+len].iter().collect();
let Some(weights) = patterns.get_weights(&subword) else { continue };
for (offset, w) in weights.iter().enumerate() { // weights.len() == its char count
let idx = start + offset;
let score = *w as i32;
if scores.get(&idx).map_or(true, |s| score > *s) { scores.insert(idx, score); }
}
}
}
// Reconstruct parts. `word` chars are 0-indexed; boundary '.' shifts by 1.
let word_chars: Vec<char> = word.chars().collect();
let mut parts: Vec<String> = vec![];
let mut part: String = word_chars[0..min_hyphen_left].iter().collect();
let break_end = text_length - min_hyphen_right;
for i in (min_hyphen_left + 1)..break_end {
if let Some(score) = scores.get(&i) {
if score & 1 != 0 { parts.push(std::mem::take(&mut part)); }
}
part.push(word_chars[i - 1]); // (i-1) maps text index → word index
}
for i in break_end..(text_length - 1) {
part.push(word_chars[i - 1]);
}
if !part.is_empty() { parts.push(part); }
parts
```
Notes:
- The **weight length equals the stored pattern's char count** (i.e. `chars.len()+1`
from parsing → but stored as the digit string; iterate its digits).
- Odd score ⇒ hyphenation point (`score & 1`).
- The two trailing loops append the final `min_hyphen_right` region without
considering break points.
- Do not "optimize" the index arithmetic; replicate it and lean on golden tests.
`trait Hyphenator { fn hyphenate(&self, w:&str)->Vec<String>; fn count_syllables(&self, w:&str)->usize; }`.
---
## 6. Syllable layer (`src/Syllable/`)
`trait SyllableCounter { fn count_syllables(&self,w:&str)->i64; fn split_syllables(&self,w:&str)->Vec<String>; }`
### 6.1 `TexSyllableCounter`
Thin delegate to the hyphenator: `count = hyphenator.count_syllables(w)`,
`split = hyphenator.hyphenate(w)`.
### 6.2 `HeuristicSyllableCounter` (`HeuristicSyllableCounter.php`) — includes v1.0.7 `vowelMode`
Config fields (all from `syllableHeuristics`):
`problem_words: Map<String,i64>`, `subtract_patterns: Vec<String>`,
`add_patterns: Vec<String>`, `prefixes: Map<String,i64>`, `suffixes: Map<String,i64>`,
`vowel_pattern: String` (default `"[aeiouy]"`), `vowel_mode: "cluster"|"individual"`
(default `"cluster"`; any value other than `"individual"` ⇒ `"cluster"`).
Precompute `vowel_chars = vowel_pattern.trim_matches(['[',']'])`.
**`count_syllables(word)`** — exact order:
1. `word = word.trim()`; if empty ⇒ `0`.
2. `lower = word.to_lowercase()`. If `problem_words` contains `lower` ⇒ return it.
3. `clean = lower` with all non-letters removed (`\p{L}` kept). If empty ⇒ `1`.
4. `affix = 0`. For each `(prefix, n)` in `prefixes` **in map order**: if `clean`
starts with `prefix`, strip it and `affix += n`.
5. For each `(suffix, n)` in `suffixes`: if `clean` ends with `suffix`, strip it and `affix += n`.
6. `vowel_groups = count_vowel_groups(clean)` (see below).
7. `count = vowel_groups + affix`.
8. For each `p` in `subtract_patterns`: `count -= if regex(p).is_match(clean) {1} else {0}`.
9. For each `p` in `add_patterns`: `count += if regex(p).is_match(clean) {1} else {0}`.
10. Return `max(count, 1)`.
> ⚠️ Steps 8–9 mirror PHP `preg_match` which returns **0 or 1** (found / not
> found), *not* the number of occurrences. So each pattern adjusts the count by
> at most ±1. Do **not** use `find_iter().count()`.
**`count_vowel_groups(clean)`**:
- `"individual"`: number of characters in `clean` matching `[vowel_chars]`
(i.e. `regex("[{vowel_chars}]").find_iter(clean).count()`).
- `"cluster"` (default): split `clean` by `[^{vowel_chars}]+` and count non-empty
pieces (== number of maximal vowel runs).
Example: `дыхание` with Russian vowels → cluster `3` (`ы,а,ие`), individual `4`
(`ы,а,и,е`). Russian/Ukrainian/Belarusian/Bulgarian ship `vowelMode:"individual"`.
**`split_syllables(word)`** — arithmetic (approximate) split into
`count_syllables(word)` roughly-equal char chunks:
- `count = count_syllables(word)`; if `count <= 1` ⇒ `[word]` (or `[]` if empty).
- `len = word.chars().count()`; if `count >= len` ⇒ one part per char.
- else split into `count` chunks: base `len/count`, the first `len % count` chunks
get one extra char. Clamp the last chunk to not overrun. Use `char`-based
substrings.
**`has_rules()`** = config present AND (`problem_words` OR `subtract_patterns` OR
`add_patterns` OR `prefixes` OR `suffixes` non-empty). Used by composite.
**`has_word(w)`** = `problem_words` contains `w.trim().to_lowercase()`.
### 6.3 `CompositeSyllableCounter` (`CompositeSyllableCounter.php`)
Holds an ordered `chain: Vec<Box<dyn SyllableCounter>>` (built as `[heuristic, tex]`).
For both `count_syllables` and `split_syllables`, iterate the chain:
- If the element is a `HeuristicSyllableCounter`: if `has_rules()` ⇒ use it and
**return**; else skip to next.
- Otherwise ⇒ use it and return.
- Fallback: last element, else `1` / `[word]`.
> Consequence to internalize: when the heuristic has rules (e.g. `en-us`, which
> ships a big `problemWords` list), **composite behaves as pure heuristic** for
> `count`/`split_syllables`. TeX is still used by `split_word` (which bypasses the
> counter and calls the hyphenator directly). Implement the trait-object downcast
> via an `enum SyllableCounterKind { Tex(..), Heuristic(..) }` chain rather than
> `dyn` + `Any` if you prefer — the observable behaviour is what matters.
### 6.4 Engine wiring (`Engine::loadSyllableCounter`)
```
tex = TexSyllableCounter(hyphenator)
mode = language.syllable_mode
if mode == "heuristic" { return heuristic }
return Composite([heuristic, tex]) // mode == "composite" (or anything else)
```
---
## 7. Text layer (`src/Text/`)
### 7.1 `TextSplitter` (`TextSplitter.php`)
Built from a `Language`. Compile the three regexes once (with Unicode):
- words: `Regex::new(&language.word_split_pattern)` used as a **splitter**.
- sentences / letters: `Regex::new(&format!("{}", language.sentence_boundary_pattern))`
and `Regex::new(&language.letter_pattern)` (PHP wraps them as `/pat/u`).
Methods:
- `split_words(text)`: `trim`; if empty ⇒ `[]`. Split text by the word regex,
**filter out empty strings**, collect. (PHP uses `mb_split`; the `regex` crate's
`Regex::split` is the analogue — remember to drop empties, incl. leading/trailing.)
- `split_sentences(text)`: `trim`; split by sentence-boundary regex with "no empty"
semantics, then `trim` each piece. (Only used indirectly; keep for parity.)
- `count_letters(text)`: `trim`; if empty ⇒ 0; else number of matches of the
letter regex (`find_iter().count()`).
- `count_words(text)`: `split_words(text).len()`.
- `count_sentences(text)`: `trim`; if empty ⇒ 0; else `m = matches of sentence
regex`; return `if m == 0 {1} else {m}`.
- `count_long_words(text, threshold)`: number of words whose `count_letters(word)
> threshold`.
### 7.2 `TextStatistics` (`TextStatistics.php`) — plain data struct
```rust
pub struct TextStatistics {
pub letter_count: i64,
pub word_count: i64,
pub sentence_count: i64,
pub syllable_count: i64,
pub polysyllable_count: i64,
pub average_syllables_per_word: f64,
pub average_words_per_sentence: f64,
pub long_word_count: i64,
pub syllable_histogram: BTreeMap<i64, i64>, // syllables -> #words, ascending keys
}
```
### 7.3 `TextAnalyzer` (`TextAnalyzer.php`)
Holds `hyphenator`, `syllable_counter`, `text_splitter`, `language`.
- `split_word(w)` → `hyphenator.hyphenate(w)`.
- `split_syllables(w)` → `syllable_counter.split_syllables(w)`.
- `syllable_count(w)` → `syllable_counter.count_syllables(w)`.
- `word_count / sentence_count / letter_count` → delegate to splitter.
- `total_syllables(text)` → sum of `count_syllables` over `split_words`.
- `average_syllables_per_word(text)` → `words = split_words`; if empty ⇒ `0.0`;
else `sum(count_syllables)/words.len()`.
- `average_words_per_sentence(text)` → `w = count_words`, `s = count_sentences`;
if `s == 0` ⇒ `w as f64`; else `w / s`.
- `words_with_more_than_n_syllables(text, n, count_proper_nouns=true)`: over
`split_words`, count words with `count_syllables > n`; when `count_proper_nouns
== false`, only count a word if its first char is **not** uppercase
(`first_char != first_char.to_uppercase()`).
- `polysyllable_count(text, count_proper_nouns=true)` = `words_with_more_than_n_syllables(text, 2, …)`.
- `histogram_syllables(text)`: over words, `s = count_syllables`; skip `s == 0`;
`histogram[s] += 1`; return ascending-key map.
- **`analyze(text) -> Result<TextStatistics, Error>`** (this is the hot path;
reproduce exactly):
1. `text = text.trim()`.
2. `words = split_words(text)`; `word_count = words.len()`; if `0` ⇒ `Err(EmptyText)`.
3. `letter_count = count_letters(text)`; `sentence_count = count_sentences(text)`.
4. Loop words: `s = count_syllables(word)`; `total_syllables += s`; if `s > 2`
⇒ `polysyllable_count += 1`; if `s > 0` ⇒ `histogram[s] += 1`.
5. `sentence_for_avg = if sentence_count == 0 {1} else {sentence_count}`.
6. LIX threshold: read `language.get_formula_config("lix")["longWordThreshold"]`
if numeric else `6`; `long_word_count = count_long_words(text, threshold)`.
7. Build `TextStatistics { …, average_syllables_per_word: total_syllables/word_count,
average_words_per_sentence: word_count/sentence_for_avg, … }` (float division).
- `add_hyphenations(map)`: forward to the hyphenator if it is a `LiangHyphenator`.
> Note: `analyze` computes `polysyllable_count` as **> 2** syllables. So does
> `words_with_more_than_n_syllables(_,2)`. Keep both consistent.
### 7.4 Caching (optional)
PHP caches parsed patterns as JSON (`Cache/JsonPatternCache.php`, version `"2.0"`,
file `syllable.{code}.json`). In Rust this is a **pure performance optimization**
and may be omitted for v1. If you implement it, keep the same on-disk JSON shape
for interop:
```json
{ "version":"2.0",
"patterns":[{"chars":["a","c","h"],"weights":[0,2,0,0]}, …],
"exceptions":{"word":"hy-phen-ated", …},
"maxPatternLength": <int> }
```
A better Rust-native approach: precompile all 86 pattern sets at build time in
`build.rs` into a static structure. Optional; not required for correctness.
---
## 8. Error type (`src/Exception/`)
PHP has an abstract `ReadabilityEngineException` and concrete subclasses. Model as
one enum:
```rust
#[derive(Debug)]
pub enum Error {
UnsupportedLanguage(String), // withCode
UnsupportedFormula { formula: String, language: String },
EmptyText,
PatternFileNotFound(String),
PatternParse { token: String, line: usize, file: String },
Io(std::io::Error),
Json(serde_json::Error),
}
```
Match the message formats where practical, e.g. UnsupportedFormula:
`"Formula \"{formula}\" is not supported for language \"{language}\"."`
---
## 9. Formula layer (`src/Formula/`) — the numeric heart
### 9.1 Contracts
```rust
pub struct FormulaResult {
pub formula_name: String,
pub language_code: String,
pub score: f64,
pub grade_level: Option<f64>,
pub interpretation: String,
pub inputs: BTreeMap<String, f64>, // debug values; ints stored as f64
}
pub trait Formula {
fn name(&self) -> &'static str;
fn description(&self) -> &'static str;
fn supported_languages(&self) -> &'static [&'static str]; // ["*"] == all
fn calculate(&self, stats: &TextStatistics, language: &Language) -> FormulaResult;
}
```
### 9.2 `FormulaRegistry` (`FormulaRegistry.php` + `FormulaRegistryFactory.php`)
Register all 17 (order below is the PHP factory order; it affects
`list_names()`/`list_for_language()` order, which some tests check):
`AutomatedReadabilityIndex, ColemanLiau, Crawford, DaleChall, FernandezHuerta,
FleschKincaidGradeLevel, FleschReadingEase, FogPL, Gulpease, GunningFog,
GutierrezPolini, Lix, Osman, SmogIndex, Spache, SzigrisztPazos, WienerSachtextformel`.
- `list_for_language(lang)`: for each formula in registration order, include if
`supported_languages() == ["*"]` or contains `lang.code`.
- `calculate(name, lang, stats)`: look up; if missing or not supported for lang ⇒
`Err(UnsupportedFormula)`; else `formula.calculate(stats, lang)`.
### 9.3 Shared helpers
- **`GradeLevelInterpretation::for_score(score)`** (`grade_level.rs`): match on
ascending thresholds (`<=`):
`≤1 Kindergarten, ≤2 1st Grade, ≤3 2nd Grade, ≤4 3rd Grade, ≤5 4th Grade,
≤6 5th Grade, ≤7 6th Grade, ≤8 7th Grade, ≤9 8th Grade, ≤10 9th Grade,
≤11 10th Grade, ≤12 11th Grade, ≤13 12th Grade, ≤16 College, else Graduate`.
- **`TextStatisticsHelper::estimate_difficult_percentage(stats)`** (`helper.rs`):
if `word_count == 0` ⇒ `0.0`; `easy = histogram.get(&1).copied().unwrap_or(0)`;
`difficult = max(word_count - easy, 0)`; return `difficult/word_count * 100.0`.
### 9.4 Rounding — **read this twice**
PHP `round($x, n)` and `round($x)` use **round-half-away-from-zero**. Rust
`f64::round()` is *also* half-away-from-zero, so:
```rust
fn round_to(x: f64, places: i32) -> f64 {
let f = 10f64.powi(places);
(x * f).round() / f
}
```
Use this for every `round($x, n)`. `round($x, 1)` ⇒ `round_to(x,1)`,
`round($x, 2)` ⇒ `round_to(x,2)`, `round($x, 4)` ⇒ `round_to(x,4)`.
**Where interpretation reads the raw vs rounded score:** in almost every formula,
`interpret()` receives the **raw (unrounded)** score while the `score` field is the
rounded value. The **one exception is Szigriszt-Pazos**, which rounds first and
then interprets the rounded value. Reproduce each formula's exact call.
`grade_level` clamps happen **after** rounding: `min(max(round_to(score,1), lo), hi)`.
### 9.5 The 17 formulas — exact definitions
Let `ASL = stats.average_words_per_sentence`, `ASW = stats.average_syllables_per_word`,
`W = max(word_count,1)`, `S = max(sentence_count,1)`, `L = letter_count`,
`poly = polysyllable_count`.
**Universal (`supported_languages() == ["*"]`):**
1. **Gunning Fog** (`gunning_fog`)
- `poly_pct = if word_count>0 { poly/word_count*100 } else {0}`
- `score = 0.4 * (ASL + poly_pct)`
- `grade_level = clamp(round1(score), 0, 19)`; `interpret(raw)`:
`<6 Very Easy, <8 Easy, <12 Standard, <14 Hard, <17 Very Hard, else Extremely Hard`
- inputs: `asl, polysyllablePct, polysyllableCount, wordCount`
2. **SMOG** (`smog`)
- `score = 1.0430 * sqrt(poly * (30.0 / S)) + 3.1291`
- `grade_level = clamp(round1, 0, 18)`; `interpret = GradeLevelInterpretation::for_score(raw)`
- inputs: `polysyllableCount, sentenceCount`
3. **Coleman-Liau** (`coleman_liau`)
- `Lv = (L / W) * 100`, `Sv = (S / W) * 100`
- `score = 0.0588*Lv - 0.296*Sv - 15.8`
- `grade_level = clamp(round1, 0, 18)`; `interpret = for_score(raw)`
- inputs: `L=round2(Lv), S=round2(Sv), letterCount, wordCount, sentenceCount`
4. **ARI** (`ari`)
- `score = 4.71*(L/W) + 0.5*(W/S) - 21.43`
- `grade_level = clamp(round1, 0, 18)`; `interpret = for_score(raw)`
- inputs: `charsPerWord=round2(L/W), wordsPerSentence=round2(W/S)`
5. **LIX** (`lix`)
- `threshold` = lix config `longWordThreshold` if numeric else 6 (only echoed in inputs)
- `long_pct = if word_count>0 { long_word_count/word_count*100 } else {0}`
- `score = ASL + long_pct`; **round to 2 places**; `grade_level = None`
- `interpret(raw)`: `<25 Children's Books, <30 Simple Texts, <40 Normal / Fiction,
<50 Factual Information, <60 Specialized Texts, else Research / Advanced`
- inputs: `asl, longWordPct=round2, threshold, longWordCount, wordCount`
**Language-specific** (`supported_languages()` lists exact codes):
6. **Flesch Reading Ease** (`flesch_reading_ease`) — langs: `en-us, en-gb, de-1996,
de-1901, de-ch-1901, ru, es, it, fr, nl, pt, tr`
- coefficients from lang JSON `flesch_reading_ease`: `base` (def 206.835),
`aslMult` (def 1.015), `aswMult` (def 84.6)
- `score = base - aslMult*ASL - aswMult*ASW`; round1; `grade_level = None`
- `interpret(raw)`: `≥90 Very Easy, ≥80 Easy, ≥70 Fairly Easy, ≥60 Standard,
≥50 Fairly Hard, ≥30 Hard, else Very Hard`
- inputs: `asl=ASL, asw=ASW`
7. **Flesch-Kincaid Grade Level** (`flesch_kincaid_grade_level`) — same lang list
- `score = 0.39*ASL + 11.8*ASW - 15.59`; round1;
`grade_level = clamp(round1, 0, 18)`
- `interpret(raw)`: `≤1 1st Grade, ≤2 2nd Grade, …, ≤12 12th Grade, ≤16 College,
else Graduate` (its own table — 1st..12th, no Kindergarten)
- inputs: `asl, asw`
8. **Wiener Sachtextformel** (`wiener_sachtextformel`) — langs: `de-1996, de-1901, de-ch-1901`
- `ms = poly/W * 100`, `sl = ASL`, `iw = if word_count>0 { long_word_count/word_count*100 } else {0}`
(⚠ the `threshold=6` argument in PHP is **ignored**; `iw` uses the already-computed
`long_word_count`, which used the language's LIX threshold), `es = one-syllable %
= histogram[1]/W*100`.
- Variants (default variant 1):
- v1: `0.1935*ms + 0.1672*sl + 0.1297*iw - 0.0327*es - 0.875`
- v2: `0.2007*ms + 0.1682*sl + 0.1373*iw - 2.779`
- v3: `0.2963*ms + 0.1905*sl - 1.1144`
- v4: `0.2744*ms + 0.2656*sl - 1.693`
- other ⇒ error
- `score = round1`; `grade_level = clamp(raw_score, 4, 15)` (**clamp uses raw, not rounded**);
`formula_name = "wiener_sachtextformel_{variant}"`; `interpret(raw)`:
`<5 Very Easy, <7 Easy, <9 Standard, <11 Fairly Hard, <13 Hard, else Very Hard`
- inputs: `ms, sl, iw, es, variant`
- Expose `calculate_variant(stats, lang, variant)`; default `calculate` uses variant 1.
9. **Gulpease** (`gulpease`) — lang: `it`
- `score = 89.0 + (300.0*sentence_count - 10.0*letter_count) / W`; round1;
`grade_level = None`
- `interpret(raw)`: `≥80 Easy for elementary school, ≥60 Easy for middle school,
≥40 Easy for high school, else Difficult for high school`
- inputs: `letterCount, wordCount, sentenceCount`
10. **Fernández-Huerta** (`fernandez_huerta`) — lang: `es`
- `score = 206.84 - 1.02*ASL - 60.0*ASW`; round1; `grade_level = None`
- `interpret(raw)`: `≥90 Very Easy, ≥80 Easy, ≥70 Fairly Easy, ≥60 Standard,
≥50 Fairly Difficult, ≥30 Difficult, else Very Difficult`
- inputs: `asl, asw`
11. **Szigriszt-Pazos** (`szigriszt_pazos`) — lang: `es`
- `spw = syllable_count / W`; `syllables_per_100 = round1(spw*100)`;
`score = round1(206.835 - 62.3*spw - ASL)`; `grade_level = None`
- **`interpret(rounded score)`** (the exception): `≥85 Very Easy, ≥75 Easy,
≥65 Fairly Easy, ≥55 Standard, ≥40 Fairly Difficult, ≥30 Difficult, else Very Difficult`
- inputs: `syllablesPer100=syllables_per_100, wordsPerSentence=round1(ASL)`
12. **Gutiérrez-Polini** (`gutierrez_polini`) — lang: `es`
- `score = 95.2 - 9.7*(L/W) - 0.35*ASL`; round1; `grade_level = None`
- `interpret(raw)`: `≥80 Very Easy, ≥70 Easy, ≥50 Standard, ≥30 Difficult, else Very Difficult`
- inputs: `lettersPerWord=round2(L/W), wordsPerSentence=round2(ASL)`
13. **Crawford** (`crawford`) — lang: `es`
- `avg_letters = L/W`, `sent_per_100 = (S/W)*100`
- `score = -0.205*avg_letters + 0.049*sent_per_100 - 3.407`; round1; `grade_level = None`
- `interpret(raw)`: `≥9 Very Easy, ≥7 Easy, ≥5 Standard, ≥3 Difficult, else Very Difficult`
- inputs: `avgLettersPerWord=round2, sentencesPer100Words=round2`
14. **FOG-PL** (`fog_pl`) — lang: `pl`
- `hard_pct = poly/W*100`, `asl = W/S`
- `score = 0.4*(asl + hard_pct)`; round1; `grade_level = clamp(round1, 0, 19)`
- `interpret(raw)`: `<6 Very Easy, <8 Easy, <12 Standard, <14 Hard, else Very Hard`
- inputs: `asl=round2, hardWordsPct=round2, polysyllableCount`
15. **Dale-Chall** (`dale_chall`) — langs: `en-us, en-gb`
- `diff = estimate_difficult_percentage(stats)`
- `raw = 0.1579*diff + 0.0496*ASL`; `adjusted = if diff>5.0 { raw+3.6365 } else { raw }`
- `score = round1(adjusted)`; `grade_level = None`; `interpret(adjusted /*raw*/)`:
`≤4.9 4th grade or below, ≤5.9 5th-6th grade, ≤6.9 7th-8th grade,
≤7.9 9th-10th grade, ≤8.9 11th-12th grade, ≤9.9 College, else Graduate`
- inputs: `difficultWordPct=round1(diff), rawScore=round4(raw), averageWordsPerSentence=ASL`
16. **Spache** (`spache`) — langs: `en-us, en-gb`
- `diff = estimate_difficult_percentage(stats)`
- `score = 0.121*ASL + 0.082*diff + 0.659`; round1;
`grade_level = clamp(round1, 0, 5)`
- `interpret(raw)`: `≤2 1st Grade, ≤2.5 2nd Grade, ≤3 3rd Grade, ≤3.5 4th Grade,
else Above 4th Grade`
- inputs: `averageWordsPerSentence=ASL, difficultWordPct=round2(diff)`
17. **OSMAN** (`osman`) — lang: `ar`
- `asl = W/S`, `avg_letters = L/W`, `hard_pct = poly/W*100`
- `score = 200.0 - 2.0*asl - 1.5*avg_letters - 0.4*hard_pct`; round1; `grade_level = None`
- `interpret(raw)`: `≥90 Very Easy, ≥70 Easy, ≥50 Standard, ≥30 Difficult, else Very Difficult`
- inputs: `asl=round2, avgLetters=round2, hardWordsPct=round2`
---
## 10. Public API — `ReadSight` facade (`src/Engine.php`)
Name the struct `ReadSight` (matches Python) or `Engine` (matches PHP) — expose
both via `pub use`. Constructor mirrors PHP `Engine::__construct`:
```rust
impl ReadSight {
pub fn new(language: &str) -> Result<Self, Error>; // embedded data
pub fn with_config(language: &str, config: Config) -> Result<Self, Error>;
// static
pub fn supported_languages(config: Option<&Config>) -> Vec<String>; // sorted codes
// accessors
pub fn language(&self) -> &Language;
pub fn supported_formulas(&self) -> Vec<String>; // registry.list_for_language
// text / syllable
pub fn split_word(&self, w:&str) -> Vec<String>;
pub fn split_syllables(&self, w:&str) -> Vec<String>;
pub fn syllable_count(&self, w:&str) -> i64;
pub fn word_count(&self, t:&str) -> i64;
pub fn sentence_count(&self, t:&str) -> i64;
pub fn letter_count(&self, t:&str) -> i64;
pub fn total_syllables(&self, t:&str) -> i64;
pub fn average_syllables_per_word(&self, t:&str) -> f64;
pub fn average_words_per_sentence(&self, t:&str) -> f64;
pub fn polysyllable_count(&self, t:&str, count_proper_nouns: bool) -> i64;
pub fn words_with_more_than_n_syllables(&self, t:&str, n:i64, count_proper_nouns:bool) -> i64;
pub fn histogram_syllables(&self, t:&str) -> BTreeMap<i64,i64>;
pub fn analyze(&self, t:&str) -> Result<TextStatistics, Error>;
pub fn add_hyphenations(&mut self, map: HashMap<String,String>);
// formulas
pub fn score(&self, formula:&str, text:&str) -> Result<FormulaResult, Error>;
// convenience wrappers, each == self.score("<name>", text):
pub fn flesch_reading_ease / flesch_kincaid_grade_level / gunning_fog /
smog_index("smog") / coleman_liau / automated_readability_index("ari") / lix /
gulpease / fernandez_huerta / szigriszt_pazos / gutierrez_polini / crawford /
fog_pl("fog_pl") / dale_chall / spache / osman(&self, text:&str) -> Result<FormulaResult,Error>;
pub fn wiener_sachtextformel(&self, text:&str, variant: i32 /*default 1*/) -> Result<FormulaResult,Error>;
}
```
Construction steps (`Engine::__construct`):
1. Resolve `Config` (defaults ⇒ embedded data).
2. `language = JsonLanguageRepository::find(language)`.
3. `hyphenator = load_hyphenator(language)` — from cache if present & version
matches, else parse `.tex` via `TexSource` (and populate cache if enabled).
4. `syllable_counter = load_syllable_counter()` (§6.4).
5. `text_analyzer = TextAnalyzer::new(hyphenator, syllable_counter, TextSplitter::new(language), language)`.
6. `formula_registry = registry_factory()`.
`score(formula, text)` = `formula_registry.calculate(formula, &language, &self.analyze(text)?)`.
For `wiener_sachtextformel` route to `calculate_variant`.
> ⚠️ `analyze` errors on **empty text** (`EmptyText`). All formula convenience
> methods therefore return `Result`. Preserve that (Python raises; PHP throws).
---
## 11. Idiomatic Rust vs. literal port
Follow these substitutions faithfully; they change **behaviour**, not just style:
| `mb_strlen($s)` | `s.chars().count()` | code points, not bytes/graphemes |
| `mb_substr($s,$i,$n)` | `s.chars().skip(i).take(n).collect::<String>()` | code points |
| `mb_strtolower` / `mb_strtoupper` | `to_lowercase` / `to_uppercase` | Unicode |
| `preg_match('/p/u', s)` (bool) | `Regex::new("p")?.is_match(s)` | returns 0/1 in PHP → bool |
| `preg_match_all('/p/u', s)` (count) | `re.find_iter(s).count()` | |
| `mb_split(pat, s)` | `re.split(s).filter(!empty)` | filter empties |
| `preg_split('/(?<!^)(?!$)/u', s)` | `s.chars()` | split into chars |
| `is_numeric($ch)` (1 char) | `ch.is_ascii_digit()` | tokens use ASCII digits |
| `round($x, n)` | `round_to(x, n)` (§9.4) | half-away-from-zero |
| `json_decode(..., true)` | `serde_json::from_str::<Value>` / typed | |
| `array<string,int>` insertion order | `IndexMap` if order matters | `problemWords`, prefixes/suffixes iterate in file order; use `serde_json::Map` (preserves insertion order with the `preserve_order` feature) or `IndexMap` |
> **Ordering caveat:** PHP associative arrays and Python dicts preserve insertion
> order, and the heuristic counter iterates `prefixes`/`suffixes` in that order.
> Enable `serde_json`'s `preserve_order` feature (or use `indexmap`) so
> `syllableHeuristics` maps keep JSON order. For the current data this rarely
> changes results, but keep parity.
The `regex` crate does not support look-around, but **none of the language
patterns need it** (they are character classes like `[^\p{L}'’-]+`, `[.!?…]+`,
`[А-Яа-яЁёҐ-ӿЀ-ӿ]`). If a pattern ever fails to compile, compare against the raw
JSON and the Python port's handling before altering it.
---
## 12. Unicode / regex parity checklist
- Enable `regex` default features (Unicode on). `\p{L}` must match the same set
PHP/PCRE matches for the test corpus.
- Word/sentence/letter regexes come **verbatim** from language JSON — compile the
string body as-is (PHP adds `/…/u` delimiters; you just enable Unicode).
- Lowercasing a word before hyphenation must be locale-independent (both PHP
`mb_strtolower` and Rust `to_lowercase` are; do not use `to_ascii_lowercase`).
- `.tex` files are UTF-8. Read as UTF-8; iterate the parser by **byte offset** for
the `%`, `\`, `{`, `}` scanning but capture tokens as full Unicode `\S+`.
---
## 13. Verification strategy (make parity provable)
Correctness = **matching the reference outputs**, not "looks right". Build a
golden-vector harness.
### 13.1 Generate golden vectors from the references
Write a tiny script against **PHP** (canonical) and/or **Python** (`pip install
readsight`) that dumps JSON for a battery of inputs, then commit the JSON under
`tests/golden/`.
Python generator sketch (fast to run):
```python
import json
from readsight import ReadSight
CASES = {
"en-us": ["hyphenation","banana","beautiful","communication","extraordinary","strengths"],
"ru": ["дыхание","беззвучная","молоко","красивый","привет"],
"de-1996": ["Verständlichkeit","Schreiben","Deutsch"],
# …cover every language you can, plus paragraphs for formulas
}
out = {}
for lang, words in CASES.items():
rs = ReadSight(lang)
out[lang] = {
"syllable_count": {w: rs.syllable_count(w) for w in words},
"split_word": {w: rs.split_word(w) for w in words},
"split_syllables":{w: rs.split_syllables(w) for w in words},
}
json.dump(out, open("tests/golden/syllable.json","w"), ensure_ascii=False, indent=2)
```
Do the analogous dump for `analyze(text)` and every `score(formula, text)` result
(score, grade_level, interpretation) over a set of representative paragraphs per
language. Cross-check PHP vs Python first — they must already agree.
### 13.2 Port the reference test suites
The PHP tests (`tests/`, ~260 assertions) and Python tests (`tests/`, 133) encode
the intended behaviour. Re-express the important ones as Rust tests:
- **Hyphenation**: `LiangHyphenatorTest`, `PatternTest`, `PatternsCollectionTest`.
- **Syllable**: `HeuristicSyllableCounterTest` (incl. `vowelMode` cluster vs
individual), `CompositeSyllableCounterTest`, `SyllableConsistencyTest`
(invariants: `split_word` reconstructs the original word;
`0 < count <= char_len`; determinism across repeated calls).
- **Text**: `TextSplitterTest`.
- **Formula**: `UniversalFormulaTest`, `LanguageSpecificFormulaTest`,
`GradeLevelInterpretationTest`, `TextStatisticsHelperTest`.
- **Integration/Multilingual**: load every language; sample words yield `count > 0`;
specific vectors (`en-us banana → 3`, `ru молоко → 3`, `ru дыхание → 4`).
### 13.3 Float comparison
Compare `score`/`grade_level` with an absolute tolerance of **1e-9** after your
`round_to`. If a value diverges by more than the last displayed decimal, you have
a real bug (usually rounding order or an ASL/ASW division difference) — fix the
port, don't loosen the tolerance.
### 13.4 Full-language smoke test
Iterate all 86 codes from `supported_languages`, construct an engine, run
`analyze` + every `supported_formulas` on a short sample; assert no panics and
finite numbers.
---
## 14. Recommended build order for the AI (dependency-first)
Implement and test each layer before moving on. Do **not** write all 54 files then
test — you will drown in compounding errors.
1. **Scaffold**: crate, `Config`, `DataSource::Embedded`, `Error`. Copy `data/`.
2. **Language**: `Script`, `LanguageCode`, `Language` (serde), repository.
Test: load all 86 JSON; `supported_languages()` sorted; spot-check fields.
3. **Hyphenation**: `Pattern`, `PatternsCollection`, exceptions, `TexSource`
parser, `LiangHyphenator`. **Gate on golden `split_word` vectors** before
proceeding — everything downstream depends on this.
4. **Syllable**: trait, `Tex`, `Heuristic` (with `vowelMode`), `Composite`; engine
wiring. Gate on golden `syllable_count`/`split_syllables` (esp. `en-us` composite
and `ru/uk/be/bg` individual-mode vectors, incl. `дыхание → 4`).
5. **Text**: `TextSplitter`, `TextStatistics`, `TextAnalyzer`. Gate on golden
`analyze` output.
6. **Formula**: helpers (`round_to`, `GradeLevelInterpretation`,
`TextStatisticsHelper`), then the 5 universal, then the 12 language-specific,
then registry. Gate each on golden `score` vectors.
7. **Facade** `ReadSight` + convenience methods. Gate on integration/multilingual.
8. **Optional**: JSON pattern cache, `build.rs` precompilation, a `demo` binary
mirroring `examples/demo.php`.
At each gate: `cargo test`, `cargo clippy -- -D warnings`, `cargo fmt --check`.
---
## 15. Pitfalls checklist (the things that will bite you)
- [ ] **Rounding**: `f64::round()` is half-away-from-zero — matches PHP. Use
`round_to`; never truncate; round *before* clamping `grade_level`.
- [ ] **Interpretation uses raw score** in every formula **except Szigriszt-Pazos**
(rounded). WSTF `grade_level` clamps the **raw** score.
- [ ] **`preg_match` returns 0/1** — heuristic subtract/add patterns adjust count by
at most ±1 each (`is_match`, not occurrence count).
- [ ] **Composite = heuristic when heuristic has rules** (e.g. `en-us`). TeX is only
reached via `split_word`.
- [ ] **`vowelMode`**: `individual` counts each vowel char; `cluster` counts vowel
runs. Default `cluster`. `ru/uk/be/bg` ship `individual`.
- [ ] **Supported-formula list** comes from `Formula::supported_languages()`, **not**
the language JSON `formulas` block. JSON only supplies coefficients + LIX
threshold.
- [ ] **WSTF `iw`** ignores the literal `6` and uses `long_word_count` computed with
the language's LIX threshold in `analyze`.
- [ ] **`analyze` throws on empty text**; `polysyllable_count` = words with **>2**
syllables; histogram excludes `0`-syllable words; keys ascending.
- [ ] **Char vs byte indexing** everywhere in hyphenation/syllable code.
- [ ] **Registration order** of formulas affects `supported_formulas()` ordering.
- [ ] **`split_by_patterns` boundary math** (`.word.`, `i-1` mapping, two trailing
loops) — port verbatim; verify with golden vectors, don't refactor first.
- [ ] **Map insertion order** for `problemWords`/prefixes/suffixes (`preserve_order`).
- [ ] **Data is the contract**: copy `data/` verbatim; do not regenerate.
- [ ] **`wiener_sachtextformel` result name** is `wiener_sachtextformel_{variant}`.
- [ ] **Language code normalization** (trim + Unicode lowercase) before every lookup.
---
## 16. Definition of done
- All 86 languages load; `supported_languages()` returns 86 sorted codes.
- Golden vectors for syllables, `split_word`, `analyze`, and all 17 formulas match
the PHP/Python references within 1e-9 (numbers) / exact (strings, splits).
- Ported PHP + Python test cases pass.
- `cargo test`, `cargo clippy -- -D warnings`, `cargo fmt --check` all clean.
- Public API mirrors the facade in §10; docs link back to the two reference repos.
---
*Generated as the porting spec for `readsight` (Rust). Canonical behaviour lives in
the PHP repo (https://github.com/MADEVAL/ReadSight); the Python port
(https://github.com/MADEVAL/ReadSightPy) is the secondary reference. When in doubt,
generate a golden vector from both and match it.*