# parla-clean
Deterministic post-processing for Brazilian Portuguese speech transcription. Removes discourse fillers ("tipo", "né", "sabe"), fixes phonetic misspellings of vocabulary terms ("sematlman" → "Sam Altman"), deduplicates ASR repetitions, and normalizes punctuation and capitalization — all locally, no LLM, no network.
Built for dictation pipelines (Whisper-style ASR): the output is the text you can type, not a summary. Pure Rust; the only runtime dependency is `log`.
## Usage
```rust
use parla_clean::clean_text;
let vocab = vec!["GitHub".to_string(), "Sam Altman".to_string()];
let clean = clean_text("tipo, o sematlman usa github né?", &vocab);
assert_eq!(clean, "O Sam Altman usa GitHub?");
```
## What it does
Four passes, in fixed order (order matters: vocabulary is fixed before anything else, so "o sematlman" becomes "O Sam Altman" with correct sentence case).
1. **Vocabulary variant correction** — "sematlman" → "Sam Altman", "git hub" → "GitHub"; case and spacing normalization. Only applies to terms present in the user vocabulary.
2. **Filler removal** — 10 pt-BR discourse markers ("tipo", "né", "assim", "então", "sabe", "tá", ...) with context guards: "que tipo de", "assim como", "estudei, então passei" are kept.
3. **ASR repetition deduplication** — function-word doubles ("eu eu"), any-word triples ("OpenAI OpenAI OpenAI"), number doubles ("14h30 14h30").
4. **Normalization** — spacing, punctuation, sentence capitalization, final period when a removed filler ended the sentence.
## What it does NOT do
- Does not rewrite grammar or content: auto-corrections, false starts, numbers, names, URLs and emails pass through untouched (URLs, emails and pt-BR numbers are atomic tokens).
- Does not translate. Does not summarize.
- Does not call an LLM or any service. No model, no API key.
- Does not touch repetition separated by punctuation ("não, não, não quero" is kept — it may be emphasis).
- Keeps 2x content-word repetition ("muito muito bom").
## Design decisions
Precision over recall is deliberate, and enforced everywhere (Ferguson et al., 2015). "When in doubt, don't touch" is implemented as context guards, positional requirements and a fixpoint rule.
Filler removal is limited to discourse markers — the category with the lowest semantic load in the disfluency literature (Uh-Mazing, 2026). Auto-corrections and false starts, the highest-load category, are never touched.
Rules are data, not code: the 10 markers are table entries (`FillerRule`), so tuning a rule never touches logic. Case comparison is full Unicode (`to_lowercase`, never ASCII-case); the tokenizer is a single lossless pass where URLs, emails and pt-BR numbers ("1.000,50", "14h30") are atomic.
Terminal punctuation follows the pragmatic function of the tag: "né?" keeps the question mark (a real tag question), while "sabe?"/"tá?" are phatic checks on a declarative — the sentence gets a period. This distinction comes from the discourse-marker literature (C-ORAL-BRASIL; UFS).
The pipeline is idempotent by construction: filler removal runs to a bounded fixpoint.
## Invariants
- **I1** — Nothing is invented: every output word is a contiguous substring of an input word or a vocabulary term.
- **I2** — Terminal punctuation survives: if the input ends in `.`/`!`/`?` and the last word is not a declarative tag, the output ends in terminal punctuation.
- **I3** — Totality and fidelity: any input produces output without panicking; tokenize → render is byte-exact.
- **I4** — Idempotence: `clean(clean(x)) == clean(x)`.
Verified by property tests with a seeded PRNG — no fuzzing dependencies.
## Evaluation
Golden corpus with word-level precision/recall/F1, aggregated and per category (filler, guarda, variante, dedupe, normaliza, link, punct, unicode, numero). The corpus is the CI gate: `guarda` must keep FP = 0 (the "when in doubt, don't touch" rule is non-negotiable), and every category must stay above the versioned baseline in `tests/golden/baseline.json`.
```bash
cargo test --lib
```
## Performance
~0.47 µs/char, measured in release on an i5-10400F (Windows, 2026-08-14). A typical dictation sentence (~150 chars) cleans in ~70 µs. The design is O(n) for typical inputs, with a prefiltered matcher for vocabulary variants.
## Known limitations
- Domain detection accepts TLDs of 2–20 ASCII letters (`.technology`, `.consulting` work; the rejection guards — no sentence punctuation, ASCII-alphabetic TLD — still filter junk).
- Characters with multi-char uppercase (ẞ/ß/ǰ) are covered by unit tests of the fold, not by the property fuzzer: their casefolding breaks the literal substring invariant.
- pt-BR is the only locale. The rules table makes adaptation to other languages possible, but it is not validated.
## References
1. Uh-Mazing: "The Role of Disfluencies in Speech Translation" — arXiv:2608.02138 (2026). Disfluency taxonomy and semantic load by category.
2. Walker & Liebling (Google Research, 2022): "Identifying Disfluencies in Natural Speech" — BERT detection, scarcity of labeled data outside English.
3. Lima & Campelo (2024): "Disfluency Detection and Removal in Speech Transcriptions via Large Language Models", STIL 2024 (UFCG) — pt-BR validation of the prompt approach (phase 2). DOI: 10.5753/stil.2024.245417.
4. Ferguson, Durrett & Klein (2015): word-level P/R/F1 metrics and disfluency detection — NAACL.
5. Snover, Dorr & Schwartz (2004): lexically-driven disfluency detection — HLT-NAACL (the rules-as-data lineage).
6. "Marcadores discursivos não são vícios de linguagem" — Interdisciplinar (UFS): real functions of "né", "tipo", "assim" in pt-BR (basis of the context guards).
7. C-ORAL-BRASIL (UFMG) and NURC — spontaneous speech corpora of Brazilian Portuguese.
## License
Dual-licensed under MIT or Apache-2.0, at your option.