parla-clean

Deterministic post-processing for Brazilian Portuguese speech transcription. Removes discourse fillers ("tipo", "né", "sabe"), fixes phonetic misspellings of vocabulary terms ("sematlman" → "Sam Altman"), deduplicates ASR repetitions, and normalizes punctuation and capitalization — all locally, no LLM, no network.

Built for dictation pipelines (Whisper-style ASR): the output is the text you can type, not a summary. Pure Rust; the only runtime dependency is log.

Usage

use parla_clean::clean_text;

let vocab = vec!["GitHub".to_string(), "Sam Altman".to_string()];
let clean = clean_text("tipo, o sematlman usa github né?", &vocab);
assert_eq!(clean, "O Sam Altman usa GitHub?");

What it does

Four passes, in fixed order (order matters: vocabulary is fixed before anything else, so "o sematlman" becomes "O Sam Altman" with correct sentence case).

1. Vocabulary variant correction — "sematlman" → "Sam Altman", "git hub" → "GitHub"; case and spacing normalization. Only applies to terms present in the user vocabulary.
2. Filler removal — 10 pt-BR discourse markers ("tipo", "né", "assim", "então", "sabe", "tá", ...) with context guards: "que tipo de", "assim como", "estudei, então passei" are kept.
3. ASR repetition deduplication — function-word doubles ("eu eu"), any-word triples ("OpenAI OpenAI OpenAI"), number doubles ("14h30 14h30").
4. Normalization — spacing, punctuation, sentence capitalization, final period when a removed filler ended the sentence.

What it does NOT do

• Does not rewrite grammar or content: auto-corrections, false starts, numbers, names, URLs and emails pass through untouched (URLs, emails and pt-BR numbers are atomic tokens).
• Does not translate. Does not summarize.
• Does not call an LLM or any service. No model, no API key.
• Does not touch repetition separated by punctuation ("não, não, não quero" is kept — it may be emphasis).
• Keeps 2x content-word repetition ("muito muito bom").

Design decisions

Precision over recall is deliberate, and enforced everywhere (Ferguson et al., 2015). "When in doubt, don't touch" is implemented as context guards, positional requirements and a fixpoint rule.

Filler removal is limited to discourse markers — the category with the lowest semantic load in the disfluency literature (Uh-Mazing, 2026). Auto-corrections and false starts, the highest-load category, are never touched.

Rules are data, not code: the 10 markers are table entries (FillerRule), so tuning a rule never touches logic. Case comparison is full Unicode (to_lowercase, never ASCII-case); the tokenizer is a single lossless pass where URLs, emails and pt-BR numbers ("1.000,50", "14h30") are atomic.

Terminal punctuation follows the pragmatic function of the tag: "né?" keeps the question mark (a real tag question), while "sabe?"/"tá?" are phatic checks on a declarative — the sentence gets a period. This distinction comes from the discourse-marker literature (C-ORAL-BRASIL; UFS).

The pipeline is idempotent by construction: filler removal runs to a bounded fixpoint.

Invariants

• I1 — Nothing is invented: every output word is a contiguous substring of an input word or a vocabulary term.
• I2 — Terminal punctuation survives: if the input ends in ./!/? and the last word is not a declarative tag, the output ends in terminal punctuation.
• I3 — Totality and fidelity: any input produces output without panicking; tokenize → render is byte-exact.
• I4 — Idempotence: clean(clean(x)) == clean(x).

Verified by property tests with a seeded PRNG — no fuzzing dependencies.

Evaluation

Golden corpus with word-level precision/recall/F1, aggregated and per category (filler, guarda, variante, dedupe, normaliza, link, punct, unicode, numero). The corpus is the CI gate: guarda must keep FP = 0 (the "when in doubt, don't touch" rule is non-negotiable), and every category must stay above the versioned baseline in tests/golden/baseline.json.

cargo test --lib

Performance

~0.47 µs/char, measured in release on an i5-10400F (Windows, 2026-08-14). A typical dictation sentence (~150 chars) cleans in ~70 µs. The design is O(n) for typical inputs, with a prefiltered matcher for vocabulary variants.

Known limitations

• Domain detection accepts TLDs of 2–20 ASCII letters (.technology, .consulting work; the rejection guards — no sentence punctuation, ASCII-alphabetic TLD — still filter junk).
• Characters with multi-char uppercase (ẞ/ß/ǰ) are covered by unit tests of the fold, not by the property fuzzer: their casefolding breaks the literal substring invariant.
• pt-BR is the only locale. The rules table makes adaptation to other languages possible, but it is not validated.

References

1. Uh-Mazing: "The Role of Disfluencies in Speech Translation" — arXiv:2608.02138 (2026). Disfluency taxonomy and semantic load by category.
2. Walker & Liebling (Google Research, 2022): "Identifying Disfluencies in Natural Speech" — BERT detection, scarcity of labeled data outside English.
3. Lima & Campelo (2024): "Disfluency Detection and Removal in Speech Transcriptions via Large Language Models", STIL 2024 (UFCG) — pt-BR validation of the prompt approach (phase 2). DOI: 10.5753/stil.2024.245417.
4. Ferguson, Durrett & Klein (2015): word-level P/R/F1 metrics and disfluency detection — NAACL.
5. Snover, Dorr & Schwartz (2004): lexically-driven disfluency detection — HLT-NAACL (the rules-as-data lineage).
6. "Marcadores discursivos não são vícios de linguagem" — Interdisciplinar (UFS): real functions of "né", "tipo", "assim" in pt-BR (basis of the context guards).
7. C-ORAL-BRASIL (UFMG) and NURC — spontaneous speech corpora of Brazilian Portuguese.

License

Dual-licensed under MIT or Apache-2.0, at your option.