pub const DENOISE_ARTIFACT_RELPATH: &str = "denoise/fastenhancer-s-48k.safetensors";Expand description
Single-channel speech enhancement: MMSE-LSA gains, decision-directed SNR, OM-LSA presence weighting, over a noise floor initialised offline and then tracked recursively.
Enrollment noise is not cosmetic: it is encoded into the x-vector and then reproduced in every utterance the cloned voice speaks (measured — cleaning a real 53 s reference dropped the synthesized output’s pause floor by 19.5 dB). This removes the stationary part of it.
Why this and not spectral subtraction. Subtracting an estimated noise magnitude minimises squared error in the spectrum, which is the wrong objective for something a listener judges and a speaker encoder reads: it punches holes in low-SNR bins, producing musical noise, and it removes real signal along with the noise (measured here at 33% of peak burst energy before this replaced it). Three pieces fix that, and they compose:
- MMSE-LSA (Ephraim & Malah 1985) estimates the log amplitude, matching how loudness is
perceived, and yields the gain
ξ/(1+ξ) · exp(½·E₁(ν)). The exponential-integral term is what makes it gentle where the a posteriori SNR is uncertain instead of gating hard. - Decision-directed a priori SNR (same paper) smooths ξ across frames using the previous frame’s own estimate. This is the specific mechanism that suppresses musical noise: isolated noise peaks never get a confident ξ, so they are never sharply attenuated or passed.
- OM-LSA (Cohen & Berdugo 2001) blends that gain toward the floor by the speech-presence
probability,
G = G_LSA^p · G_min^(1−p), so bins that are probably noise settle to a constant bed rather than being tracked.
Why the noise floor is initialised offline rather than by minimum statistics. IMCRA’s online minimum tracking exists because a streaming denoiser cannot see the future. Enrollment can: the file is already on disk, so a low quantile of each bin over the whole recording is a better starting floor than any causal estimator’s, with none of the convergence transient. An earlier revision here did run minimum statistics, and it is instructive why that was removed rather than debugged: seeded from frame 0 of a reference that opens on speech, the refined minimum locked above the speech level, which drove the presence probability to zero, which unfroze the noise update, which let the noise estimate absorb the voice — a positive feedback that left ~10% of every burst after the first. Speech presence here instead comes from the likelihood ratio in ξ and ν, which is self-correcting: it cannot conclude “no speech” about a bin whose own a priori SNR is high.
Nonstationary noise is still tracked, by the recursive average that the presence probability gates — the offline quantile only sets where that average starts.
This is the state of the art among methods that need no trained weights. Neural enhancers (DeepFilterNet and friends) do beat it, at the cost of shipping and running another model — which is not a trade this CLI should make silently for an enrollment preprocessing step.
Deliberately conservative even so: the speaker encoder reads breath, sibilance, and room as
part of identity, so this stays opt-in (--denoise) per the project’s doctrine that a lever
which can damage speaker identity ships behind a named switch until blind listening clears it.
Phase is preserved untouched; only per-bin magnitude is scaled.
Where ftts pull lands the neural denoiser, relative to the model root.