Expand description
NVIDIA RE-USE (nvidia/RE-USE, universal speech enhancement: SEMamba,
30 bidirectional time-frequency Mamba blocks between a dense
convolutional encoder and two decoders) on an AMD XDNA NPU: noisy,
reverberant, clipped, band-limited or codec-damaged speech at 8 to 48
kHz in, enhanced speech at the same rate out.
The bundle iron/applications/reuse/export_reuse.py writes holds every
compiled IRON kernel and the weights (the NPU ones pre-packed); this
crate replays the forward the Python app runs (reuse_common.py /
reuse_npu.py):
| NPU | host (here) | |
|---|---|---|
| STFT (n_fft 320 at 8 kHz, scaled with the rate; hop 1/8 of it) | stft.rs (and its inverse) | |
| encoder (dense block of 4 dilated 3 x 3 convs, a strided conv) | every conv as an flm.GEMM | the 1 x 1 input conv, the map layouts, the dilated taps’ sums, InstanceNorm + PReLU |
| 30 x (time Mamba, frequency Mamba), both directions | in_proj, causal conv1d + x_proj, the selective scan, the output projection (the MambaScan operator’s phases) | token layouts, both directions summed, LayerNorm, residual |
| decoders (dense block, two sub-pixel upsampling convs) | every conv as an flm.GEMM | layouts, InstanceNorm + PReLU, the 1 x 1 heads, atan2 |
Reuse::enhance takes any length: up to the bundle’s longest Mamba
grid (t_max, ~5 s) in one pass, as the checkpoint’s inference.py;
longer audio in 5 s chunks at a 2.5 s hop, overlap-added under a Hann
window, as its inference_chunk.py.
Modules§
- media
- Compressed audio in (feature
decode): MP3, AAC in MP4 / M4A, FLAC and WAV through symphonia, and a resampler for rates the bundle has no kernels for. - model
- The forward the Python app runs (
reuse_common.py), host side here and every convolution, projection and Mamba scan on the NPU: - npu
- The NPU side: the bundle’s kernels (kernels naming the same xclbin
share its hardware context: one an
flm.GEMMshape group and one a MambaScan phase, 7 of NPU2’s 16, shared by every process), flat bf16 device buffers, and the dispatches. - stft
- The model’s STFT and its inverse, as
torch.stft/torch.istftcompute them in the checkpoint’s inference.py: a periodic Hann window of the FFT size, centred frames (the signal reflect-padded by n_fft / 2), one-sided spectra; the magnitude compressed by log1p. Transforms in f64. - wav
- WAV files: PCM 16 / 24 / 32-bit and IEEE float 32 in (as torchaudio scales them: int16 / 32768, …), 16-bit PCM or float 32 out.
Structs§
- Config
- The model’s constants and the bundle’s layout (the manifest’s params).
- Reuse
- Timing
- Wall time per stage, in first-seen order; NPU dispatch time is kept
under
npu:<kernel>.
Enums§
Constants§
Functions§
- snr
- SNR of
xagainstreference, in dB.