# audiofp - Internal Architecture & Algorithm Reference
> This document is the **internal** engineering reference for `audiofp`.
> It explains how audio moves through the DSP pipeline, how each
> fingerprinter packs bits, how streaming maintains offline parity, and how
> optional neural and watermark components are organized. For the user-facing
> SDK guide, see [USAGE.md](USAGE.md).
---
## Table of Contents
1. [Project Overview](#project-overview)
2. [Architecture](#architecture)
3. [Core Algorithms](#core-algorithms)
- [Wang](#1-wang-landmark-pairs)
- [Panako](#2-panako-triplets)
- [Haitsma](#3-haitsma-kalker-band-power-hash)
- [Streaming](#4-streaming-extraction)
- [Neural Embedder](#5-neural-embedder)
- [Watermark Detector](#6-watermark-detector)
4. [DSP Internals](#dsp-internals)
5. [Data Structures & Memory Layouts](#data-structures--memory-layouts)
6. [I/O and Resampling](#io-and-resampling)
7. [Edge Cases & Invariants](#edge-cases--invariants)
8. [Module Map](#module-map)
---
## Project Overview
**audiofp** is a Rust SDK (current version pinned in `Cargo.toml`, edition 2024, MSRV 1.85) for extracting
deterministic fingerprints from mono PCM audio. It provides three classical
fingerprinters, streaming variants with offline parity, reusable DSP
primitives, optional ONNX log-mel embeddings, and optional AudioSeal-compatible
watermark detection.
It serves:
- Music identification and Shazam-style matching
- Audio deduplication across codec transcodes
- Rights enforcement and catalog matching
- Compact dense audio IDs
- Embedding-based similarity search with BYO ONNX models
- Watermark verification for generated or distributed audio
### Design Pillars
| **Determinism** | Same input samples + same config -> same hashes. |
| **Stable hash bytes** | `WangHash`, `PanakoHash`, and `Peak` are `repr(C)` and `bytemuck::Pod`. |
| **Streaming parity** | Streaming output matches offline extraction under arbitrary chunking. |
| **DSP reuse** | STFT, peak picking, mel filters, windows, and resampling are public primitives. |
| **Feature isolation** | Classical DSP is designed for `no_std + alloc`; file I/O, neural, and watermark are feature-gated, and the current FFT dependency still makes the no_std path host-only. |
| **Hot-path reuse** | FFT plans, peak-picker scratch, and streaming buffers are held on the extractor. |
---
## Architecture
### The Four-Stage Pipeline
Classical fingerprints flow through four stages:
```text
audio file or mono PCM
|
v
+-------------------------------------------------+
+-------------------------------------------------+
|
v
+-------------------------------------------------+
+-------------------------------------------------+
|
v
+-------------------------------------------------+
+-------------------------------------------------+
|
v
+-------------------------------------------------+
+-------------------------------------------------+
```
The crate accepts borrowed mono `f32` samples through `AudioBuffer`. File
decoding is optional and lives behind the default `std` feature.
### Classical Pipeline Diagram
```mermaid
flowchart TD
A["Input file or mono PCM"] --> B{"std decoder?"}
B -->|yes| C["decode_to_mono / decode_to_mono_at"]
B -->|no| D["AudioBuffer { samples, rate }"]
C --> D
D --> E{"algorithm"}
E -->|Wang| W1["8 kHz, n_fft=1024, hop=128"]
W1 --> W2["power STFT"]
W2 --> W3["10*log10(power floor)"]
W3 --> W4["31x31 peak picker, 30 peaks/s"]
W4 --> W5["anchor-target pairs"]
W5 --> W6["WangHash { hash, t_anchor }"]
E -->|Panako| P1["8 kHz, n_fft=1024, hop=128"]
P1 --> P2["power STFT"]
P2 --> P3["10*log10(power floor)"]
P3 --> P4["31x31 peak picker, 30 peaks/s"]
P4 --> P5["anchor-target-target triplets"]
P5 --> P6["PanakoHash { hash, t_anchor, t_b, t_c }"]
E -->|Haitsma| H1["5 kHz, n_fft=2048, hop=64"]
H1 --> H2["power STFT"]
H2 --> H3["33 log-spaced bands, 300-2000 Hz"]
H3 --> H4["32 band-difference delta bits/frame"]
H4 --> H5["u32 frame hashes"]
classDef input fill:#dbeafe,stroke:#3b82f6,color:#1e3a8a,stroke-width:1px;
classDef router fill:#f3f4f6,stroke:#9ca3af,color:#111827,stroke-width:1px;
classDef wang fill:#fef3c7,stroke:#f59e0b,color:#92400e,stroke-width:1px;
classDef panako fill:#ede9fe,stroke:#8b5cf6,color:#5b21b6,stroke-width:1px;
classDef haitsma fill:#ccfbf1,stroke:#14b8a6,color:#115e59,stroke-width:1px;
classDef output fill:#e5e7eb,stroke:#6b7280,color:#111827,stroke-width:1px;
class A,B,C,D input;
class E router;
class W1,W2,W3,W4,W5 wang;
class W6 output;
class P1,P2,P3,P4,P5 panako;
class P6 output;
class H1,H2,H3,H4 haitsma;
class H5 output;
```
### Runtime Sequence (Wang offline path)
```mermaid
sequenceDiagram
participant User
participant FP as Wang
participant STFT as ShortTimeFFT
participant Picker as PeakPicker
participant Pack as build_hashes
User->>FP: extract(AudioBuffer)
FP->>FP: validate sample rate and minimum length
FP->>STFT: power_flat(samples)
STFT-->>FP: power spectrogram, n_frames, n_bins
FP->>FP: log_spec[i] = 10*log10(max(power, floor))
FP->>Picker: pick(log_spec, frames_per_sec)
Picker-->>FP: peaks sorted by (t_frame, f_bin)
FP->>Pack: build anchor-target hashes
Pack-->>FP: Vec<WangHash>
FP->>FP: sort by (t_anchor, hash)
FP-->>User: WangFingerprint
```
### Feature Surface
| `std` | Yes | `audiofp::io` Symphonia decoder |
| `neural` | No | ONNX log-mel embedder via Tract |
| `watermark` | No | AudioSeal-style detector via Tract |
| `mimalloc` | No | `mimalloc::MiMalloc` global allocator |
With default features disabled, the crate is `no_std + alloc` in API shape for
DSP and classical fingerprinters, but the current FFT dependency chain keeps
the no_std path host-only today.
---
## Core Algorithms
### 1. Wang Landmark Pairs
#### Theory
Wang is a landmark-pair algorithm inspired by the Shazam paper. It finds stable
spectral peaks, treats each peak as an anchor, and pairs it with strong later
peaks in a bounded target zone.
#### Fixed Front-End
```text
sample rate: 8,000 Hz
n_fft: 1024
hop: 128
frame rate: 62.5 fps
window: Hann
center framing: false
frequency bins: 513
peak neighborhood: 15 frames x 15 bins each side, full 31x31 window
```
#### Default Config
```text
fan_out: 10
target_zone_t: 63 frames
target_zone_f: 64 FFT bins
peaks_per_sec: 30
min_anchor_mag_db: -50.0
```
#### Algorithm Walkthrough
```text
INPUT: mono f32 samples at 8 kHz, at least 16000 samples
Step 1 - STFT:
power_flat(samples)
n_bins = 513
frames_per_sec = 8000 / 128 = 62.5
Step 2 - Log magnitude:
log_spec[i] = 10 * log10(max(power[i], 1e-12))
Step 3 - Peak picking:
rolling max over 31x31 neighborhood
candidate if value > -50 dB and value >= local rolling max
keep top 30 peaks per one-second bucket
sort survivors by (t_frame, f_bin)
Step 4 - Landmark pairs:
for each anchor peak:
scan later peaks while 1 <= dt <= target_zone_t
require abs(df) <= target_zone_f
keep strongest fan_out targets by magnitude
sort kept targets by magnitude desc, then position asc
Step 5 - Pack:
f_a_q = floor(anchor_bin * 512 / 513)
f_b_q = floor(target_bin * 512 / 513)
dt = target_frame - anchor_frame
```
#### Hash Layout
```text
WangHash::hash (32 bits)
[31..23] f_a_q 9 bits, anchor frequency bucket
[22..14] f_b_q 9 bits, target frequency bucket
[13.. 0] dt 14 bits, frame delta, minimum 1
WangHash (8 bytes, repr(C), Pod)
├── hash: u32
└── t_anchor: u32
```
#### Packing Formula
```text
hash =
((f_a_q & 0x1ff) << 23)
| ((f_b_q & 0x1ff) << 14)
| (dt & 0x3fff)
```
#### Wang Pipeline Diagram
```mermaid
flowchart TD
A["8 kHz mono PCM"] --> B["STFT 1024/128 Hann"]
B --> C["power spectrogram"]
C --> D["10*log10(power floor)"]
D --> E["rolling max peak picker"]
E --> F["top 30 peaks/s"]
F --> G["for each anchor, keep top fan_out targets"]
G --> H["pack f_a_q | f_b_q | dt"]
H --> I["WangFingerprint"]
classDef input fill:#dbeafe,stroke:#3b82f6,color:#1e3a8a,stroke-width:1px;
classDef wang fill:#fef3c7,stroke:#f59e0b,color:#92400e,stroke-width:1px;
classDef output fill:#e5e7eb,stroke:#6b7280,color:#111827,stroke-width:1px;
class A input;
class B,C,D,E,F,G,H wang;
class I output;
```
#### Key Properties
- **Sparse landmarks**: output size depends on peaks and fan-out.
- **Codec tolerant**: strong spectral peaks tend to survive lossy encoders.
- **Time-local**: `t_anchor` enables offset voting in a matcher.
- **Stable ordering**: hashes sort by `(t_anchor, hash)` before returning.
---
### 2. Panako Triplets
#### Theory
Panako uses the same spectral front-end as Wang but emits triplet hashes. Each
anchor chooses two later target peaks. The packed hash includes a tempo ratio
`beta`, making the signature more tolerant of small time-scale changes.
#### Fixed Front-End
```text
sample rate: 8,000 Hz
n_fft: 1024
hop: 128
frame rate: 62.5 fps
window: Hann
center framing: false
frequency bins: 513
peak neighborhood: 31x31
```
#### Default Config
```text
fan_out: 5
target_zone_t: 96 frames, strict dt < 96
target_zone_f: 96 bins, strict abs(df) < 96
peaks_per_sec: 30
min_anchor_mag_db: -50.0
```
#### Algorithm Walkthrough
```text
INPUT: mono f32 samples at 8 kHz, at least 16000 samples
Step 1 - Shared front-end:
power STFT -> dB log_spec -> peak picker
Step 2 - Collect target cone:
for each anchor:
collect later peaks where:
1 <= dt < target_zone_t
abs(df) < target_zone_f
Step 3 - Enumerate target pairs:
for every ordered pair (b, c) from the target cone, b before c:
score = b.mag + c.mag
keep top fan_out triplets by score
Step 4 - Pack each triplet:
df_ab = clamp(f_b - f_a, -127, 127)
df_bc = clamp(f_c - f_b, -127, 127)
beta = round((t_c - t_b) / (t_c - t_a) * 31), clamped 0..31
```
#### Hash Layout
```text
PanakoHash::hash (32 bits)
[31..30] sign 2 bits, signs of df_ab and df_bc
[29..28] mag_order 2 bits, strongest of a/b/c
[27..23] beta 5 bits, relative timing ratio
[22..15] df_ab 8 bits signed two's-complement
[14.. 7] df_bc 8 bits signed two's-complement
[ 6.. 0] reserved 7 bits, zero
PanakoHash (16 bytes, repr(C), Pod)
├── hash: u32
├── t_anchor: u32
├── t_b: u32
└── t_c: u32
```
#### Packing Formula
```text
hash =
((sign & 0x3) << 30)
| ((mag_order & 0x3) << 28)
| ((beta & 0x1f) << 23)
| ((df_ab_u8 & 0xff) << 15)
| ((df_bc_u8 & 0xff) << 7)
```
#### Panako Pipeline Diagram
```mermaid
flowchart TD
A["8 kHz mono PCM"] --> B["same STFT/log/peak front-end as Wang"]
B --> C["peaks sorted by time/bin"]
C --> D["target cone per anchor"]
D --> E["all (b,c) target pairs"]
E --> F["top fan_out by b.mag + c.mag"]
F --> G["pack sign | mag_order | beta | df_ab | df_bc"]
G --> H["PanakoFingerprint"]
classDef input fill:#dbeafe,stroke:#3b82f6,color:#1e3a8a,stroke-width:1px;
classDef panako fill:#ede9fe,stroke:#8b5cf6,color:#5b21b6,stroke-width:1px;
classDef output fill:#e5e7eb,stroke:#6b7280,color:#111827,stroke-width:1px;
class A input;
class B,C,D,E,F,G panako;
class H output;
```
#### Key Properties
- **Tempo-aware**: `beta` preserves relative target timing under mild tempo shifts.
- **Richer hash**: stores three frame positions for offset and stretch-aware matching.
- **Strict cone bounds**: Panako uses `< target_zone_t` and `< target_zone_f`, not `<=`.
- **Stable ordering**: hashes sort by `(t_anchor, t_b, t_c, hash)`.
---
### 3. Haitsma-Kalker Band-Power Hash
#### Theory
Haitsma-Kalker emits a dense 32-bit frame hash from spectral band energy
differences. It is compact and fast because it does not perform peak picking or
landmark pairing.
#### Fixed Front-End
```text
sample rate: 5,000 Hz
n_fft: 2048
hop: 64
frame rate: 78.125 fps
window: Hann
center framing: false
frequency bands: 33 log-spaced bands
default fmin: 300 Hz
default fmax: 2000 Hz
```
#### Algorithm Walkthrough
```text
INPUT: mono f32 samples at 5 kHz, at least 10000 samples
Step 1 - STFT:
power_flat(samples) -> power[n_frame][bin]
Step 2 - Band energies:
build 34 log-spaced edges from fmin to fmax
map each FFT bin to one of 33 bands or None
E[n][b] = sum power bins assigned to band b
Step 3 - Frame bits:
for every frame n >= 1:
for every band b in 0..32:
lhs = E[n][b] - E[n][b+1]
rhs = E[n-1][b] - E[n-1][b+1]
bit[b] = (lhs - rhs) > 0
Step 4 - Pack:
band 0 -> bit 31
band 31 -> bit 0
```
#### Frame Layout
```text
Haitsma frame (u32)
bit 31 band 0
bit 30 band 1
...
bit 0 band 31
```
The output type is `HaitsmaFingerprint { frames: Vec<u32>, frames_per_sec:
f32 }`. Each frame is one 32-bit hash word.
#### Haitsma Pipeline Diagram
```mermaid
flowchart TD
A["5 kHz mono PCM"] --> B["STFT 2048/64 Hann"]
B --> C["power spectrogram"]
C --> D["sum into 33 log-spaced bands"]
D --> E["compare adjacent-band differences across adjacent frames"]
E --> F["pack 32 sign bits MSB-first"]
F --> G["HaitsmaFingerprint"]
classDef input fill:#dbeafe,stroke:#3b82f6,color:#1e3a8a,stroke-width:1px;
classDef haitsma fill:#ccfbf1,stroke:#14b8a6,color:#115e59,stroke-width:1px;
classDef output fill:#e5e7eb,stroke:#6b7280,color:#111827,stroke-width:1px;
class A input;
class B,C,D,E,F haitsma;
class G output;
```
#### Key Properties
- **Dense**: roughly one 32-bit word per STFT frame from frame 1 onward.
- **Fast**: no 2-D peak picker, no target search.
- **Compact**: about 312 bytes/s at 78.125 fps.
- **Silence invariant**: all-zero input produces all-zero frame hashes; Haitsma still emits one zero-valued hash per frame from frame 1 onward.
---
### 4. Streaming Extraction
#### Goal
Every streaming implementation must emit the same hash sequence or multiset as
the corresponding offline `extract` call for the same total samples, regardless
of how input is chunked.
#### Streaming Wang and Panako
Wang and Panako need future context for peak picking and target zones, so they
defer emission until the relevant information has ripened.
```text
push(samples):
append samples to sample_carry
while sample_carry has a complete STFT frame:
process one frame to power
convert to dB log power
append row to rolling spectrogram window
when frame_idx >= peak_neighborhood:
detect the row whose forward neighborhood is now available
drain consumed hop samples
finalize per-second peak buckets whose frames are fully detected
emit anchors whose whole target zone has been finalized
```
Important state:
```text
sample_carry leftover PCM below one full frame
spec rolling log-power spectrogram window
spec_first_frame absolute frame index of row 0 in spec
n_frames_total absolute frame counter
last_pd_frame last peak-detected frame
bucket_pending candidate peaks grouped by one-second bucket
last_finalized_bucket last bucket after top-N thresholding
pending_anchors anchors waiting for full target-zone lookahead
```
Latency formulas:
```text
StreamingWang:
(target_zone_t + 15 + ceil(62.5)) * 128 / 8000 seconds
StreamingPanako:
(target_zone_t + 15 + ceil(62.5)) * 128 / 8000 seconds
```
Default values:
```text
Wang: (63 + 15 + 63) * 128 / 8000 = 2256 ms
Panako: (96 + 15 + 63) * 128 / 8000 = 2784 ms
```
#### Streaming Haitsma
Haitsma is simpler because each output depends only on the current and previous
frame's band energies.
```text
push(samples):
append samples to sample_carry
while a full 2048-sample frame exists:
process frame power
sum power into 33 bands
if a previous frame exists:
pack_frame_bits(curr, prev)
emit at absolute frame timestamp
prev = curr
advance by hop=64
drain consumed samples
```
Latency:
```text
2048 samples / 5000 Hz = 409 ms
```
#### Streaming Neural
The neural streaming embedder buffers PCM until one complete analysis window is
available, runs the same `embed_window_into` routine as offline extraction, then
drains `hop_samples`.
```text
while sample_carry.len() >= window_samples:
embed first window
timestamp = samples_consumed * 1000 / sample_rate
drain hop_samples
samples_consumed += hop_samples
```
Partial windows do not emit on `flush`; non-centered framing means they cannot
produce a complete embedding.
---
### 5. Neural Embedder
#### Purpose
The `neural` feature provides a generic ONNX embedding path. The user supplies a
model whose first input accepts a log-mel spectrogram shaped
`[1, n_mels, n_frames]` and whose first output is an embedding vector.
#### Default Config
```text
sample_rate: 16000
n_fft: 1024
hop: 320
n_mels: 128
fmin: 0
fmax: sample_rate / 2
mel_scale: Slaney
window: Hann
window_secs: 1.0
hop_secs: 1.0
l2_normalize: true
```
#### Front-End
```text
window_samples = round(window_secs * sample_rate)
hop_samples = round(hop_secs * sample_rate)
n_frames = (window_samples - n_fft) / hop + 1
for each STFT frame:
process_frame_power()
MelFilterBank::log_mel_from_power()
write into tensor position [0, mel, frame]
```
The tensor layout is `[1, n_mels, n_frames]`. The implementation writes
strided mel rows directly into the Tract tensor, avoiding an intermediate
transpose.
#### Model Lifecycle
```text
NeuralEmbedder::new(config):
validate config
load ONNX model
set concrete input shape [1, n_mels, n_frames]
type and optimize model
build runnable plan
run a zero probe to determine embedding_dim
build LogMelFrontend
```
Extraction builds one embedding per analysis window, timestamps it at the
window start, and optionally L2-normalizes it.
#### Output Types
```text
NeuralEmbedding
├── vector: Vec<f32>
└── t_start: TimestampMs
NeuralFingerprint
├── embeddings: Vec<NeuralEmbedding>
├── embedding_dim: usize
└── frames_per_sec: f32 = 1.0 / hop_secs
```
---
### 6. Watermark Detector
#### Purpose
The `watermark` feature wraps AudioSeal-style ONNX detectors. It expects a model
that accepts `[1, 1, T]` waveform input and returns at least two outputs:
detection scores and message logits.
The current implementation caches the typed model after the first `detect`
call, so repeated calls are best when buffer lengths stay consistent.
#### Config
```text
model_path: String
message_bits: 16 by default, must be <= 32
threshold: 0.5 by default, must be in [0, 1]
sample_rate: 16000 by default
```
#### Detection Walkthrough
```text
INPUT: AudioBuffer at config.sample_rate, non-empty
Step 1 - Build input:
Tensor shape [1, 1, samples.len()]
Step 2 - Concretize model:
on first call, set input fact for this exact T
type model, cache typed plan, build runnable
subsequent calls reuse the cached typed model
Step 3 - Run inference:
outputs[0] = detection scores
outputs[1] = message logits
Step 4 - Decode:
localization = detection scores as Vec<f32>
confidence = mean(localization), or 0.0 if empty
detected = confidence > threshold
message bit i = logits[i] >= 0, packed LSB-first
```
#### Output
```text
WatermarkResult
├── detected: bool
├── confidence: f32
├── message: u32
└── localization: Vec<f32>
```
---
## DSP Internals
### ShortTimeFFT
`ShortTimeFFT` owns:
```text
StftConfig
FFT plan: Arc<dyn RealToComplex<f32>>
window: Vec<f32>
scratch_in: Vec<f32>
scratch_out: Vec<Complex<f32>>
```
Config validation:
- `n_fft` must be non-zero and a power of two.
- `hop` must satisfy `0 < hop <= n_fft`.
Frame count:
```text
center=true:
n_frames = 1 + n_samples / hop
center=false:
if n_samples < n_fft: 0
else: 1 + (n_samples - n_fft) / hop
```
Classical fingerprinters use `center=false` so frame 0 starts at sample 0 and
streaming can match offline exactly.
### Power vs Magnitude
Classical extractors use `power_flat` or `process_frame_power`:
```text
power = re^2 + im^2
```
Wang and Panako then compute:
```text
10 * log10(power)
```
This is equivalent to `20 * log10(sqrt(power))` and avoids a per-bin `sqrt`.
### PeakPicker
`PeakPicker` finds local maxima in a row-major spectrogram:
```text
candidate if:
value > min_magnitude
value >= rolling_max_2d(value's neighborhood)
```
The 2-D rolling max is separable:
1. run a 1-D monotonic-deque max over frequency for every row
2. run a 1-D monotonic-deque max over time for every column
This is amortized `O(n_frames * n_bins)` and independent of neighborhood size.
After local maximum selection, `adaptive_per_second` keeps only the top
`target_per_sec` peaks per one-second bucket.
### MelFilterBank
The mel filterbank owns a row-major matrix:
```text
matrix shape = (n_mels, n_fft / 2 + 1)
```
It supports HTK and Slaney scales. Filters are triangular and Slaney-normalized
to unit area in linear frequency.
Log-mel from power:
```text
out[mel] = log10(sum(matrix[mel, bin] * power[bin]) + 1e-10)
```
### Resampler
Two resamplers are provided:
```text
linear(input, from_sr, to_sr)
cheap interpolation baseline
SincResampler
windowed-sinc convolution
Kaiser window
default half_taps = 32
default beta = 8.6
default polyphase_steps = 256
cutoff = min(from_sr, to_sr) / from_sr / 2
```
`decode_to_mono_at` uses `SincResampler::new` when the decoded file sample rate
does not match the requested target rate.
---
## Data Structures & Memory Layouts
### Shared Value Types
```text
SampleRate(NonZeroU32)
AudioBuffer<'a> { samples: &'a [f32], rate: SampleRate }
TimestampMs(u64)
```
`AudioBuffer` is a borrowed mono PCM view. It does not validate sample range;
downstream code clips or interprets samples as needed.
### Peak
```text
Peak - repr(C), bytemuck::Pod, 12 bytes
+---------------+---------+
| t_frame: u32 | 4 bytes |
| f_bin: u16 | 2 bytes |
| _pad: u16 | 2 bytes |
| mag: f32 | 4 bytes |
+---------------+---------+
```
The explicit `_pad` field prevents implicit padding and keeps the layout
stable for `Pod`.
### WangHash
```text
WangHash - repr(C), bytemuck::Pod, 8 bytes
+---------------+---------+
| hash: u32 | 4 bytes |
| t_anchor: u32 | 4 bytes |
+---------------+---------+
```
### PanakoHash
```text
PanakoHash - repr(C), bytemuck::Pod, 16 bytes
+---------------+---------+
| hash: u32 | 4 bytes |
| t_anchor: u32 | 4 bytes |
| t_b: u32 | 4 bytes |
| t_c: u32 | 4 bytes |
+---------------+---------+
```
### Fingerprint Containers
```text
WangFingerprint
├── hashes: Vec<WangHash>
└── frames_per_sec: f32
PanakoFingerprint
├── hashes: Vec<PanakoHash>
└── frames_per_sec: f32
HaitsmaFingerprint
├── frames: Vec<u32>
└── frames_per_sec: f32
```
Container structs are not `Pod` because they own `Vec`s. The individual hash
rows are `Pod` and can be byte-cast for persistence.
---
## I/O and Resampling
### Decoder
The `std` feature enables Symphonia-backed helpers:
```text
decode_to_mono(path) -> Result<(Vec<f32>, u32)>
decode_to_mono_at(path, target_sr) -> Result<Vec<f32>>
```
Supported formats follow the enabled Symphonia features:
- MP3
- FLAC
- WAV
- OGG/Vorbis
- AAC in MP4
- raw PCM
### Downmixing
Decoded multi-channel audio is downmixed by averaging channels per frame:
```text
mono[i] = sum(channel[c][i]) / channel_count
```
### Recoverable Decode Errors
Recoverable per-packet `IoError` and `DecodeError` values from Symphonia are
skipped. Fatal probe, packet, decoder construction, and unrecoverable decode
errors become `AfpError::Io`.
### Sample-Rate Contract
Classical extractors are sample-rate strict:
```text
Wang: 8000 Hz
Panako: 8000 Hz
Haitsma: 5000 Hz
```
Use `decode_to_mono_at` or `SincResampler` before calling `extract` if source
audio is at a different rate.
---
## Edge Cases & Invariants
### Input Invariants
- Classical APIs require mono `f32` input.
- Wrong sample rates return `AfpError::UnsupportedSampleRate`.
- Inputs shorter than `min_samples()` return `AfpError::AudioTooShort`.
- Wang and Panako require at least 2 seconds at 8 kHz.
- Haitsma requires at least 2 seconds at 5 kHz.
- Silence is valid input: Wang/Panako emit empty hashes; Haitsma emits zero frames.
### STFT Invariants
- `n_fft` must be a power of two.
- `hop` must be in `(0, n_fft]`.
- Classical algorithms use `center=false`.
- `n_bins = n_fft / 2 + 1`.
- Streaming frame timestamps use `frame_index * hop * 1000 / sample_rate`.
### Wang Invariants
- Frequency bins are quantized from 513 bins to 512 buckets.
- Pair `dt` is at least 1.
- Pair target zone uses `dt <= target_zone_t` and `abs(df) <= target_zone_f`.
- Output is sorted by `(t_anchor, hash)`.
- Algorithm name is `"wang-v1"`.
### Panako Invariants
- Target cone uses strict bounds: `dt < target_zone_t`, `abs(df) < target_zone_f`.
- `df_ab` and `df_bc` are clamped to signed 8-bit range.
- `beta` is clamped to 0..31.
- Low 7 bits of the packed hash are reserved zero.
- Output is sorted by `(t_anchor, t_b, t_c, hash)`.
- Algorithm name is `"panako-v2"`.
### Haitsma Invariants
- Band range must satisfy `fmin > 0`, `fmax > fmin`, and `fmax < sample_rate / 2`.
- There are 33 band energies and 32 output bits per frame.
- Band 0 maps to bit 31; band 31 maps to bit 0.
- Frame 0 has no output hash because it has no previous frame.
- Algorithm name is `"haitsma-v1"`.
### Streaming Invariants
- Streaming output matches offline output for the same complete input.
- `push` work is proportional to new complete frames/windows, not the total stream length.
- `flush` drains pending complete work.
- Partial Haitsma and neural windows do not emit.
- Streaming Wang/Panako flush finalizes remaining buckets and anchors.
### Neural Invariants
- `sample_rate > 0`.
- `n_fft` is a power of two and at least 2.
- `0 < hop <= n_fft`.
- `n_mels > 0`.
- `0 <= fmin < fmax <= sample_rate / 2`.
- `window_samples >= n_fft`.
- `0 < hop_samples <= window_samples`.
- Model input is concretized as `[1, n_mels, n_frames]`.
### Watermark Invariants
- `message_bits <= 32`.
- `threshold` is in `[0, 1]`.
- Input sample rate must match `WatermarkConfig::sample_rate`.
- Empty input is rejected.
- Message bits are packed LSB-first into `u32`.
---
## Module Map
```text
src/
├── lib.rs # crate docs, feature gates, re-exports, VERSION
├── error.rs # AfpError, Result
├── types.rs # SampleRate, AudioBuffer, TimestampMs
├── fp.rs # Fingerprinter, StreamingFingerprinter traits
├── classical/
│ ├── mod.rs # re-exports and algorithm overview
│ ├── wang.rs # Wang offline and streaming implementations
│ ├── panako.rs # Panako offline and streaming implementations
│ └── haitsma.rs # Haitsma offline and streaming implementations
├── dsp/
│ ├── mod.rs # DSP module exports
│ ├── stft.rs # ShortTimeFFT, StftConfig
│ ├── peaks.rs # PeakPicker, Peak, rolling max
│ ├── resample.rs # linear and sinc resamplers
│ ├── mel.rs # MelFilterBank, MelScale
│ └── windows.rs # Hann and other window functions
├── io/
│ ├── mod.rs
│ └── decoder.rs # Symphonia decode_to_mono helpers
├── neural/
│ ├── mod.rs
│ ├── frontend.rs # LogMelFrontend
│ ├── embedder.rs # NeuralEmbedder, config, output types
│ ├── streaming.rs # StreamingNeuralEmbedder
│ └── test_support.rs # ONNX fixture helpers for tests
└── watermark/
├── mod.rs
└── detector.rs # WatermarkDetector, config, result
```
### Public API Surface
```text
audiofp
├── VERSION
├── AfpError / Result
├── AudioBuffer / SampleRate / TimestampMs
├── Fingerprinter / StreamingFingerprinter
├── classical::{Wang, Panako, Haitsma}
├── classical::{StreamingWang, StreamingPanako, StreamingHaitsma}
├── dsp::{stft, peaks, resample, mel, windows}
├── io::{decode_to_mono, decode_to_mono_at} # std
├── neural::{NeuralEmbedder, StreamingNeuralEmbedder} # neural
└── watermark::{WatermarkDetector, WatermarkConfig} # watermark
```
---
## End-to-End Example Trace
```text
Input:
"song.mp3"
Decode:
decode_to_mono_at("song.mp3", 8000)
Symphonia probes and decodes packets
channels are averaged to mono
SincResampler converts native rate to 8000 Hz if needed
Fingerprint:
Wang::extract(AudioBuffer { samples, rate: HZ_8000 })
validate rate and length
STFT power spectrogram with n_fft=1024, hop=128
convert power to dB log magnitude
pick spectral peaks with 31x31 rolling max
keep top 30 peaks per second
build anchor-target landmark pairs
pack 32-bit Wang hashes
sort output deterministically
Output:
WangFingerprint {
hashes: Vec<WangHash>,
frames_per_sec: 62.5,
}
```
---
## Practical Tuning Notes
### Pick the Algorithm by Storage and Robustness
| Wang | General music ID and offset voting |
| Panako | Matching with mild tempo/time-scale changes |
| Haitsma | Compact dense fingerprints and fast extraction |
| Neural | Semantic similarity, cover/remix search, learned embeddings |
| Watermark | Detecting model-specific embedded marks |
### Keep Sample Rates Explicit
Classical extractors intentionally do not resample internally. That keeps hash
bytes stable and avoids hidden quality tradeoffs. Use:
```rust
use audiofp::io::decode_to_mono_at;
let wang_ready = decode_to_mono_at("song.flac", 8_000)?;
let haitsma_ready = decode_to_mono_at("song.flac", 5_000)?;
# Ok::<_, audiofp::AfpError>(())
```
### Persist Algorithm Names Beside Bytes
When storing hashes, persist `Fingerprinter::name()` with the bytes:
```text
wang-v1
panako-v2
haitsma-v1
neural-onnx-v0
```
The version suffix is the semantic compatibility boundary for hash bytes.