# audiofp
[](https://crates.io/crates/audiofp)
[](https://docs.rs/audiofp)
[](LICENSE)
[](https://github.com/themankindproject/audiofp/actions/workflows/ci.yml)
Audio fingerprinting library for Rust with **classical landmark and band-power algorithms**, **streaming extraction**, **file decoding**, and **AudioSeal-compatible watermark detection**.
## Overview
`audiofp` provides three complementary classical fingerprinters for music identification, each with offline and streaming variants:
| **Wang** | Music ID, Shazam-style matching | 8 kHz | 62.5 fps | ~2.4 KB/s (fan-out 10) |
| **Panako** | Music ID with ±5 % tempo robustness | 8 kHz | 62.5 fps | ~2.0 KB/s (fan-out 5) |
| **Haitsma** | Compact dense IDs, fastest extraction | 5 kHz | 78.125 fps | 312 B/s |
| **Streaming** | Real-time hash emission | (per algorithm) | (per algorithm) | Bit-exact offline parity |
| **Watermark** | AudioSeal detection (BYO ONNX) | 16 kHz | (per model) | Detection + 16-bit message |
Perfect for:
- Music identification ("what is this song?")
- Audio deduplication at scale
- Royalty / rights enforcement against re-encoded content
- Embedding-based similarity search and cover/remix detection (BYO ONNX model via the `neural` feature)
- Watermark verification on generative-AI audio
## Features
- **Three Classical Algorithms** - Wang (landmark pairs) + Panako (triplet hashes with tempo β) + Haitsma–Kalker (32-bit/frame band sign)
- **Truly Incremental Streaming** - Per-push CPU proportional to new samples, not total stream length. Rolling spectrogram + per-bucket finalisation + per-anchor target accumulator. Bit-exact parity with offline `extract` (verified by the test suite at every chunk size).
- **Bit-Exact Determinism** - Same input always produces the same hashes; verified down to 1-sample-per-push streaming chunks
- **`bytemuck::Pod` Hash Types** - Persist hashes directly to mmap'd files or ship over a C ABI without serialization
- **Audio File Decoding** - MP3, FLAC, WAV, OGG-Vorbis, AAC-in-MP4, raw PCM via Symphonia
- **High-Quality Resampling** - Built-in windowed-sinc Kaiser resampler with auto anti-aliasing cutoff
- **Watermark Detection** - AudioSeal-compatible ONNX wrapper (Tract backend); typed model is cached per input length and rebuilt automatically when the length changes
- **Neural Embedder** - Generic ONNX log-mel embedder with offline + streaming modes; build-once-runnable, zero-alloc `try_push_with` callback (scratch is allocated at construction, reused on every push)
- **DSP Primitives Reusable** - Public `dsp::stft`, `dsp::mel`, `dsp::peaks`, `dsp::resample`, `dsp::windows`
- **Allocation-Free Hot Path** - Streaming `push` reuses pre-allocated scratch after warmup
- **`no_std + alloc` Capable** - DSP and classical fingerprinters compile without std (host-only today; bare-metal in roadmap)
- **Feature-Gated Heavy Deps** - Symphonia and Tract both opt-in via Cargo features
- **Optional `mimalloc`** - Single-flag opt-in to install `mimalloc` as the global allocator
## Installation
```toml
[dependencies]
audiofp = "0.3"
```
### Feature Flags
| `std` | Yes | Enables `audiofp::io` (Symphonia file decoder) |
| `watermark` | No | Enables `audiofp::watermark` via Tract ONNX runtime |
| `neural` | No | Enables `audiofp::neural`: generic ONNX log-mel embedder via Tract (BYO model) |
| `mimalloc` | No | Installs `mimalloc::MiMalloc` as the process-wide `#[global_allocator]` |
Minimal build (no_std + alloc, DSP and classical only):
```toml
[dependencies]
audiofp = { version = "0.3", default-features = false }
```
## Quick Start
### Fingerprint a file
```rust
use audiofp::classical::Wang;
use audiofp::io::decode_to_mono_at;
use audiofp::{AudioBuffer, Fingerprinter, SampleRate};
fn main() -> Result<(), Box<dyn std::error::Error>> {
// Decode any supported file format and resample to Wang's 8 kHz.
// Needs ≥ ~2 s of audio or extract returns AudioTooShort.
let samples = decode_to_mono_at("song.mp3", 8_000)?;
let mut wang = Wang::default();
let buf = AudioBuffer::new(&samples, SampleRate::HZ_8000);
let fp = wang.extract(buf)?;
println!("{} hashes at {:.1} fps", fp.hashes.len(), fp.frames_per_sec);
for h in fp.hashes.iter().take(5) {
println!(" t_anchor={} hash={:08x}", h.t_anchor, h.hash);
}
Ok(())
}
```
### Streaming Mode
```rust
use audiofp::classical::StreamingWang;
use audiofp::StreamingFingerprinter;
fn main() {
let mut s = StreamingWang::default();
// Synthetic 8 kHz mono chunks (16 ms ≈ 128 samples). Swap for mic/file chunks.
let chunk = vec![0.0_f32; 128];
for _ in 0..100 {
for (timestamp, hash) in s.push(&chunk) {
println!("{:?} {:08x}", timestamp, hash.hash);
}
}
// Drain whatever's pending at end-of-stream.
for (timestamp, hash) in s.flush() {
println!("{:?} {:08x}", timestamp, hash.hash);
}
println!("latency: {} ms", s.latency_ms());
}
```
## Documentation
For complete API reference and usage examples, see [USAGE.md](USAGE.md).
## Architecture
### Fingerprint Types
Each algorithm emits a strongly-typed, `bytemuck::Pod`-castable result:
```
Wang offline Panako offline
┌──────────────────────────┐ ┌──────────────────────────┐
│ WangFingerprint │ │ PanakoFingerprint │
│ hashes: Vec<WangHash> │ │ hashes: Vec<PanakoHash>│
│ frames_per_sec: f32 │ │ frames_per_sec: f32 │
└──────────────────────────┘ └──────────────────────────┘
WangHash (8 bytes, repr(C)) PanakoHash (16 bytes, repr(C))
├── hash: u32 ├── hash: u32
└── t_anchor: u32 ├── t_anchor: u32
├── t_b: u32
└── t_c: u32
Haitsma offline
┌──────────────────────────┐
│ HaitsmaFingerprint │
│ frames: Vec<u32> │ one u32 per spectrogram frame ≥ 1
│ frames_per_sec: f32 │
└──────────────────────────┘
```
## Performance
Offline extract (`cargo bench --bench extract`, 30 s of synthetic audio):
| `Wang` | 79 ms | 380× |
| `Panako` | 81 ms | 370× |
| `Haitsma` | 42 ms | 714× |
Streaming push (`cargo bench --bench streaming`, 10 s of synthetic audio):
| `StreamingWang` | 10.5 ms | 10.6 ms | 2 256 ms |
| `StreamingPanako` | 11.6 ms | 11.4 ms | 2 784 ms |
| `StreamingHaitsma` | 6.3 ms | 6.7 ms | 409 ms |
Neural front-end (`cargo bench --features neural --bench neural_frontend`):
| `log_mel_pipeline_1s_window` | 297 µs |
| `strided_tensor_write` | 7.6 µs |
| `l2_normalize_1024d` | 2.5 µs |
Run benchmarks for your own host:
```bash
cargo bench --bench extract
cargo bench --bench streaming
cargo bench --bench extract -- --save-baseline main # save for diffing later
```
## Robustness
- **Codec-tolerant by design** — Wang and Panako are spectral-peak based; Haitsma is band-power-difference based. All three survive lossy re-encoding, verified by the test suite on real music:
| Codec | Wang (Jaccard) | Panako (Jaccard) | Haitsma (bit-sim) |
|-------|---------------|-----------------|-------------------|
| WAV/FLAC (lossless) | 1.000 | — | 1.000 |
| MP3 128 kbps | 0.40 | 0.45 | 0.93 |
| OGG-Vorbis | 0.36 | 0.42 | 0.91 |
| AAC (M4A) | 0.50 | 0.54 | 0.77 |
| AIFF (lossless) | 1.000 | — | — |
| **Cross-track (different song)** | **0.001** | — | — |
> Test audio: "Galway" and "Furious Freak" by Kevin MacLeod, 16 s each, 6 codec variants.
> Thresholds: Wang ≥ 0.25, Panako ≥ 0.20, Haitsma ≥ 0.75.
> In practice, 5–10 matching hashes suffice for confident identification.
- **Two-track discrimination verified** — different songs produce <0.1% hash overlap (random collision floor), while the same song across codecs produces 25–80% overlap.
- **421 tests** including adversarial stress tests, real-audio E2E across 6 codecs, and property-based streaming/offline parity checks. See [ROBUSTNESS.md](ROBUSTNESS.md) for full methodology.
## Comparison with Alternatives
| Pure Rust | Yes | No (FFI to C lib) | No |
| Wang landmarks | Yes | No | Yes |
| Panako triplets (tempo-robust) | Yes | No | No |
| Haitsma–Kalker | Yes | No | No |
| Streaming variants | Yes | Limited | No |
| Bit-exact streaming/offline parity | Yes | No | N/A |
| File decoding included | Yes (Symphonia) | Yes (limited) | Yes (FFmpeg) |
| Watermark detection | Yes (AudioSeal) | No | No |
| `no_std + alloc` capable | Yes (host) | No | N/A |
| `bytemuck::Pod` hash types | Yes | No | N/A |
| Built-in resampler | Yes | No | No |
## Examples
The `examples/` directory contains complete working programs that can be run with `cargo run --example <name>`:
- `enroll_file` — fingerprint a single audio file and print the unique Wang landmark count (`--features` default / `std`).
- `match_two_files` — print the number of Wang hash collisions between two files (the canonical "is this the same recording?" check).
- `compare_algorithms` — run Wang, Panako, and Haitsma–Kalker over the same file and report per-algorithm timing and hash counts.
- `stream_buffer` — feed Wang's streaming fingerprinter from an `io::Read` chunk-by-chunk.
- `dsp_starter` — STFT → mel → peaks pipeline on synthetic audio (no file, no optional features).
- `neural_embed` — load a BYO ONNX embedder and print embedding dim (`--features neural`).
- `watermark_detect` — load an AudioSeal-compatible ONNX model and print confidence (`--features watermark`).
```bash
cargo run --example dsp_starter
cargo run --example neural_embed --features neural -- path/to/model.onnx
cargo run --example watermark_detect --features watermark -- path/to/audioseal.onnx [audio.wav]
```
The doctests across the public API and [USAGE.md](USAGE.md) cover the full surface for users wiring `audiofp` into their own binary.
## Security
See [SECURITY.md](SECURITY.md) for the threat model (audio / PCM / ONNX / hash outputs) and how to report vulnerabilities privately. Fingerprints are **perceptual**, not cryptographic MACs — use `DecodeLimits` with `decode_to_mono_limited` for untrusted uploads.
## Contributing
See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines. Quick start:
```bash
git clone https://github.com/themankindproject/audiofp && cd audiofp
cargo test --all-features
cargo clippy --all-targets --all-features -- -D warnings
cargo fmt --all -- --check
```
CI runs `fmt`, `clippy`, and `test` on ubuntu/macOS/Windows on every push and PR.
## License
MIT License — see [LICENSE](LICENSE) for details.
## References
- Avery Wang, *An Industrial-Strength Audio Search Algorithm* (ISMIR 2003) — Wang landmarks
- Joren Six & Marc Leman, *Panako: A Scalable Acoustic Fingerprinting System* (ISMIR 2014); 2021 update — triplet β hash
- Jaap Haitsma & Ton Kalker, *A Highly Robust Audio Fingerprinting System* (ISMIR 2002) — band-power sign bits
- San Roman, R., Fernandez, P., Elsahar, H., Défossez, A., Furon, T. & Tran, T. *Proactive Detection of Voice Cloning with Localized Watermarking.* arXiv:2401.17264, 2024 (AudioSeal) — watermark model. <https://arxiv.org/abs/2401.17264>