codec-corpus

Runtime access to the imazen/codec-corpus test image collection and third-party fuzz/conformance corpora. No data ships with the crate — datasets download on first use and cache locally.

let corpus = codec_corpus::Corpus::new()?;
let valid = corpus.get("webp-conformance/valid")?;
for entry in std::fs::read_dir(valid)? {
    let path = entry?.path();
    // decode, validate, benchmark...
}

What it does

1. You call corpus.get("some-folder/optional-subpath")
2. If the folder isn't cached (or the cache is stale), it downloads via git sparse checkout or HTTP tarball
3. Returns the local path. Done.

Downloads use shell git (preferred) with fallback to curl/wget/powershell. No heavy HTTP crate dependencies.

Install

[dev-dependencies]
codec-corpus = "1"

Usage

imazen/codec-corpus (default)

Any path without a recognized third-party prefix downloads from imazen/codec-corpus:

use codec_corpus::Corpus;

#[test]
#[ignore] // network access required
fn jpeg_conformance() {
    let corpus = Corpus::new().unwrap();
    let valid = corpus.get("jpeg-conformance/valid").unwrap();

    for entry in std::fs::read_dir(valid).unwrap() {
        let path = entry.unwrap().path();
        let data = std::fs::read(&path).unwrap();
        // test your decoder...
    }
}

Third-party sources (built-in registry)

Prefixed paths resolve to known external sources automatically:

let corpus = codec_corpus::Corpus::new().unwrap();

// OSS-Fuzz backup corpora (downloaded as ZIP from GCS)
let ossfuzz_jpeg = corpus.get("oss-fuzz/libjpeg-turbo").unwrap();
let ossfuzz_png = corpus.get("oss-fuzz/libpng").unwrap();
let ossfuzz_jxl = corpus.get("oss-fuzz/libjxl").unwrap();

// dvyukov/go-fuzz-corpus subfolders (git sparse checkout)
let gofuzz_gif = corpus.get("go-fuzz-corpus/gif").unwrap();
let gofuzz_png = corpus.get("go-fuzz-corpus/png").unwrap();
let gofuzz_jpeg = corpus.get("go-fuzz-corpus/jpeg").unwrap();

// libjpeg-turbo fuzz seed corpus (git sparse checkout)
let ljt_fuzz = corpus.get("libjpeg-turbo-fuzz").unwrap();

// image-rs/image test images (git sparse checkout)
let imagers = corpus.get("image-rs/tests/images").unwrap();

Third-party sources are cached under third-party/ in the cache root and re-fetched if the .fetched marker is older than 7 days (configurable via with_max_age()).

Ad-hoc sources

Fetch from arbitrary locations without needing them in the built-in registry:

let corpus = codec_corpus::Corpus::new().unwrap();

// Arbitrary GitHub repo subfolder
let path = corpus.github_repo("niclas-aspect/jxl-rs", "test-data", "main").unwrap();

// Download and cache a ZIP URL
let path = corpus.zip_url("custom-corpus", "https://example.com/corpus.zip").unwrap();

// Download and cache a tarball URL
let path = corpus.tar_url("my-corpus", "https://example.com/corpus.tar.gz").unwrap();

// Point at a local directory (validates existence, no download)
let path = corpus.local_path("/home/user/my-images").unwrap();

Custom cache location

let corpus = Corpus::with_cache_root("/mnt/fast-storage")?;

Or via environment variable:

CODEC_CORPUS_CACHE=/mnt/fast-storage cargo test -- --ignored

Cache staleness

Third-party sources use a .fetched timestamp marker. By default, sources older than 7 days are re-fetched:

use std::time::Duration;
let corpus = codec_corpus::Corpus::new()?
    .with_max_age(Duration::from_secs(30 * 24 * 3600)); // 30 days

Check what's cached

if corpus.is_cached("pngsuite") {
    println!("already downloaded");
}

for name in corpus.list_cached() {
    println!("cached: {name}");
}

Built-in third-party registry

Datasets (imazen/codec-corpus)

Any top-level folder in the codec-corpus repo is a valid path. Pass any path into get() — the first component determines the download unit.

Quality calibration

Format conformance

Decoder robustness

Full dataset descriptions and per-file attribution: codec-corpus README.

Cache layout

~/.cache/codec-corpus/v1/          # Linux
~/Library/Caches/codec-corpus/v1/  # macOS
%LOCALAPPDATA%\codec-corpus\v1\    # Windows

  .version          # "1.0.3" — triggers re-download on version change
  .lock             # fd-lock for concurrent access
  pngsuite/
  jpeg-conformance/
  third-party/                     # Third-party sources (not version-gated)
    oss-fuzz__libpng/
      .fetched      # Unix timestamp — re-fetch if >7 days old
      ...files...
    go-fuzz-corpus__gif/
    github__owner__repo__path/
    ...

Different major versions coexist (v1/, v2/). imazen/codec-corpus datasets re-download on any crate version change. Third-party sources use time-based staleness (default 7 days) and survive version changes.

CI integration

- uses: actions/cache@v4
  with:
    path: ~/.cache/codec-corpus
    key: corpus-v1
- run: cargo test --release -- --ignored

No special setup. The crate handles downloading; the CI cache avoids re-downloading across runs.

Using codec-corpus from WASM

codec-corpus compiles for wasm32-wasip1 (and wasm32-unknown-unknown). On WASM, downloads are not available since the crate relies on host subprocesses (git, curl). Instead, prefetch on the host and preopen the cache for the WASM runtime.

1. Prefetch on the host (native)

# Download the datasets you need into a local cache directory
CODEC_CORPUS_CACHE=$(pwd)/corpus cargo run --example prefetch -- cid22 webp-conformance

Or just run your native tests once — Corpus::new() populates ~/.cache/codec-corpus/ automatically.

2. Run WASM tests with a preopened cache

# Map the host cache directory into the WASM sandbox
CODEC_CORPUS_CACHE=/corpus \
  CARGO_TARGET_WASM32_WASIP1_RUNNER="wasmtime --dir $(pwd)/corpus::/corpus" \
  cargo test --target wasm32-wasip1

The --dir host_path::guest_path flag gives the WASM process read access to the host directory at the guest mount point.

3. In your WASM code

// Point at the preopened cache root
let corpus = codec_corpus::Corpus::with_cache_root("/corpus").unwrap();

// These work — they only read the filesystem
assert!(corpus.is_cached("webp-conformance"));
let datasets = corpus.list_cached();
let path = corpus.get("webp-conformance").unwrap();

// This returns Error::DownloadUnsupported — by design
let err = corpus.get("not-yet-cached").unwrap_err();
// "downloads are not supported on this platform; dataset 'not-yet-cached' must be pre-cached on the host"

Calling get() for an uncached dataset on WASM returns Error::DownloadUnsupported (not NetworkUnavailable). This is intentional — downloads must happen host-side.

Dependencies

Four Rust crates, all small:

• dirs — cross-platform cache directory
• fd-lock — file locking for concurrent safety (native only, not on WASM)
• serde + serde_json — manifest parsing for R2/JSONL corpus sources

Archive extraction uses system commands (tar, unzip, powershell). No reqwest, ureq, gix, or toml.

License

The crate itself is Apache-2.0. Each dataset in the corpus has its own license — see the table above and the full license summary.