assetpack

assetpack provides Rust crates for content-addressed asset packing. It focuses on splitting files into stable chunks, deduplicating payloads by SHA3-256, writing SQLite-backed/Sealed packs with content-addressed objects, and recording recipes that can rebuild the original bytes.

Crates

assetpack-core: core hashing, chunking, codec, recipe, FileReader, pipeline, and SQLite/Sealed pack primitives.
assetpack-transform-precomp2: optional file transforms built on precomp2, with zstd and lzma wrapping variants.
Core features serde, sqlite-pack, rusqlite-store, and file-transform are enabled by default. sqlx-store, sealed, and sealed-encryption are opt-in.

Features

SHA3-256 object identity and recipe integrity checks.
FastCDC content-defined chunking with stable default bounds.
Per-chunk raw, zstd, and Brotli storage with automatic fallback when compression is not worthwhile.
SQLite object storage for chunks and recipes, plus optional namespaced stores for embedding.
File transform selection with decode verification before accepting transformed output.
Merkle index rebuilds, integrity checks, membership proofs, and batched proof lookup.
Optional precomp2, precomp2-zstd, and precomp2-lzma transforms.

Usage

Choosing a store

The synchronous backends implement ObjectSource; SQLx implements AsyncObjectSource:

SealedPack: an immutable single-file snapshot for distribution or archival. It is built once from a complete object set, can be encrypted, and is never updated in place — changing content means building a new pack.
SqlitePack: owns a rusqlite connection and standalone database-file lifecycle.
RusqliteStore<'_>: borrows an existing rusqlite connection or transaction and supports a table prefix.
SqlxStore: owns an existing SQLx SqlitePool, supports a table prefix, and accepts caller-owned SQLx transactions.

sqlite-pack and rusqlite-store do not enable SQLx, Tokio, or futures. Async applications choose their own scheduling boundary when using a synchronous backend.

Storing files

Pipeline splits an input into chunks, compresses each one, and computes everything needed to rebuild the file. Persist the plan and keep the returned recipe hash:

use assetpack_core::{Codec, FileHint, Hash32, ObjectKind, ObjectRecord, Pipeline, PipelineConfig, SqlitePack, build_recipe};

fn store_file(pack: &mut SqlitePack, data: Vec<u8>) -> assetpack_core::Result<Hash32> {
  let original_hash = Hash32::sha3_256(&data);
  let hint = FileHint { size: data.len() as u64, extension: None, head: None };
  let plan = Pipeline::new(PipelineConfig::default()).run(data, &hint, original_hash, None)?;

  let recipe_chunks: Vec<_> = plan.chunks.iter().map(|chunk| (chunk.hash, chunk.raw_len)).collect();
  let recipe = build_recipe(plan.original_size, &recipe_chunks, original_hash, plan.transform_id, plan.transform_version);
  let recipe_hash = Hash32::sha3_256(&recipe);

  let objects: Vec<_> = plan
    .chunks
    .into_iter()
    .map(|chunk| ObjectRecord {
      hash: chunk.hash,
      kind: ObjectKind::Chunk,
      decoded_len: u64::from(chunk.raw_len),
      codec: chunk.codec,
      stored_bytes: chunk.payload.expect("payload is kept unless discard_payload is set"),
    })
    .collect();
  pack.transaction(|store| {
    store.put_objects_batch(&objects)?;
    store.put_recipe(recipe_hash, &recipe, Codec::Raw)
  })?;
  Ok(recipe_hash)
}

To also try precomp2-family transforms during selection, build a TransformSelector from assetpack_transform_precomp2::default_specs() and pass it as the last argument of run.

Building a SealedPack

use assetpack_core::{DEFAULT_FORMAT_TAG, SealedPackBuilder, SqlitePack};

fn export(pack: &SqlitePack, root_recipe: assetpack_core::Hash32) -> assetpack_core::Result<Vec<u8>> {
  let mut objects = Vec::new();
  loop {
    let page = pack.object_records(objects.len() as u64, 256)?;
    if page.is_empty() { break; }
    objects.extend(page);
  }
  SealedPackBuilder::build_plain(DEFAULT_FORMAT_TAG, root_recipe, objects)
}

Restoring a file

Construct a TransformDecoderRegistry with every transform version you have ever written. Use FileReader with Sealed/rusqlite sources and AsyncFileReader with SqlxStore:

use assetpack_core::{FileReadLimits, FileReader, Hash32, ObjectSource, TransformDecoderRegistry};

fn restore<S: ObjectSource + ?Sized>(source: &S, root: Hash32) -> assetpack_core::Result<Vec<u8>> {
  let decoders = TransformDecoderRegistry::default();
  FileReader::new(source, &decoders, FileReadLimits::default()).read_file(root)
}

Decoders are selected by the exact (transform_id, transform_version) recorded in the recipe; unknown pairs fail with UnsupportedTransform instead of falling back silently. For precomp2 support, register assetpack_transform_precomp2::default_decoders(&config). Restoration returns the complete file; seek or range decoding of transformed files is not supported.

What a successful call proves

ObjectSource::read_object: the returned bytes match the requested object hash.
AsyncObjectSource::read_object: the same contract, with explicit asynchronous source scheduling.
SealedPackReader::verify_all_objects(): every object in the container passes that same check.
FileReader::read_file / restore_file: the restored bytes equal the original file, in size and SHA3-256.

Compatibility

These crates are pre-1.0 and the public API may still change. The SealedPack v1 on-disk format is pinned by a byte-exact golden fixture; any format change requires a new revision. Parsers and the shared read path are fuzzed continuously in CI: curated seeds live in fuzz/corpus, dynamic corpus stays in CI caches, and every crash becomes a deterministic regression test.

License

Licensed under the AGPL-3.0-or-later license.

assetpack-core 0.3.0