Crate openjd_snapshots 

Content-addressed directory tree snapshots with S3 integration.

Status: experimental. This crate is under active development and its public API may change without notice between releases. Some on-disk formats are not yet stable — see below for the per-format status.

This crate captures directory tree snapshots, computes diffs, and transfers data to/from content-addressed storage (S3 or local filesystem). It is a standalone library with no dependency on openjd-model or openjd-expr.

On-disk Manifest Formats

The crate supports two JSON manifest formats, identified at runtime by the codec::ManifestFormat enum:

• V2023 — Stable. The on-disk format used by AWS Deadline Cloud’s job attachments. The encode_snapshot_v2023 / decode_v2023 entry points and the "manifestVersion": "2023-03-03" wire format are the authoritative reference for Deadline Cloud interop.
• V2025 — Experimental draft. A proposed evolution with richer features (diffs, explicit directories, symlinks, file chunking). Its on-disk specificationVersion strings already carry the beta-2025-12 tag (e.g. "absolute-manifest-snapshot-beta-2025-12"). The wire format is expected to change before any stable release; do not use it for long-term storage or for interop with external systems.

Manifest Types

A manifest describes a set of files, directories, and symlinks with their metadata. Four concrete types are organized by two dimensions — path style and manifest kind:

	Full snapshot	Diff (changes only)
Relative paths	Snapshot	SnapshotDiff
Absolute paths	AbsSnapshot	AbsSnapshotDiff

These are type aliases over Manifest<P, K> with phantom type parameters that provide compile-time safety for path style and manifest kind.

Typical Workflows

Upload (collect → hash+upload → extract relative manifest):

COLLECT → AbsSnapshot → HASH_UPLOAD → AbsSnapshot (hashed) → SUBTREE → Snapshot

Download (join to absolute paths → download):

Snapshot → JOIN → AbsSnapshot → DOWNLOAD → files on disk

Incremental sync (diff → compose → upload only changes):

COLLECT → DIFF(old, new) → SnapshotDiff → COMPOSE(base, diffs) → Snapshot

Operations

Operation	Function	Description
COLLECT	collect_abs_snapshot	Walk filesystem into an AbsSnapshot
HASH	hash_abs_manifest	Compute content hashes (CPU-parallel via rayon)
HASH_UPLOAD	hash_upload_abs_manifest	Hash and upload in a single pipelined pass
DOWNLOAD	download_abs_manifest	Download files from content-addressed storage
DIFF	diff_snapshots	Compute changes between two snapshots
COMPOSE	compose_snapshot_with_diffs, compose_diffs	Layer manifests together
FILTER	filter_manifest	Include/exclude paths by glob pattern
SUBTREE	subtree_snapshot	Extract and rebase a subdirectory
JOIN	join_snapshot	Prepend a root path to make paths absolute
PARTITION	partition_manifest	Split by root directories
CACHE_SYNC	cache_sync_manifest	Copy data between caches (S3↔filesystem)

Re-exports

pub use codec::decode_manifest;

pub use codec::decode_v2023;

pub use codec::decode_v2023_as_diff;

pub use codec::decode_v2025;

pub use codec::encode_abs_snapshot_diff_v2025;

pub use codec::encode_abs_snapshot_v2025;

pub use codec::encode_snapshot_diff_v2023;

pub use codec::encode_snapshot_diff_v2025;

pub use codec::encode_snapshot_v2023;

pub use codec::encode_snapshot_v2025;

pub use codec::DecodedManifest;

pub use codec::ManifestFormat;

pub use data_cache::AsyncDataCache;

pub use data_cache::CopyResult;

pub use data_cache::FileSystemDataCache;

pub use data_cache::MultipartDataCache;

pub use data_cache::RangeReadDataCache;

pub use data_cache::S3DataCache;

pub use error::Result;

pub use error::SnapshotError;

pub use hash::human_readable_file_size;

pub use hash::HashAlgorithm;

pub use hash::DEFAULT_FILE_CHUNK_SIZE;

pub use hash::DEFAULT_S3_MULTIPART_PART_SIZE;

pub use hash::WHOLE_FILE_CHUNK_SIZE;

pub use hash_cache::HashCache;

pub use manifest::AbsManifest;

pub use manifest::AbsSnapshot;

pub use manifest::AbsSnapshotDiff;

pub use manifest::DirEntry;

pub use manifest::FileEntry;

pub use manifest::Manifest;

pub use manifest::ManifestEntry;

pub use manifest::ManifestRef;

pub use manifest::RelManifest;

pub use manifest::Snapshot;

pub use manifest::SnapshotDiff;

pub use manifest::SymlinkPolicy;

pub use ops::cache_sync_manifest;

pub use ops::collect_abs_snapshot;

pub use ops::compose_diffs;

pub use ops::compose_snapshot_with_diffs;

pub use ops::diff_snapshots;

pub use ops::download_abs_manifest;

pub use ops::entries_differ;

pub use ops::filter_manifest;

pub use ops::hash_abs_manifest;

pub use ops::hash_abs_snapshot;

pub use ops::hash_abs_snapshot_diff;

pub use ops::hash_upload_abs_manifest;

pub use ops::join_manifest;

pub use ops::join_manifest_rel;

pub use ops::join_snapshot;

pub use ops::join_snapshot_diff;

pub use ops::join_snapshot_diff_rel;

pub use ops::join_snapshot_rel;

pub use ops::partition_manifest;

pub use ops::partition_rel_manifest;

pub use ops::subtree_manifest;

pub use ops::subtree_rel_manifest;

pub use ops::subtree_rel_snapshot;

pub use ops::subtree_rel_snapshot_diff;

pub use ops::subtree_snapshot;

pub use ops::subtree_snapshot_diff;

pub use ops::CacheSyncOptions;

pub use ops::CacheSyncResult;

pub use ops::CacheSyncStatistics;

pub use ops::CollectOptions;

pub use ops::DiffOptions;

pub use ops::DownloadOptions;

pub use ops::DownloadResult;

pub use ops::DownloadStatistics;

pub use ops::FileConflictResolution;

pub use ops::HashOptions;

pub use ops::HashResult;

pub use ops::HashStatistics;

pub use ops::HashUploadOptions;

pub use ops::IncludeExcludePathsFilter;

pub use ops::PartitionOptions;

pub use ops::UploadResult;

pub use ops::UploadStatistics;

pub use s3_check_cache::S3CheckCache;

Modules

codec

Encode/decode manifests to/from on-disk v2023 and v2025 JSON formats.

data_cache

error

hash

hash_cache

manifest

ops

s3_check_cache