Expand description
zblob — generic resumable chunked blob and directory transfer over Zenoh.
A small, self-contained library for moving a large artifact (a file, a report bundle, a pcap, a directory tree) between Zenoh peers with progress, BLAKE3 verified streaming, range resume, and bounded memory. It carries no application-specific types.
§The three tiers
| Tier | What it moves | Entry points |
|---|---|---|
| 1 — blob by id | one blob, named by a caller-chosen BlobId | BlobServer, BlobClient |
| 2 — content-addressed trees | a directory snapshot, deduplicated chunk-wise | TreeServer, TreeClient, StoreClient, Publisher |
fanout (feature fanout) | one-to-many rollout of one blob | fanout::fanout_file, fanout::receive_fanout |
Tier 1 is the whole of the model described below. Tier 2 is the casync
model: a snapshot is a TreeIndex (a depth-first entry list whose files
reference their chunks by BLAKE3 hash) plus a ContentStore keyed
<prefix>/blake3/<hex>. A client fetches only the chunks it is missing,
in batched rounds, and materializes defensively — so an interrupted pull
resumes for free and identical chunks transfer once across files, versions
and producers. A producer can serve it live with a TreeServer, or
Publisher it into a router-hosted Zenoh storage and exit (see
docs/router-storage.md). TreeClient::fetch_file pulls one path out of
a snapshot without materializing the tree, and the probes
(StoreClient::probe, TreeClient::probe_snapshot) report partial
possession, so a client can choose a holder before fetching.
Every key expression is built through keys, and every prefix is typed
by the role it plays: a server owns a concrete ServePrefix, a client
asks through a QueryPrefix that may name several origins. A server
cannot be built on a wildcard because there is no value to build one from.
let serve = ServePrefix::new("demo/blobs")?;
let server = BlobServer::new(&session, serve.clone());
let manifest = server.register_file(BlobSpec::new("blob-1"), path).await?;
let handle = server.spawn().await?;
// Transfers are call builders: what to fetch and where it goes are
// positional; progress, cancellation and overwrite policy are optional.
let client = BlobClient::new(&session, QueryPrefix::from(&serve));
let stats = client
.download_to(&DownloadRequest::pinned("blob-1", manifest.root), dest)
.await?;§Model (wire v3)
One queryable serves every blob under a key prefix:
queryable on: <prefix>/**
manifest GET: <prefix>/<id>/manifest -> the Manifest (one reply)
slice GET: <prefix>/<id>/**?ranges=<spec> -> bao slice replies (any order)
slice reply: <prefix>/<id>/slice/<index>A download is a manifest GET followed by range-set slice GETs. The
manifest-first step is not cosmetic — Zenoh does not order query
replies — and each slice reply is a bao slice: the chunk’s bytes plus
the parent hashes proving them against the manifest’s BLAKE3 root
(see crate::wire and the verify module). Every reply is therefore
independently verified before it touches disk, out of order, at 16 KiB
granularity; there is no end-of-transfer hash pass. Memory stays
O(chunk_size) regardless of blob size and arrival order.
The ranges parameter is a comma-separated list of half-open chunk-index
spans ("0-5,9,12-20"; a bare k means k..k+1), which is how resume
works: the client persists a chunk bitfield next to the .part file and
re-queries exactly its holes. See keys::slice_selector / keys::parse_ranges.
§Three facts this design relies on
- Backpressure is automatic.
Session::getdefaults toCongestionControl::Block, and replies inherit the query’s congestion control, so chunk replies block (rather than drop) when the link backs up. We therefore set no congestion control explicitly on queries — the only setter is behind Zenoh’sinternalfeature, which this crate deliberately does not enable. Do not “fix” this by enablinginternal. (Reply consolidation is a different knob: clients setConsolidationMode::Noneso replies stream instead of being buffered until query finalization. Publications — thefanouttier — are not queries and do setBlockexplicitly, since their default isDrop.) - Reply keys must intersect the query. Replies use
ReplyKeyExpr::MatchingQueryby default, so the client must GET the wildcard<prefix>/<id>/**for theslice/<i>replies to be accepted.keys::slice_selectorenforces the wildcard. The failure is not silent and not on the client: a bare-<id>GET makes the server’sreply()fail with a “does not intersect” error, so the diagnosis appears on the serving origin. (accept_replies(ReplyKeyExpr::Any)lifts the rule where a protocol genuinely needs disjoint reply keys.) - Silence is how a server says “not mine”, and it is cheap. A query finalizes once every matching queryable has completed, and completing without replying is immediate — an unknown id resolves in about a millisecond, not on the query timeout. That is what lets several servers share one prefix, each owning a disjoint set of ids.
- Any peer can answer, so one bad reply must not be fatal. A queryable key range is open: replies that fail decoding, validation, id matching, or root pinning are skipped rather than aborting the query, so a hostile or stale responder cannot deny a fetch that an honest replica still answers.
Re-exports§
Modules§
- fanout
- One-to-many fanout tier (optional
fanoutfeature). - gc
- Content-store lifecycle: tags + mark-and-sweep garbage collection.
- keys
- Zenoh key expressions for every plane this crate speaks, and the parsers that read them back.
- seed
- Local seeding (the desync model): satisfy a snapshot’s chunks from data already on this machine before touching the network.
- wire
- Wire encoding (v3) — postcard for every control message.
Structs§
- Blob
Client - Downloads blobs served by a
crate::BlobServerunder the same key prefix. - Blob
Client Builder - Builder for a
BlobClient(seeBlobClient::builder). - BlobId
- A validated blob identifier: one non-empty, non-reserved key segment.
- Blob
Probe - What one origin said about a blob, from
BlobClient::probe. - Blob
Server - Serves registered blobs over a Zenoh queryable at
<prefix>/**. - Blob
Server Builder - Builder for a
BlobServer(seeBlobServer::builder). - Blob
Spec - Caller-side parameters for registering a blob: everything in the
Manifestthat isn’t computed from the bytes. - Cancel
Token - A shared cancellation token. Clone it freely; cancelling any clone signals every in-flight transfer holding one to stop as soon as it can — without waiting for a reply that may never come.
- CdcParams
- FastCDC (v2020) content-defined chunking parameters for Tier 2.
- Channel
Sink - A
ProgressSinkthat forwards events onto a channel, fromprogress_channel. - Chunk
Probe - What one origin holds, from
StoreClient::probe. - Chunk
Ref - A reference to one content-addressed chunk.
- DirStore
- A filesystem
ContentStore:<root>/blake3/<xx>/<hex>(two-hex-char fanout so huge stores don’t melt directory listings), atomic puts (unique temp file → fsync → rename → dir fsync), and optional verify-on-read. - Download
- A configured download, awaited to run it.
- Download
Request - What to download: a blob id, optionally with a pinned BLAKE3 root.
- Download
ToWriter - A configured download into a writer, awaited to run it. See
BlobClient::download_to_writer. - File
Blob Source - A
BlobSourcebacked by a file on disk (e.g. a TTL’d report bundle). - Hash
- A 32-byte content hash, rendered as lowercase hex on the wire and in keys.
- Hash
Parse Error - Error parsing a
Hashfrom a hex string. - Manifest
- Describes a single blob: identity, size, chunking, and BLAKE3 root.
- Materialize
Policy - What a snapshot is allowed to do to the destination directory.
- Memory
Blob Source - A
BlobSourceserving a shared in-memory buffer (generated artifacts, tests). The buffer is behind anArc, so opening is free and concurrent transfers share one allocation. - Memory
Store - An in-memory
ContentStore(tests, ephemeral caches). - Publisher
- Publishes content-addressed chunks into a storage under one prefix.
- Push
Config - How a server accepts verified pushes (uploads), passed to
BlobServerBuilder::accept_push. - Query
Prefix - A key prefix a client may query. Single-segment wildcards (
*,$*…) are permitted;**is not. - Retry
Policy - Retry/backoff policy for the download loop. An attempt is a slice query that made no progress; queries that verify at least one new chunk reset the budget.
- Serve
Prefix - A key prefix a server or publisher may own: concrete, no wildcards.
- Server
Handle - Handle to a spawned server task: keeps it running, stops it on demand.
- Snapshot
Publisher - A
Publisherthat also publishes tree indices — seePublisher::snapshots. - Source
Fingerprint - A cheap identity snapshot of a
BlobSource(seeBlobSource::fingerprint). - Staged
- Where a staged download landed, from
BlobClient::download_staged. - Staged
Download - A configured staged download, awaited to run it. See
BlobClient::download_staged. - Store
Client - Reads single chunks from a content-addressed store by their hash.
- Store
Client Builder - Builder for a
StoreClient. - Store
Key - A 32-byte store encryption key. Generate it randomly, keep it outside the store directory, and give every store its own key.
- Transfer
Chunks - Fixed-size transfer-chunk arithmetic for one blob:
(chunk_size, total_len)fully determine every chunk’s index, byte range, and count. - Transfer
Stats - Statistics for one completed (or resumed-to-completion) transfer, returned
by
BlobClient::download_toandTreeClient::download_tree. - Tree
Client - Downloads a tree snapshot. Stateless server; persistent client (the store).
- Tree
Client Builder - Builder for a
TreeClient(seeTreeClient::builder). - Tree
Download - A configured snapshot download, awaited to run it. See
TreeClient::download_tree. - Tree
Index - A directory snapshot: an ordered entry list + a digest over it (
root_hash). - Tree
Server - Serves a tree snapshot: an index queryable + a content-addressed chunk
queryable, both backed by a
ContentStore. Each query is served on its own task under an in-flight semaphore, so one slow client cannot stall the rest. - Tree
Server Builder - Builder for a
TreeServer(seeTreeServer::builder). - Upload
- A configured upload, awaited to run it. See
BlobClient::upload_fileandBlobClient::upload_source.
Enums§
- Blob
Error - Errors raised by the blob server and client.
- Chunk
Compression - Compression policy for produced chunk containers.
- Entry
- One entry in a tree snapshot (paths are relative,
/-separated, UTF-8 — non-UTF-8 names are a build error, not silently mangled). - Error
Kind - Coarse classification of a
BlobError, fromBlobError::kind. - Hash
Algo - The hash algorithm a message or key names.
- Overwrite
- What to do when the destination path already exists at completion time.
- Progress
- A progress event emitted by a download.
- Settle
Coverage - How much of a published snapshot the read-back phase actually checks.
Constants§
- DEFAULT_
CHUNK_ SIZE - Default transfer chunk size: 256 KiB (16 bao verification groups, and four Zenoh fragments).
- MAX_
CHUNK_ SIZE - Largest allowed transfer chunk size (4 MiB) — bounds per-reply RAM.
- MIN_
CHUNK_ SIZE - Smallest allowed transfer chunk size (64 KiB).
Traits§
- Blob
Source - Opens a fresh reader over a blob’s bytes. Called once per registration (to
hash) and once per query (to serve) — and, on the client side, once to
hash and once to send per
BlobClient::upload_source— so the source can be reopened many times. Opening is synchronous — implementations should be cheap (anopen(2), not a download) and are always invoked on the blocking pool. - Content
Store - A local store of content-addressed chunks.
- Progress
Sink - A sink for
Progressevents. Implemented for anyFn(Progress)and for()(a no-op), so callers can pass a closure or nothing. - Push
Policy - Authorization hook for the push (upload) protocol: called for the initial offer and for every pushed slice, with the offered manifest and the opaque token the uploader attached to the query (if any).
- Read
AtSize - A positional, sized, thread-safe byte source: what a
BlobSourceopens.
Functions§
- build_
tree - Build a
TreeIndexfor the directory atroot, streaming every file through the content-defined chunker andputting each new chunk intostore. Memory stays O(max chunk size) regardless of tree size. - build_
tree_ from build_tree, reusing a previous snapshot’s work where the tree has not changed.- frame_
chunk - Frame raw chunk bytes into a Tier-2 container (the inverse of
unframe_chunk). - progress_
channel - A
ProgressSinkand the receiver its events arrive on. - unframe_
chunk - Decode a Tier-2 chunk container back to the chunk’s raw bytes.
Type Aliases§
- Error
Callback - Callback invoked with every error raised while serving a query.
- Result
- Convenience alias for fallible blob operations.