Skip to main content

Crate zblob

Crate zblob 

Source
Expand description

zblob — generic resumable chunked blob and directory transfer over Zenoh.

A small, self-contained library for moving a large artifact (a file, a report bundle, a pcap, a directory tree) between Zenoh peers with progress, BLAKE3 verified streaming, range resume, and bounded memory. It carries no application-specific types.

§The three tiers

TierWhat it movesEntry points
1 — blob by idone blob, named by a caller-chosen BlobIdBlobServer, BlobClient
2 — content-addressed treesa directory snapshot, deduplicated chunk-wiseTreeServer, TreeClient, StoreClient, Publisher
fanout (feature fanout)one-to-many rollout of one blobfanout::fanout_file, fanout::receive_fanout

Tier 1 is the whole of the model described below. Tier 2 is the casync model: a snapshot is a TreeIndex (a depth-first entry list whose files reference their chunks by BLAKE3 hash) plus a ContentStore keyed <prefix>/blake3/<hex>. A client fetches only the chunks it is missing, in batched rounds, and materializes defensively — so an interrupted pull resumes for free and identical chunks transfer once across files, versions and producers. A producer can serve it live with a TreeServer, or Publisher it into a router-hosted Zenoh storage and exit (see docs/router-storage.md). TreeClient::fetch_file pulls one path out of a snapshot without materializing the tree, and the probes (StoreClient::probe, TreeClient::probe_snapshot) report partial possession, so a client can choose a holder before fetching.

Every key expression is built through keys, and every prefix is typed by the role it plays: a server owns a concrete ServePrefix, a client asks through a QueryPrefix that may name several origins. A server cannot be built on a wildcard because there is no value to build one from.

let serve = ServePrefix::new("demo/blobs")?;
let server = BlobServer::new(&session, serve.clone());
let manifest = server.register_file(BlobSpec::new("blob-1"), path).await?;
let handle = server.spawn().await?;

// Transfers are call builders: what to fetch and where it goes are
// positional; progress, cancellation and overwrite policy are optional.
let client = BlobClient::new(&session, QueryPrefix::from(&serve));
let stats = client
    .download_to(&DownloadRequest::pinned("blob-1", manifest.root), dest)
    .await?;

§Model (wire v3)

One queryable serves every blob under a key prefix:

queryable on:   <prefix>/**
manifest GET:   <prefix>/<id>/manifest                -> the Manifest (one reply)
slice GET:      <prefix>/<id>/**?ranges=<spec>       -> bao slice replies (any order)
slice reply:    <prefix>/<id>/slice/<index>

A download is a manifest GET followed by range-set slice GETs. The manifest-first step is not cosmetic — Zenoh does not order query replies — and each slice reply is a bao slice: the chunk’s bytes plus the parent hashes proving them against the manifest’s BLAKE3 root (see crate::wire and the verify module). Every reply is therefore independently verified before it touches disk, out of order, at 16 KiB granularity; there is no end-of-transfer hash pass. Memory stays O(chunk_size) regardless of blob size and arrival order.

The ranges parameter is a comma-separated list of half-open chunk-index spans ("0-5,9,12-20"; a bare k means k..k+1), which is how resume works: the client persists a chunk bitfield next to the .part file and re-queries exactly its holes. See keys::slice_selector / keys::parse_ranges.

§Three facts this design relies on

  1. Backpressure is automatic. Session::get defaults to CongestionControl::Block, and replies inherit the query’s congestion control, so chunk replies block (rather than drop) when the link backs up. We therefore set no congestion control explicitly on queries — the only setter is behind Zenoh’s internal feature, which this crate deliberately does not enable. Do not “fix” this by enabling internal. (Reply consolidation is a different knob: clients set ConsolidationMode::None so replies stream instead of being buffered until query finalization. Publications — the fanout tier — are not queries and do set Block explicitly, since their default is Drop.)
  2. Reply keys must intersect the query. Replies use ReplyKeyExpr::MatchingQuery by default, so the client must GET the wildcard <prefix>/<id>/** for the slice/<i> replies to be accepted. keys::slice_selector enforces the wildcard. The failure is not silent and not on the client: a bare-<id> GET makes the server’s reply() fail with a “does not intersect” error, so the diagnosis appears on the serving origin. (accept_replies(ReplyKeyExpr::Any) lifts the rule where a protocol genuinely needs disjoint reply keys.)
  3. Silence is how a server says “not mine”, and it is cheap. A query finalizes once every matching queryable has completed, and completing without replying is immediate — an unknown id resolves in about a millisecond, not on the query timeout. That is what lets several servers share one prefix, each owning a disjoint set of ids.
  4. Any peer can answer, so one bad reply must not be fatal. A queryable key range is open: replies that fail decoding, validation, id matching, or root pinning are skipped rather than aborting the query, so a hostile or stale responder cannot deny a fetch that an honest replica still answers.

Re-exports§

pub use zenoh::qos::Priority;
pub use bao_tree::io::sync::ReadAt;
pub use bao_tree::io::sync::Size;

Modules§

fanout
One-to-many fanout tier (optional fanout feature).
gc
Content-store lifecycle: tags + mark-and-sweep garbage collection.
keys
Zenoh key expressions for every plane this crate speaks, and the parsers that read them back.
seed
Local seeding (the desync model): satisfy a snapshot’s chunks from data already on this machine before touching the network.
wire
Wire encoding (v3) — postcard for every control message.

Structs§

BlobClient
Downloads blobs served by a crate::BlobServer under the same key prefix.
BlobClientBuilder
Builder for a BlobClient (see BlobClient::builder).
BlobId
A validated blob identifier: one non-empty, non-reserved key segment.
BlobProbe
What one origin said about a blob, from BlobClient::probe.
BlobServer
Serves registered blobs over a Zenoh queryable at <prefix>/**.
BlobServerBuilder
Builder for a BlobServer (see BlobServer::builder).
BlobSpec
Caller-side parameters for registering a blob: everything in the Manifest that isn’t computed from the bytes.
CancelToken
A shared cancellation token. Clone it freely; cancelling any clone signals every in-flight transfer holding one to stop as soon as it can — without waiting for a reply that may never come.
CdcParams
FastCDC (v2020) content-defined chunking parameters for Tier 2.
ChannelSink
A ProgressSink that forwards events onto a channel, from progress_channel.
ChunkProbe
What one origin holds, from StoreClient::probe.
ChunkRef
A reference to one content-addressed chunk.
DirStore
A filesystem ContentStore: <root>/blake3/<xx>/<hex> (two-hex-char fanout so huge stores don’t melt directory listings), atomic puts (unique temp file → fsync → rename → dir fsync), and optional verify-on-read.
Download
A configured download, awaited to run it.
DownloadRequest
What to download: a blob id, optionally with a pinned BLAKE3 root.
DownloadToWriter
A configured download into a writer, awaited to run it. See BlobClient::download_to_writer.
FileBlobSource
A BlobSource backed by a file on disk (e.g. a TTL’d report bundle).
Hash
A 32-byte content hash, rendered as lowercase hex on the wire and in keys.
HashParseError
Error parsing a Hash from a hex string.
Manifest
Describes a single blob: identity, size, chunking, and BLAKE3 root.
MaterializePolicy
What a snapshot is allowed to do to the destination directory.
MemoryBlobSource
A BlobSource serving a shared in-memory buffer (generated artifacts, tests). The buffer is behind an Arc, so opening is free and concurrent transfers share one allocation.
MemoryStore
An in-memory ContentStore (tests, ephemeral caches).
Publisher
Publishes content-addressed chunks into a storage under one prefix.
PushConfig
How a server accepts verified pushes (uploads), passed to BlobServerBuilder::accept_push.
QueryPrefix
A key prefix a client may query. Single-segment wildcards (*, $*…) are permitted; ** is not.
RetryPolicy
Retry/backoff policy for the download loop. An attempt is a slice query that made no progress; queries that verify at least one new chunk reset the budget.
ServePrefix
A key prefix a server or publisher may own: concrete, no wildcards.
ServerHandle
Handle to a spawned server task: keeps it running, stops it on demand.
SnapshotPublisher
A Publisher that also publishes tree indices — see Publisher::snapshots.
SourceFingerprint
A cheap identity snapshot of a BlobSource (see BlobSource::fingerprint).
Staged
Where a staged download landed, from BlobClient::download_staged.
StagedDownload
A configured staged download, awaited to run it. See BlobClient::download_staged.
StoreClient
Reads single chunks from a content-addressed store by their hash.
StoreClientBuilder
Builder for a StoreClient.
StoreKey
A 32-byte store encryption key. Generate it randomly, keep it outside the store directory, and give every store its own key.
TransferChunks
Fixed-size transfer-chunk arithmetic for one blob: (chunk_size, total_len) fully determine every chunk’s index, byte range, and count.
TransferStats
Statistics for one completed (or resumed-to-completion) transfer, returned by BlobClient::download_to and TreeClient::download_tree.
TreeClient
Downloads a tree snapshot. Stateless server; persistent client (the store).
TreeClientBuilder
Builder for a TreeClient (see TreeClient::builder).
TreeDownload
A configured snapshot download, awaited to run it. See TreeClient::download_tree.
TreeIndex
A directory snapshot: an ordered entry list + a digest over it (root_hash).
TreeServer
Serves a tree snapshot: an index queryable + a content-addressed chunk queryable, both backed by a ContentStore. Each query is served on its own task under an in-flight semaphore, so one slow client cannot stall the rest.
TreeServerBuilder
Builder for a TreeServer (see TreeServer::builder).
Upload
A configured upload, awaited to run it. See BlobClient::upload_file and BlobClient::upload_source.

Enums§

BlobError
Errors raised by the blob server and client.
ChunkCompression
Compression policy for produced chunk containers.
Entry
One entry in a tree snapshot (paths are relative, /-separated, UTF-8 — non-UTF-8 names are a build error, not silently mangled).
ErrorKind
Coarse classification of a BlobError, from BlobError::kind.
HashAlgo
The hash algorithm a message or key names.
Overwrite
What to do when the destination path already exists at completion time.
Progress
A progress event emitted by a download.
SettleCoverage
How much of a published snapshot the read-back phase actually checks.

Constants§

DEFAULT_CHUNK_SIZE
Default transfer chunk size: 256 KiB (16 bao verification groups, and four Zenoh fragments).
MAX_CHUNK_SIZE
Largest allowed transfer chunk size (4 MiB) — bounds per-reply RAM.
MIN_CHUNK_SIZE
Smallest allowed transfer chunk size (64 KiB).

Traits§

BlobSource
Opens a fresh reader over a blob’s bytes. Called once per registration (to hash) and once per query (to serve) — and, on the client side, once to hash and once to send per BlobClient::upload_source — so the source can be reopened many times. Opening is synchronous — implementations should be cheap (an open(2), not a download) and are always invoked on the blocking pool.
ContentStore
A local store of content-addressed chunks.
ProgressSink
A sink for Progress events. Implemented for any Fn(Progress) and for () (a no-op), so callers can pass a closure or nothing.
PushPolicy
Authorization hook for the push (upload) protocol: called for the initial offer and for every pushed slice, with the offered manifest and the opaque token the uploader attached to the query (if any).
ReadAtSize
A positional, sized, thread-safe byte source: what a BlobSource opens.

Functions§

build_tree
Build a TreeIndex for the directory at root, streaming every file through the content-defined chunker and putting each new chunk into store. Memory stays O(max chunk size) regardless of tree size.
build_tree_from
build_tree, reusing a previous snapshot’s work where the tree has not changed.
frame_chunk
Frame raw chunk bytes into a Tier-2 container (the inverse of unframe_chunk).
progress_channel
A ProgressSink and the receiver its events arrive on.
unframe_chunk
Decode a Tier-2 chunk container back to the chunk’s raw bytes.

Type Aliases§

ErrorCallback
Callback invoked with every error raised while serving a query.
Result
Convenience alias for fallible blob operations.