Skip to main content

Module sync

Module sync 

Source
Expand description

Streaming store-to-store snapshot copy.

sync_snapshot copies ONE snapshot — its manifest plus every raw object it references — directly from a source StreamStore to a destination StreamStore, through memory only. There is no local filesystem staging: the function signature deliberately takes no Path anywhere, so a blob read out of the source can only ever flow into the destination (it never touches scratch/cache on disk).

§Sync methods, rayon threads — not the async driver

StreamStore’s methods are synchronous: the network backends drive their async SDK calls on an internal runtime via block_on. Driving them from the async run_concurrent / RateLimiter would nest one tokio runtime inside another and panic. So this orchestrator parallelizes object copies across a rayon ThreadPool sized to TransferConfig::concurrency — exactly the pattern FileStore::parallel_copy. Rayon workers are plain OS threads, so each one may safely call the block_on-ing sync get_object/put_object. Bandwidth is throttled by the synchronous BlockingRateLimiter (one shared bucket via Arc), never the async RateLimiter.

§Invariants

  • Skip-present / incremental: an object the destination already has_object is not re-copied.
  • Manifest-last / all-or-nothing: the destination manifest is written only after every referenced object has landed. On the first object error the copy stops and NO manifest is written, so a destination manifest always implies its objects are present (mirroring push).
  • Verified: every blob is BLAKE3-verified by the underlying StreamStore on both read and write, and the source manifest is verified to hash to id by get_manifest.

Structs§

SyncReport
Outcome of a sync_snapshot call.

Functions§

sync_snapshot
Copies one snapshot’s manifest + raw objects directly from from to to, through memory only (no local filesystem staging).