Skip to main content

Crate snapdir_stores

Crate snapdir_stores 

Source
Expand description

snapdir stores library.

Storage backends for snapdir snapshots plus the store-routing and external-store shim that implement snapdir’s store dispatch:

  • FileStore — the in-process file:// backend.
  • S3Store — the native AWS-SDK s3:// backend (ring rustls).
  • B2Store — the native AWS-SDK b2:// backend, pointed at Backblaze B2’s S3-compatible endpoint (wraps S3Store with a custom endpoint).
  • GcsStore — the native google-cloud-storage gs:// backend (ring rustls; ADC credential chain).
  • router — scheme → adapter resolution, including the hardcoded gs://gcs special case for the Google Cloud Storage adapter.
  • shim (ExternalStore) — the emit-command shim that dispatches third-party snapdir-<name>-store binaries via the documented get-manifest-command / get-fetch-files-command / get-push-command contract.
  • transfer (TransferConfig, RateLimiter, run_concurrent) — the concurrency + bandwidth-limiting foundation each store carries via a TransferConfig for the (later) concurrent transfer loops.
  • adaptive (AdaptiveGate, AdaptiveController) — pure, injectable adaptive control: a resizable concurrency permit pool (async + blocking) plus a deterministic slow-start/AIMD controller that turns injected op samples + system metrics into a concurrency limit and target byte-rate (wiring into the live transfer loops is a later gate).
  • stream (StreamStore) — object/manifest-level, content-addressed, verified read/write primitives (the foundation for store-to-store sync), implemented for FileStore, S3Store, GcsStore, and B2Store.
  • sync (sync_snapshot, SyncReport) — streaming store-to-store snapshot copy: walks a source manifest and copies its raw objects source → dest through memory only (no local filesystem staging), parallelized across a rayon pool and throttled by a BlockingRateLimiter; writes the manifest last (all-or-nothing).

Re-exports§

pub use adaptive::p95_object_size;
pub use adaptive::AdaptiveController;
pub use adaptive::AdaptiveGate;
pub use adaptive::AdaptivePolicy;
pub use adaptive::ControllerDriver;
pub use adaptive::Decision;
pub use adaptive::OpResult;
pub use adaptive::OpSample;
pub use b2_store::B2Store;
pub use file_store::FileStore;
pub use gcs_store::GcsLocation;
pub use gcs_store::GcsStore;
pub use router::resolve_adapter;
pub use router::Adapter;
pub use router::RouteError;
pub use s3_store::S3Location;
pub use s3_store::S3Store;
pub use shim::ExternalStore;
pub use stream::StreamStore;
pub use sync::sync_snapshot;
pub use sync::SyncReport;
pub use transfer::classify_error;
pub use transfer::run_adaptive;
pub use transfer::run_concurrent;
pub use transfer::AdaptivePolicy as TransferAdaptivePolicy;
pub use transfer::BlockingRateLimiter;
pub use transfer::RateLimiter;
pub use transfer::TransferConfig;

Modules§

adaptive
Adaptive concurrency + throughput control (pure control logic).
b2_store
B2Store: the b2:// storage backend, backed by Backblaze B2’s S3-compatible endpoint via the native AWS SDK.
file_store
FileStore: the file:// storage backend.
gcs_store
GcsStore: the gs:// storage backend, backed by the native google-cloud-storage SDK.
router
Store routing: scheme → adapter / binary-name resolution.
s3_store
S3Store: the s3:// storage backend, backed by the native AWS SDK.
shim
External-store emit-command shim.
stream
Object/manifest-level, content-addressed streaming over a Store.
sync
Streaming store-to-store snapshot copy.
transfer
Transfer configuration, rate limiting, and bounded-concurrency driver.