objstore-r

A small object-store abstraction shared by graph-r and link-r: one GAT trait, three backends, generation preconditions everywhere. It exists so both libraries can keep their on-disk formats byte-for-byte and still live on Google Cloud Storage: an object holds exactly the bytes of a local file, and moving a store between disk and a bucket is a plain copy.

uu-objstore-r = "0.2"                                      # LocalDir + Mem, no HTTP stack
uu-objstore-r = { version = "0.2", features = ["http"] }   # + authorised HTTPS transport
uu-objstore-r = { version = "0.2", features = ["gcs"] }    # + Google Cloud Storage (implies http)
uu-objstore-r = { version = "0.2", features = ["mirror"] } # + graph-r store mirroring
uu-objstore-r = { version = "0.2", features = ["gzip"] }   # + per-object Content-Encoding: gzip

use objstore_r::{Bytes, Cond, Mem, ObjectStore};

# futures::executor::block_on(async {
let s = Mem::default();
let g1 = s.put("HEAD", Bytes::from_static(b"v1"), Cond::CreateOnly).await?;
// Compare-and-swap: only succeeds if nobody wrote since we read g1.
let g2 = s.put("HEAD", Bytes::from_static(b"v2"), Cond::IfGeneration(g1)).await?;
assert!(s.put("HEAD", Bytes::from_static(b"v3"), Cond::IfGeneration(g1)).await.is_err());
# assert_ne!(g1, g2);
# Ok::<(), objstore_r::Error>(()) }).unwrap();

The trait

Futures are named associated types (GAT + TAIT), never boxed; the trait has no Send bound, and a backend's futures are Send whenever its state is, so both single-threaded and work-stealing executors work. Bodies are bytes::Bytes end to end: no copies between the network and the caller.

Keys are /-separated segments of [A-Za-z0-9._-], at most 1024 bytes, no empty segments, no segment starting or ending with a dot (dot-names are reserved for backend internals), no Windows device names. Every key is therefore a safe relative path on every OS and needs no escaping beyond percent-encoding for URLs. Anything else is Error::InvalidKey before any I/O.

Backends

• Mem — deterministic and in-process, with scripted fault injection (Fault::Fail, Fault::LostResponse: the write lands but the caller sees an error), for crash and retry tests of code built on top.
• LocalDir — a directory. Writes go to a unique temporary file, are fsynced and renamed into place, and the directory is fsynced. The generation is the file's mtime in nanoseconds, set explicitly and stepped by the filesystem's probed timestamp precision, so every write is distinguishable even on coarse-timestamp filesystems. Conditional writes are serialized by an exclusive File::lock, so CAS is atomic across threads and processes on one host. Paths never escape the root: symlinks inside the root are refused for directories and never followed for objects. A filesystem cannot hold both k and k/…; such writes fail with InvalidKey.
• Gcs<T: TokenProvider> (feature gcs) — the JSON API over reqwest/rustls with HTTP/2 connection reuse. ifGenerationMatch for conditions; media GETs with Range; single-request uploads up to 8 MiB and resumable uploads above; paginated listing. Retries 408/429/5xx and transport errors with capped, jittered exponential backoff — never 404 or 412. A lost response on a conditional write (it landed, the reply did not) is resolved by reading back and comparing bytes, so a retried CAS never reports a spurious conflict. Responses are size-capped, redirects are refused, and plain http is accepted only for a loopback emulator endpoint.

Content encoding (feature gzip)

Opt in per object: put_encoded(key, body, cond, ContentEncoding::Gzip) stores the body compressed and records the encoding — on GCS as the object's contentEncoding (the contentEncoding upload parameter), on LocalDir in a sidecar tree no key can name, in Mem beside the bytes. get decodes once into the returned Bytes, which is shared by reference count from there. On GCS every read asks for the stored bytes (Accept-Encoding: gzip, so the server never transcodes) and checks them against the object's CRC-32C before decoding. A ranged read of an encoded object is refused (Error::EncodedRange): a byte range of a gzip stream means nothing. Decoding stops at MAX_INFLATED_BYTES (1 GiB; per store with with_max_inflated), so a small object cannot inflate without bound (Error::TooLarge); a damaged or truncated stream is Error::Corrupt. Plain writes are unchanged, byte for byte. Encode only objects that are read whole.

Fenced publishing and graph-r mirroring

fenced is the single-writer publish protocol every snapshot publisher shares: create-only immutable objects, a sealed HEAD record swapped by compare-and-swap (a lost response is recognised by reading back; a racing writer gets Error::Fenced), and verified fetches. Feature mirror builds graph-r store mirroring on it (mirror::Mirror): byte-identical base and log files, incremental log shipping, leases, replicas refreshed with one metadata request when nothing changed. Mirror::encode_bases(Gzip) stores each immutable base compressed (log segments and HEAD stay plain); replicas decode it once into the file graph-r maps, and need no setting.

Other Google APIs (feature http)

http::Client<T: TokenProvider> implements the dependency-free transport::Transport trait: authorised HTTPS with the same TLS stack, tokens, retries and size caps as the GCS backend, for Vertex AI, BigQuery or any Google REST API. send returns a bounded body; open streams chunks (server-sent events) and cancels the request when the stream is dropped.

Credentials (features http / gcs)

token::Adc::discover() follows Application Default Credentials for the cases that need no private-key signing: GOOGLE_APPLICATION_CREDENTIALS or gcloud's well-known file (user credentials from gcloud auth application-default login), then the metadata server (Cloud Run, GCE, GKE workload identity). Service-account key files are unsupported on purpose. Applications with their own credential stack implement the TokenProvider GAT trait. Tokens are cached until a minute before expiry; their Debug output is redacted, and no error message ever contains one.

Python

pip install uu-objstore-r → import objstore_r: the same stores and guarantees from Python (see bindings/python/README.md).

import objstore_r
store = objstore_r.Store.gcs("my-bucket")            # ADC; .local(path) / .memory() alike
g = store.put("state/HEAD", b"v1", create_only=True)
store.put("state/HEAD", b"v2", if_generation=g)      # compare-and-swap
view = memoryview(store.get("state/HEAD"))           # zero-copy

Testing

scripts/ci-local.sh runs the whole CI locally (it starts fake-gcs-server in Docker when available). The live test runs only when you export OBJSTORE_LIVE_BUCKET yourself; it writes under a unique prefix and removes it.

Licence

See LICENSE.