Skip to main content

Module object_store_registry

Module object_store_registry 

Source
Expand description

An object-store registry that builds cloud stores on first use.

DataFusion resolves an object store by scheme+authority through the runtime’s ObjectStoreRegistry. The default registry only knows what was explicitly registered, which forces every code path that might touch s3:// to remember to register the bucket first. That is a rule nothing enforces, and the paths that forgot did not fail loudly:

  • the stage builder planned on a context with no store, so register_parquet errored, the caller read that as “decline to stage”, and the query silently ran on ONE executor instead of the cluster;
  • the executor decoding a dfplan: fragment failed outright with “No suitable object store found for s3://…”, because a serialized physical plan carries file paths but no way to register their backing store.

The executor case in particular cannot be fixed by registering ahead of time: the executor learns which buckets a plan touches only by decoding the plan, and the decode is what needs the store. So resolution has to be lazy.

This registry delegates to the default one and, on a miss for an object-store scheme it can construct, builds the store, caches it, and returns it. Explicit registration still wins, so a caller that wants specific credentials or an emulator endpoint can install its own store and this never overrides it.

Structs§

LazyCloudObjectStoreRegistry
Registry that lazily constructs S3-compatible stores on first reference.