Expand description
An object-store registry that builds cloud stores on first use.
DataFusion resolves an object store by scheme+authority through the
runtime’s ObjectStoreRegistry. The default registry only knows what was
explicitly registered, which forces every code path that might touch
s3:// to remember to register the bucket first. That is a rule nothing
enforces, and the paths that forgot did not fail loudly:
- the stage builder planned on a context with no store, so
register_parqueterrored, the caller read that as “decline to stage”, and the query silently ran on ONE executor instead of the cluster; - the executor decoding a
dfplan:fragment failed outright with “No suitable object store found for s3://…”, because a serialized physical plan carries file paths but no way to register their backing store.
The executor case in particular cannot be fixed by registering ahead of time: the executor learns which buckets a plan touches only by decoding the plan, and the decode is what needs the store. So resolution has to be lazy.
This registry delegates to the default one and, on a miss for an object-store scheme it can construct, builds the store, caches it, and returns it. Explicit registration still wins, so a caller that wants specific credentials or an emulator endpoint can install its own store and this never overrides it.
Structs§
- Lazy
Cloud Object Store Registry - Registry that lazily constructs S3-compatible stores on first reference.