Module stores 

Object stores, registered the first time a query names a URL.

DataFusion only knows the local filesystem until a store is registered for a URL’s scheme://authority. As datafusion-cli does, the engine registers one lazily: when a query first reads (or COPYs to) s3://bucket/…, a store for s3://bucket is built from the standard environment and kept for the session.

Credentials come from each backend’s environment variables and default credential chain (object_store’s from_env): AWS_* (plus web identity, container and instance credentials), GOOGLE_* (plus application default credentials), and AZURE_*. A public S3 bucket can be read with AWS_SKIP_SIGNATURE=true.

With the aws-profile feature (on by default in the ursa binary), S3 also reads the AWS profile files, ~/.aws/config and ~/.aws/credentials (or AWS_CONFIG_FILE and AWS_SHARED_CREDENTIALS_FILE), for the profile AWS_PROFILE names, or default. It comes after the environment’s keys and before web identity, container and instance credentials, the order the AWS CLI uses. A profile can hold static keys, an SSO login, a role to assume or a credential_process; aws-config resolves each as the AWS SDKs do. See [aws_profile].

Functions

build_object_store

A store for url’s scheme and bucket, configured from the environment.

ensure_object_store

Make sure runtime has an object store for url, building and registering one if this is the first time the session sees its scheme://authority. Local paths need none.

ensure_object_store_for

ensure_object_store for a location written in SQL (a COPY … TO target or an external table’s LOCATION). A local path needs no store.