StorageOptionsProvider

Trait StorageOptionsProvider 

Source
pub trait StorageOptionsProvider:
    Send
    + Sync
    + Debug {
    // Required methods
    fn fetch_storage_options<'life0, 'async_trait>(
        &'life0 self,
    ) -> Pin<Box<dyn Future<Output = Result<Option<HashMap<String, String>>>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait;
    fn provider_id(&self) -> String;
}
Expand description

Trait for providing storage options with expiration tracking

Implementations can fetch storage options from various sources (namespace servers, secret managers, etc.) and are usable from Python/Java.

§Current Use Cases

  • Temporary Credentials: Fetch short-lived AWS temporary credentials that expire after a set time period, with automatic refresh before expiration

§Future Possible Use Cases

  • Dynamic Storage Location Resolution: Resolve logical names to actual storage locations (bucket aliases, S3 Access Points, region-specific endpoints) that may change based on region, tier, data migration, or failover scenarios
  • Runtime S3 Tags Assignment: Inject cost allocation tags, security labels, or compliance metadata into S3 requests based on the current execution context (user, application, workspace, etc.)
  • Dynamic Endpoint Configuration: Update storage endpoints for disaster recovery, A/B testing, or gradual migration scenarios
  • Just-in-time Permission Elevation: Request elevated permissions only when needed for sensitive operations, then immediately revoke them
  • Secret Manager Integration: Fetch encryption keys from AWS Secrets Manager, Azure Key Vault, or Google Secret Manager with automatic rotation
  • OIDC/SAML Federation: Integrate with identity providers to obtain storage credentials based on user identity and group membership

§Equality and Hashing

Implementations must provide provider_id() which returns a unique identifier for equality and hashing purposes. Two providers with the same ID are considered equal and will share the same cached ObjectStore in the registry.

Required Methods§

Source

fn fetch_storage_options<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Result<Option<HashMap<String, String>>>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait,

Fetch fresh storage options

Returns None if no storage options are available, or Some(HashMap) with the options. If the EXPIRES_AT_MILLIS_KEY key is present in the HashMap, it should contain the epoch time in milliseconds when the options expire, and credentials will automatically refresh before expiration. If EXPIRES_AT_MILLIS_KEY is not provided, the options are considered to never expire.

Source

fn provider_id(&self) -> String

Return a human-readable unique identifier for this provider instance

This is used for equality comparison and hashing in the object store registry. Two providers with the same ID will be treated as equal and share the same cached ObjectStore.

The ID should be human-readable for debugging and logging purposes. For example: "namespace[dir(root=/data)],table[db$schema$table1]"

The ID should uniquely identify the provider’s configuration.

Implementors§