Struct RuntimeEnv 

pub struct RuntimeEnv {
    pub memory_pool: Arc<dyn MemoryPool>,
    pub disk_manager: Arc<DiskManager>,
    pub cache_manager: Arc<CacheManager>,
    pub object_store_registry: Arc<dyn ObjectStoreRegistry>,
    pub parquet_encryption_factory_registry: Arc<EncryptionFactoryRegistry>,
}

Execution runtime environment that manages system resources such as memory, disk, cache and storage.

A RuntimeEnv can be created using RuntimeEnvBuilder and has the following resource management functionality:

• MemoryPool: Manage memory
• DiskManager: Manage temporary files on local disk
• CacheManager: Manage temporary cache data during the session lifetime
• ObjectStoreRegistry: Manage mapping URLs to object store instances

Example: Create default RuntimeEnv

let runtime_env = RuntimeEnv::default();

Example: Create a RuntimeEnv from RuntimeEnvBuilder with a new memory pool

// restrict to using at most 100MB of memory
let pool_size = 100 * 1024 * 1024;
let runtime_env = RuntimeEnvBuilder::new()
  .with_memory_pool(Arc::new(GreedyMemoryPool::new(pool_size)))
  .build()
  .unwrap();

Fields

memory_pool: Arc<dyn MemoryPool>

Runtime memory management

disk_manager: Arc<DiskManager>

Manage temporary files during query execution

cache_manager: Arc<CacheManager>

Manage temporary cache during query execution

object_store_registry: Arc<dyn ObjectStoreRegistry>

Object Store Registry

parquet_encryption_factory_registry: Arc<EncryptionFactoryRegistry>

Available on crate feature parquet_encryption only.

Parquet encryption factory registry

Implementations

impl RuntimeEnv

pub fn register_object_store( &self, url: &Url, object_store: Arc<dyn ObjectStore>, ) -> Option<Arc<dyn ObjectStore>>

Registers a custom ObjectStore to be used with a specific url. This allows DataFusion to create external tables from urls that do not have built in support such as hdfs://namenode:port/....

Returns the ObjectStore previously registered for this scheme, if any.

See ObjectStoreRegistry for more details

Example: Register local file system object store

let url = Url::try_from("file://").unwrap();
let object_store = object_store::local::LocalFileSystem::new();
// register the object store with the runtime environment
runtime_env.register_object_store(&url, Arc::new(object_store));

Example: Register remote URL object store like Github

// create a new object store via object_store::http::HttpBuilder;
let base_url = Url::parse("https://github.com").unwrap();
// (note this example can't depend on the http feature)
// let http_store = HttpBuilder::new()
//    .with_url(base_url.clone())
//    .build()
//    .unwrap();
// register the object store with the runtime environment
runtime_env.register_object_store(&base_url, Arc::new(http_store));

pub fn object_store(&self, url: impl AsRef<Url>) -> Result<Arc<dyn ObjectStore>>

Retrieves a ObjectStore instance for a url by consulting the registry. See ObjectStoreRegistry::get_store for more details.

pub fn register_parquet_encryption_factory( &self, id: &str, encryption_factory: Arc<dyn EncryptionFactory>, ) -> Option<Arc<dyn EncryptionFactory>>

Available on crate feature parquet_encryption only.

Register an EncryptionFactory with an associated identifier that can be later used to configure encryption when reading or writing Parquet. If an encryption factory with the same identifier was already registered, it is replaced and returned.

pub fn parquet_encryption_factory( &self, id: &str, ) -> Result<Arc<dyn EncryptionFactory>>

Available on crate feature parquet_encryption only.

Retrieve an EncryptionFactory by its identifier

Trait Implementations

impl Clone for RuntimeEnv

fn clone(&self) -> RuntimeEnv

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for RuntimeEnv

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for RuntimeEnv

fn default() -> Self

Returns the “default value” for a type.