Struct RuntimeEnv

pub struct RuntimeEnv {
    pub memory_pool: Arc<dyn MemoryPool>,
    pub disk_manager: Arc<DiskManager>,
    pub cache_manager: Arc<CacheManager>,
    pub object_store_registry: Arc<dyn ObjectStoreRegistry>,
}

Execution runtime environment that manages system resources such as memory, disk, cache and storage.

A RuntimeEnv can be created using RuntimeEnvBuilder and has the following resource management functionality:

• MemoryPool: Manage memory
• DiskManager: Manage temporary files on local disk
• CacheManager: Manage temporary cache data during the session lifetime
• ObjectStoreRegistry: Manage mapping URLs to object store instances

Example: Create default RuntimeEnv

let runtime_env = RuntimeEnv::default();

Example: Create a RuntimeEnv from RuntimeEnvBuilder with a new memory pool

// restrict to using at most 100MB of memory
let pool_size = 100 * 1024 * 1024;
let runtime_env = RuntimeEnvBuilder::new()
  .with_memory_pool(Arc::new(GreedyMemoryPool::new(pool_size)))
  .build()
  .unwrap();

Fields

memory_pool: Arc<dyn MemoryPool>

Runtime memory management

disk_manager: Arc<DiskManager>

Manage temporary files during query execution

cache_manager: Arc<CacheManager>

Manage temporary cache during query execution

object_store_registry: Arc<dyn ObjectStoreRegistry>

Object Store Registry

Implementations

impl RuntimeEnv

pub fn new(config: RuntimeConfig) -> Result<Self>

👎Deprecated since 43.0.0: please use RuntimeEnvBuilder instead

pub fn try_new(config: RuntimeConfig) -> Result<Self>

👎Deprecated since 44.0.0: please use RuntimeEnvBuilder instead

Create env based on configuration

pub fn register_object_store( &self, url: &Url, object_store: Arc<dyn ObjectStore>, ) -> Option<Arc<dyn ObjectStore>>

Registers a custom ObjectStore to be used with a specific url. This allows DataFusion to create external tables from urls that do not have built in support such as hdfs://namenode:port/....

Returns the ObjectStore previously registered for this scheme, if any.

See ObjectStoreRegistry for more details

Example: Register local file system object store

let url = Url::try_from("file://").unwrap();
let object_store = object_store::local::LocalFileSystem::new();
// register the object store with the runtime environment
runtime_env.register_object_store(&url, Arc::new(object_store));

Example: Register remote URL object store like Github

// create a new object store via object_store::http::HttpBuilder;
let base_url = Url::parse("https://github.com").unwrap();
// (note this example can't depend on the http feature)
// let http_store = HttpBuilder::new()
//    .with_url(base_url.clone())
//    .build()
//    .unwrap();
// register the object store with the runtime environment
runtime_env.register_object_store(&base_url, Arc::new(http_store));

pub fn object_store(&self, url: impl AsRef<Url>) -> Result<Arc<dyn ObjectStore>>

Retrieves a ObjectStore instance for a url by consulting the registry. See ObjectStoreRegistry::get_store for more details.

Trait Implementations

impl Clone for RuntimeEnv

fn clone(&self) -> RuntimeEnv

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for RuntimeEnv

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for RuntimeEnv

fn default() -> Self

Returns the “default value” for a type.