Struct DataFusionEngine 

pub struct DataFusionEngine { /* private fields */ }

DataFusion-backed OLAP engine.

Supports pluggable storage via StorageMode:

• InMemory: stores Arrow data in memory, registers as MemTable
• ArrowIpc/Parquet: stores data as files on disk, registers as ListingTable
• S3Parquet/GcsParquet (cloud-storage feature): Parquet files on object storage, registered as ListingTable once the object store is wired into the SessionContext

Implementations

impl DataFusionEngine

pub fn with_storage(mode: StorageMode) -> Result<DataFusionEngine, DfOlapError>

Create a new DataFusion engine with the given storage mode.

For file-backed modes, creates the base directory if it doesn’t exist and scans for existing data files to initialize the file counter (avoiding overwrites on restart).

For cloud modes (S3Parquet / GcsParquet, requires the cloud-storage feature), registers the appropriate object_store implementation with the DataFusion SessionContext. Credentials are resolved from the environment at construction time.

pub fn new() -> DataFusionEngine

Create a new in-memory DataFusion engine (default).

pub fn storage_mode(&self) -> &StorageMode

Returns the storage mode of this engine.

Trait Implementations

impl Default for DataFusionEngine

fn default() -> DataFusionEngine

Returns the “default value” for a type.

impl OlapEngine for DataFusionEngine

type Error = DfOlapError

Engine-specific error type returned by all fallible methods.

async fn query( &self, sql: &str, ) -> Result<Vec<RecordBatch>, <DataFusionEngine as OlapEngine>::Error>

Execute an analytical SQL query and return results as Arrow RecordBatches buffered in memory.

async fn query_stream( &self, sql: &str, ) -> Result<Pin<Box<dyn Stream<Item = Result<RecordBatch, Box<dyn Error + Send + Sync>>> + Send>>, <DataFusionEngine as OlapEngine>::Error>

Execute a SQL query and return a lazy RecordBatchBoxStream.

async fn execute( &self, sql: &str, ) -> Result<u64, <DataFusionEngine as OlapEngine>::Error>

Execute a DDL or DML statement without returning rows.

async fn load_arrow( &self, table: &str, batches: &[RecordBatch], ) -> Result<u64, <DataFusionEngine as OlapEngine>::Error>

Bulk-load Arrow RecordBatches into table.

async fn create_table( &self, table_name: &str, schema: &Arc<Schema>, _primary_key: &[String], ) -> Result<(), <DataFusionEngine as OlapEngine>::Error>

Create a table from an Arrow schema with an optional primary key.

async fn table_exists( &self, table_name: &str, ) -> Result<bool, <DataFusionEngine as OlapEngine>::Error>

Return true if table_name exists in the OLAP catalog.

async fn add_column( &self, table_name: &str, column_name: &str, data_type: &DataType, ) -> Result<(), <DataFusionEngine as OlapEngine>::Error>

Add a nullable column to an existing table.

async fn drop_column( &self, table_name: &str, column_name: &str, ) -> Result<(), <DataFusionEngine as OlapEngine>::Error>

Remove a column from an existing table.

fn supports_transactions(&self) -> bool

Returns true if this backend supports ACID transactions via BEGIN / COMMIT / ROLLBACK.