Skip to main content

Module engine

Module engine 

Source
Expand description

DataFusion-backed OLAP engine.

Uses Apache DataFusion as the query engine with pluggable storage:

  • InMemory: Vec<RecordBatch> in a HashMap (default, lost on shutdown)
  • ArrowIpc: persistent .arrow files per table
  • Parquet: persistent .parquet files per table

§DML Strategy

DataFusion’s MemTable / ListingTable don’t support INSERT/UPDATE/DELETE DML. Instead, we maintain our own table store (in-memory or file-backed) and re-register tables with DataFusion after mutations. For queries, DataFusion’s SQL engine executes against the registered tables.

Structs§

DataFusionEngine
DataFusion-backed OLAP engine.
SharedDataFusionEngine
A cheaply-cloneable, Arc-wrapped DataFusionEngine that implements rhei_core::OlapEngine.