1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
//! DataFusion OLAP backend for the Rhei HTAP engine.
//!
//! # Architecture position
//!
//! `rhei-datafusion` is the **default OLAP backend** in Rhei (enabled by the
//! `datafusion-backend` workspace feature, which is on by default). It
//! implements [`rhei_core::OlapEngine`] for Apache DataFusion 53 and is used
//! by the [`rhei`](https://docs.rs/rhei) facade's `OlapBackend` enum.
//!
//! # Storage modes
//!
//! The engine is parameterised by a [`StorageMode`] chosen at construction time:
//!
//! | Variant | Durability | Notes |
//! |---------|------------|-------|
//! | [`StorageMode::InMemory`] | None — lost on shutdown | Fastest; ideal for tests and ephemeral workloads |
//! | [`StorageMode::ArrowIpc`] | Durable — `.arrow` files | Zero serialisation overhead; no compression |
//! | [`StorageMode::Parquet`] | Durable — `.parquet` files | Columnar compression + predicate pushdown; slower writes |
//! | `StorageMode::S3Parquet` | Cloud (Amazon S3) | Available with the `cloud-storage` feature only |
//! | `StorageMode::GcsParquet` | Cloud (Google Cloud Storage) | Available with the `cloud-storage` feature only |
//!
//! # Streaming queries
//!
//! [`rhei_core::OlapEngine::query_stream`] on [`DataFusionEngine`] returns a [`rhei_core::RecordBatchBoxStream`]
//! backed by DataFusion's own `DataFrame::execute_stream()`. A thin
//! `StreamAdapter` newtype adapts `SendableRecordBatchStream` to the common
//! `RecordBatchBoxStream` type without buffering — results flow row-batch by
//! row-batch directly to the caller. This is a key advantage over the DuckDB
//! backend, which buffers the full result set before streaming.
//!
//! # No transactions
//!
//! DataFusion does not support SQL transactions.
//! [`rhei_core::OlapEngine::supports_transactions`] returns `false` for this
//! backend. The Rhei sync engine handles partial-failure recovery via CDC
//! sequence numbers instead of relying on `BEGIN`/`COMMIT`.
//!
//! # DML strategy
//!
//! DataFusion's `MemTable` and `ListingTable` are read-only. DML
//! (INSERT / UPDATE / DELETE) is implemented by mutating the engine's own
//! table store and re-registering tables with the `SessionContext`. SQL
//! statements are parsed with `sqlparser-rs` (SQLite dialect) so that the sync
//! engine's generated SQL is handled correctly.
//!
//! # Cloud storage
//!
//! S3 and GCS backends are gated behind the `cloud-storage` workspace feature.
//! When enabled, `object_store` (with `aws` and `gcp` sub-features) is pulled
//! in. Credentials are resolved from the environment at engine construction
//! time following `object_store` conventions (e.g. `AWS_ACCESS_KEY_ID` /
//! `GOOGLE_APPLICATION_CREDENTIALS`).
//!
//! # Feature flags
//!
//! - `cloud-storage` — enables `StorageMode::S3Parquet` and
//! `StorageMode::GcsParquet` via the `object_store` crate.
pub use ;
pub use DfOlapError;
pub use StorageMode;