1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
//! DataFusion OLAP backend for the Rhei HTAP engine.
//!
//! # Architecture position
//!
//! `rhei-datafusion` is the **default OLAP backend** in Rhei (enabled by the
//! `datafusion-backend` workspace feature, which is on by default). It
//! implements [`rhei_core::OlapEngine`] for Apache DataFusion 53 and is used
//! by the [`rhei`](https://docs.rs/rhei) facade's `OlapBackend` enum.
//!
//! # Storage modes
//!
//! The engine is parameterised by a [`StorageMode`] chosen at construction time:
//!
//! | Variant | Durability | Notes |
//! |---------|------------|-------|
//! | [`StorageMode::InMemory`] | None — lost on shutdown | Fastest; ideal for tests and ephemeral workloads |
//! | [`StorageMode::Vortex`] | Durable — `.vortex` files (local or S3) | Auto-detects local vs S3 from URL scheme |
//!
//! # Streaming queries
//!
//! [`rhei_core::OlapEngine::query_stream`] on [`DataFusionEngine`] returns a
//! [`rhei_core::RecordBatchBoxStream`] backed by DataFusion's own
//! `DataFrame::execute_stream()`. A thin `StreamAdapter` newtype adapts
//! `SendableRecordBatchStream` to the common `RecordBatchBoxStream` type
//! without buffering — results flow row-batch by row-batch directly to the
//! caller.
//!
//! # No transactions
//!
//! DataFusion does not support SQL transactions.
//! [`rhei_core::OlapEngine::supports_transactions`] returns `false` for this
//! backend. The Rhei sync engine handles partial-failure recovery via CDC
//! sequence numbers instead of relying on `BEGIN`/`COMMIT`.
//!
//! # DML strategy
//!
//! `InMemory`: mutations update an in-memory `HashMap` of `Vec<RecordBatch>`
//! and re-register a `MemTable` with DataFusion after each change.
//!
//! `Vortex`: INSERT routes through DataFusion's `VortexFormatFactory` sink (SQL
//! `INSERT INTO … SELECT * FROM tmp`). UPDATE/DELETE use a read-modify-write
//! cycle: read all data via `SELECT *`, apply mutations in-memory, clear the
//! table directory, and re-insert.
//!
//! # Cloud storage
//!
//! S3-compatible backends are gated behind the `cloud-storage` workspace feature.
//! When enabled, `object_store` (with `aws` sub-feature) is pulled in.
//! Credentials are resolved from the environment at engine construction time
//! following `object_store` conventions (e.g. `AWS_ACCESS_KEY_ID`).
//! S3-compatible services (MinIO, Cloudflare R2, Wasabi, Ceph RGW) work via
//! `AWS_ENDPOINT_URL`.
//!
//! # Feature flags
//!
//! - `cloud-storage` — enables `StorageMode::Vortex` with `s3://` URLs via
//! the `object_store` crate.
pub use ;
pub use DfOlapError;
pub use StorageMode;