Struct DataEngine 

pub struct DataEngine {
    pub ctx: SessionContext,
    pub rt: Arc<Runtime>,
}

Synchronous wrapper around DataFusion’s async SessionContext.

Fields

ctx: SessionContext

rt: Arc<Runtime>

Implementations

impl DataEngine

pub fn new() -> Self

Create a new DataEngine with default configuration. Backward-compatible with existing code.

pub fn with_config(config: DataEngineConfig) -> Self

Create a new DataEngine with custom configuration.

pub fn collect(&self, df: DataFrame) -> Result<Vec<RecordBatch>, String>

Execute a DataFusion DataFrame and collect results synchronously.

pub fn format_batches(batches: &[RecordBatch]) -> Result<String, String>

Format collected batches as a pretty table string.

pub fn register_batch( &self, name: &str, batch: RecordBatch, ) -> Result<(), String>

Register a RecordBatch as a named table in the session.

pub fn register_batches( &self, name: &str, schema: Arc<Schema>, batches: Vec<RecordBatch>, ) -> Result<(), String>

Register multiple RecordBatches as a single named table. Each batch becomes a separate partition, enabling DataFusion parallelism.

pub fn sql(&self, query: &str) -> Result<DataFrame, String>

Run a SQL query and return results.

pub fn session_ctx(&self) -> &SessionContext

Get the underlying session context for DataFusion operations.

pub fn runtime(&self) -> &Arc<Runtime>

Get a reference to the tokio Runtime.

impl DataEngine

pub fn read_csv(&self, path: &str) -> Result<DataFrame, String>

Read a CSV file into a DataFusion DataFrame. Supports local paths and s3:// URLs (after register_s3).

pub fn read_parquet(&self, path: &str) -> Result<DataFrame, String>

Read a Parquet file into a DataFusion DataFrame. Supports local paths and s3:// URLs (after register_s3).

pub fn write_csv(&self, df: DataFrame, path: &str) -> Result<(), String>

Write a DataFrame to a CSV file.

pub fn write_parquet(&self, df: DataFrame, path: &str) -> Result<(), String>

Write a DataFrame to a Parquet file.

impl DataEngine

pub fn read_postgres( &self, conn_str: &str, table_name: &str, ) -> Result<DataFrame, String>

Read a PostgreSQL table into a DataFusion DataFrame.

pub fn query_postgres( &self, conn_str: &str, query: &str, register_as: &str, ) -> Result<DataFrame, String>

Execute a custom SQL query against PostgreSQL and return a DataFusion DataFrame. Fetches 1M rows per cursor round trip, then subdivides into 100K-row RecordBatches for DataFusion parallelism. Falls back to a simple query if cursors are not supported (e.g., PgBouncer, some cloud proxies, or restricted permissions).

impl DataEngine

pub fn fill_null( &self, df: DataFrame, column: &str, strategy: &str, fill_value: Option<f64>, ) -> Result<DataFrame, String>

Fill null values in a column using a strategy. strategy: “value” (use fill_value), “mean”, “zero”

pub fn drop_null( &self, df: DataFrame, column: &str, ) -> Result<DataFrame, String>

Drop rows where a column is null.

pub fn dedup( &self, df: DataFrame, columns: &[String], ) -> Result<DataFrame, String>

Remove duplicate rows based on specified columns.

pub fn clamp( &self, df: DataFrame, column: &str, min_val: f64, max_val: f64, ) -> Result<DataFrame, String>

Clamp values in a column to [min_val, max_val].

pub fn data_profile(&self, df: DataFrame) -> Result<DataFrame, String>

Generate a statistical profile of all numeric columns. Returns a table with: column_name, count, null_count, null_rate, min, max, mean, stddev

pub fn row_count(&self, df: DataFrame) -> Result<i64, String>

Get the row count of a DataFrame.

pub fn null_rate(&self, df: DataFrame, column: &str) -> Result<f64, String>

Get the null rate of a column (0.0 to 1.0).

pub fn is_unique(&self, df: DataFrame, column: &str) -> Result<bool, String>

Check if a column’s non-null values are all unique.

impl DataEngine

pub fn read_redshift( &self, conn_str: &str, query: &str, ) -> Result<DataFrame, String>

Read from Amazon Redshift using a connection string and SQL query. Delegates to the PostgreSQL connector (Redshift is PG wire-compatible). Ensures SSL mode is set for Redshift cluster connections.

Trait Implementations

impl Default for DataEngine

fn default() -> Self

Returns the “default value” for a type.