Crate delta_funnel

Expand description

Read Delta Lake tables, query them with DataFusion SQL, preview results, and load selected outputs into Microsoft SQL Server with high-throughput native TDS bulk writes, without ODBC or JDBC.

Delta Funnel provides a session-oriented workflow for:

registering one or more Delta Lake sources;
building lazy tables from DataFusion SQL;
executing bounded previews without contacting SQL Server;
validating output plans with dry runs;
streaming single or multiple outputs through native TDS bulk writes; and
collecting structured source, validation, cache, and write reports.

§Quickstart

DeltaFunnelSession owns registered sources and lazy tables. Synchronous applications can use DeltaFunnelRuntime to run the session’s async query actions.

use delta_funnel::{
    DeltaFunnelRuntime, DeltaFunnelSession, DeltaSourceConfig, LoadMode,
    MssqlConnectionConfig, MssqlOutputTarget, MssqlTargetConfig,
    MssqlTargetTable, OutputWritePlan, RunMode, SessionOptions,
};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let ado_connection_string = concat!(
        "server=tcp:localhost,1433;",
        "database=warehouse;",
        "User ID=etl_user;",
        "Password=REPLACE_ME;",
        "encrypt=true;",
        "TrustServerCertificate=yes",
    );

    let default_connection = MssqlConnectionConfig::new(ado_connection_string)?
        .with_display_label("warehouse");
    let mut session = DeltaFunnelSession::new(
        SessionOptions::new().with_default_mssql_connection(default_connection),
    )?;
    let runtime = DeltaFunnelRuntime::new()?;

    // Register the Delta table as "orders" so SQL can reference it.
    let _orders = runtime.delta_lake(
        &mut session,
        DeltaSourceConfig::new("orders", "file:///path/to/orders-delta"),
    )?;

    // Build a lazy DataFusion SQL query. No rows are read yet.
    let daily_orders = runtime.table_from_sql(
        &mut session,
        r#"
        select customer_id, order_date, total_amount
        from orders
        where order_date >= date '2026-01-01'
        "#,
    )?;

    // Preview executes the DataFusion query with a limit.
    let preview = runtime.preview_table(&session, &daily_orders, 20)?;
    println!("{}", preview.text());

    // Write executes the query and loads the result into SQL Server.
    let target = MssqlTargetConfig::new(MssqlTargetTable::new("dbo", "daily_orders")?)
        .with_load_mode(LoadMode::CreateAndLoad);
    let output = OutputWritePlan::new(
        daily_orders,
        MssqlOutputTarget::new("daily_orders", target, RunMode::Execute),
    );
    let report = runtime.write_to_mssql(&session, &output)?;

    println!("wrote output {}", report.output_name());
    Ok(())
}

Previewing executes the DataFusion query with the requested row limit. It does not contact SQL Server or write rows.

§Dry runs and multiple outputs

Build an OutputWritePlan with RunMode::DryRun and pass it to DeltaFunnelRuntime::dry_run_to_mssql to validate an output plan without writing rows. DeltaFunnelRuntime::write_all can execute multiple outputs while caching shared lazy-query dependencies when beneficial.

See the Rust quickstart for a complete write example and the project documentation for installation, SQL Server, and workflow guidance.

Re-exports§

pub use error::DeltaFunnelError;
pub use error::SqlTablePhase;

Modules§

error: Shared error pattern for DeltaFunnel.

Structs§

BatchHandoffOutcome: Successful result for one completed query-output handoff.
BatchHandoffStats: Per-output batch handoff counters.
DeltaDataFusionMetricsSnapshot: Stable Delta Funnel view of one DataFusion Delta scan’s metrics.
DeltaFunnelRuntime: Blocking runtime boundary for high-level Delta Funnel session actions.
DeltaFunnelSession: Rust backing session for lazy query-load workflows.
DeltaProtocolReport: Protocol details for one named Delta source.
DeltaProviderSchedulingReport: Configured provider scheduling options included with a Delta source report.
DeltaReadMetricsSnapshot: Stable Delta Funnel view of one Delta reader scan’s metrics.
DeltaSourceConfig: Caller-provided configuration for one named Delta source.
DeltaSourceReport: Source-level readiness report for one registered Delta source.
LazyTable: Lazy table identity owned by a query-load session.
MssqlBatchShapingReport: Per-output query stream and identity batch-shaping counters.
MssqlConnectionConfig: A SQL Server connection configuration with secret-bearing material hidden from display.
MssqlConnectionSummary: Redacted SQL Server connection context suitable for reports and errors.
MssqlDdlPlan: Planned SQL Server DDL artifacts for one selected output.
MssqlDryRunOutputFieldReport: Output schema field included in an MSSQL dry-run report.
MssqlDryRunOutputReport: Dry-run planning report for one selected MSSQL output.
MssqlDryRunSqlIdentityReport: Redacted SQL identity included in an MSSQL dry-run output report.
MssqlDryRunWorkflowReport: Dry-run planning report for a multi-output MSSQL workflow.
MssqlLifecyclePlan: Planned SQL Server table lifecycle behavior for one selected output.
MssqlOutputBatchValidationReport: Redacted report for a successful planned-output schema validation.
MssqlOutputFieldReport: Output schema field included in an MSSQL execute report.
MssqlOutputTarget: MSSQL output target selected from a lazy table.
MssqlOutputWriteJob: One deferred SQL Server output write job.
MssqlPreparedTarget: Prepared SQL Server target identity and redacted report.
MssqlPreparedTargetReport: Redacted report for a prepared SQL Server target.
MssqlSchemaDiagnostic: DeltaFunnel-owned schema planning diagnostic entry.
MssqlSchemaDiagnosticField: Field context for a DeltaFunnel-owned schema planning diagnostic entry.
MssqlSchemaPlan: Planned SQL Server schema mapping for one selected output.
MssqlSchemaPlanOptions: Planning options for Arrow-to-SQL Server conversion.
MssqlTableName: SQL Server table name.
MssqlTargetConfig: Target configuration for writing one selected output to SQL Server.
MssqlTargetOutputPlan: Complete SQL Server target planning artifact for one selected output.
MssqlTargetResolutionContext: Context defaults available while resolving one selected output target.
MssqlTargetSummary: Redacted report for one resolved SQL Server target.
MssqlTargetTable: SQL Server table identity before arrow-sql-server identifier validation.
MssqlWorkflowWriteOptions: SQL Server multi-output workflow options.
MssqlWorkflowWriteReport: Structured report for a multi-output SQL Server write workflow.
MssqlWriteFailureContext: Structured context for a one-output SQL Server write failure.
MssqlWriteFailureReport: Structured report for the first failed SQL Server output.
MssqlWriteReport: Redacted per-output SQL Server write report.
MssqlWriteSkippedReport: Structured report for a skipped SQL Server output.
MssqlWriteStats: Per-output SQL Server write statistics.
OutputWritePlan: Planned output write request before schema planning or execution.
PhaseTimingReport: Timing evidence for one stable workflow phase.
PlannedMssqlOutput: Planned MSSQL output request for one selected lazy table.
PreviewFailureContext: Structured diagnostics retained when a bounded preview fails.
PreviewOptions: Options for a bounded lazy-table preview.
QueryExecutionMetric: One redacted metric attached to a query execution operator.
QueryExecutionOperatorProfile: Immutable profile for one unique physical execution-plan node.
QueryExecutionProfile: Immutable execution profile for one Delta Funnel query scope.
QueryOptions: Query execution options applied before DataFusion planning and execution.
RegisteredDeltaSource: One registered Delta source.
RegisteredDeltaSources: Registered Delta sources visible to a DataFusion session.
RegisteredDerivedTable: Registered SQL-derived table alias tracked by a query-load session.
RegisteredSessionSource: Registered Delta source tracked by a query-load session.
ResolvedMssqlTarget: Resolved SQL Server target for one selected output.
SessionOptions: Session-wide options for lazy query-load orchestration.
TablePreview: Rendered bounded preview of a lazy table.
ValidationOptions: Validation and scan-summary options checked before workflow side effects.
WriteAllCacheAliasReport: Selected registered derived alias cache metadata.
WriteAllCacheCandidateSkip: Registered derived alias skipped during cache selection.
WriteAllCacheFailure: Structured details retained when a cache-enabled write_all call fails.
WriteAllOptions: Execution options for one multi-output write_all call.
WriteAllReport: Report for one write_all call that reached the sequential workflow.

Enums§

BatchHandoffError: Terminal failure from a query-output handoff.
BatchPipelinePhase: Phase for batch pipeline setup and configuration failures.
DryRunScanSummaryMode: Controls how much source scan metadata a dry-run report should collect.
ExecutionProfileMode: Controls whether a query execution profile is collected.
FileCount: File count evidence for a report field.
FileCountKind: Classification for a reported file count.
LazyTableKind: Kind of lazy table represented by a session handle.
LoadMode: SQL Server table lifecycle mode requested for one selected output.
MssqlBinaryPolicy: Binary conversion policy.
MssqlConnectionSource: Where an effective SQL Server connection came from.
MssqlDate64Policy: Date64 conversion policy.
MssqlDecimal256Policy: Decimal256 conversion policy.
MssqlDecimalPolicy: Decimal conversion policy.
MssqlDryRunSqlIdentityState: SQL identity state included in an MSSQL dry-run output report.
MssqlFloatPolicy: Floating-point conversion policy.
MssqlLifecycleExecutionGuardrail: An execution operation guarded by successful lifecycle planning.
MssqlLifecycleGuardrailPolicy: Whether guarded execution may proceed after lifecycle planning.
MssqlNanosecondPolicy: Nanosecond timestamp precision policy.
MssqlOutputWriteStatus: Final write status for one selected SQL Server output.
MssqlPreparedTargetAction: Side effect completed while preparing a SQL Server target for loading.
MssqlStringPolicy: String conversion policy.
MssqlTargetCleanupStatus: Cleanup reporting state for a SQL Server target owned by create-and-load.
MssqlTargetTableState: Expected live state of the target table before loading starts.
MssqlTimestampPolicy: Timezone-free timestamp target policy.
MssqlTimezonePolicy: Timezone-aware timestamp conversion policy.
MssqlUInt64Policy: Unsigned 64-bit integer conversion policy.
MssqlWriteBackend: SQL Server write backend selection.
MssqlWritePhase: Phase of one-output SQL Server write execution.
MssqlWriteSkippedReason: Reason one selected SQL Server output was skipped.
OutputStatus: Output-level workflow status.
OutputStatusKind: Classification for an output-level workflow status.
PhaseStatus: Status for a workflow phase such as planning, loading, writing, or validation.
PhaseStatusKind: Classification for a workflow phase status.
QueryExecutionMetricCategory: Stability category assigned to a DataFusion execution metric.
QueryExecutionMetricValue: Typed value captured for one DataFusion execution metric.
QueryExecutionOutcome: Describes how query execution reached its terminal state.
QueryExecutionScope: Identifies why Delta Funnel executed a query plan.
ReportReasonCode: Stable reason codes for skipped, unavailable, and not-executed report states.
RowCount: Row count evidence for a report field.
RowCountKind: Classification for a reported row count.
RunMode: Query-load action mode requested by a caller.
SourceUsageStatus: Conservative source usage status for a workflow report.
TargetValidationMode: Controls whether target-side validation should run when a workflow supports it.
ValidationStatus: Target-side validation status for a report scope.
ValidationStatusKind: Classification for target-side validation status.
WorkflowStatus: Workflow-level status for a report.
WorkflowStatusKind: Classification for a workflow-level status.
WriteAllCacheAliasStatus: Cache lifecycle status for one selected alias in a write_all report.
WriteAllCacheCandidateSkipReason: Reason a cache candidate was skipped during write_all cache planning.
WriteAllCacheMode: Cache policy for one multi-output write_all call.
WriteAllCacheReport: Cache metadata for one write_all call.
WriteAllNoCacheReason: Conservative reason no cache alias was selected for write_all.

Constants§

VERSION: Current crate version.

Traits§

RecordBatchConsumer: Downstream consumer for one query-output RecordBatch.

Functions§

connect_mssql_client_from_ado_string: Connects to SQL Server from an ADO-style connection string.
datafusion_query_output_stream: Executes one selected DataFusion query output as a single merged stream.
datafusion_session_config: Builds a DataFusion session config from DeltaFunnel query options.
datafusion_session_context: Builds a DataFusion session context from DeltaFunnel query options.
default_mssql_write_backend: Returns Delta Funnel’s default SQL Server write backend.
duration_to_micros_saturating: Saturates a Duration into the public microsecond timing report shape.
handoff_datafusion_query_output: Executes one selected DataFusion query output and hands it to a consumer.
handoff_record_batch_stream: Hands one DataFusion RecordBatch stream to one downstream consumer.
mssql_schema_diagnostic_reports: Converts arrow-sql-server diagnostics into DeltaFunnel-owned report entries.
mssql_write_backend_for_output_plan: Builds a write backend from a planned SQL Server output target.
plan_mssql_create_table_ddl: Plans optional create-table DDL for one selected output schema plan.
plan_mssql_lifecycle: Plans lifecycle behavior for one selected output.
plan_mssql_output_schema: Plans one selected output Arrow schema for the resolved SQL Server target.
plan_mssql_target_for_output: Plans one selected output schema for SQL Server writing.
plan_mssql_target_for_resolved_output: Plans one selected output schema for an already resolved SQL Server target.
plan_mssql_target_output: Builds a complete target output plan from an existing schema plan.
register_delta_sources: Registers loaded Delta sources into a DataFusion session.
register_delta_sources_with_scan_execution_options: Registers loaded Delta sources with explicit reader execution bounds.
sanitize_uri_for_display: Sanitizes a URI for display in logs, errors, and reports.
u128_to_u64_saturating: Saturates a wide count into the public u64 report shape.
usize_to_u64_saturating: Saturates a platform-sized count into the public u64 report shape.
validate_mssql_output_record_batch: Validates a runtime RecordBatch schema against a planned SQL Server output.
validate_mssql_output_schema: Validates a runtime Arrow schema against a planned SQL Server output.
version: Returns the current crate version.
write_mssql_outputs_to_mssql: Writes multiple SQL Server outputs sequentially.
write_output_batches_to_mssql: Writes one resolved output to SQL Server from an Arrow record batch stream.

Type Aliases§

MssqlOutputBatchStream: Lazy stream produced only when a SQL Server output is attempted.
MssqlOutputBatchStreamFactory: Async factory that constructs a direct batch stream for one attempted output.