Crate delta_funnel

Expand description

Core library for Delta Funnel.

This crate will own the high-level export orchestration from table formats such as Delta Lake into Microsoft SQL Server. Low-level Arrow to TDS bulk loading is expected to stay in arrow-tiberius.

Module boundaries should follow the load workflow. table_formats owns upstream table-format integrations, starting with Delta source configuration and snapshot loading. Later DataFusion provider, query execution, SQL Server sink, and orchestration work should land in their own modules when the first real implementation slice needs them.

Re-exports§

pub use error::DeltaFunnelError;
pub use error::SqlTablePhase;

Modules§

error: Shared error pattern for DeltaFunnel.

Structs§

BatchHandoffOutcome: Successful result for one completed query-output handoff.
BatchHandoffStats: Per-output batch handoff counters.
DeltaFunnelRuntime: Blocking runtime boundary for high-level Delta Funnel session actions.
DeltaFunnelSession: Rust backing session for lazy query-load workflows.
DeltaProtocolReport: Protocol details for one named Delta source.
DeltaProviderReadStatsSnapshot: Immutable view of provider read progress for one physical scan.
DeltaProviderScanExecutionOptions: Bounded scheduling options for one Delta DataFusion provider scan.
DeltaProviderSchedulingReport: Configured provider scheduling options included with a Delta source report.
DeltaSourceConfig: Caller-provided configuration for one named Delta source.
DeltaSourceReport: Source-level readiness report for one registered Delta source.
DeltaTableProviderConfig: Delta source and preflight state used to build a DataFusion table provider.
LazyTable: Lazy table identity owned by a query-load session.
MssqlBatchShapingReport: Per-output query stream and identity batch-shaping counters.
MssqlConnectionConfig: A SQL Server connection configuration with secret-bearing material hidden from display.
MssqlConnectionSummary: Redacted SQL Server connection context suitable for reports and errors.
MssqlDdlPlan: Planned SQL Server DDL artifacts for one selected output.
MssqlDryRunOutputFieldReport: Output schema field included in an MSSQL dry-run report.
MssqlDryRunOutputReport: Dry-run planning report for one selected MSSQL output.
MssqlDryRunSqlIdentityReport: Redacted SQL identity included in an MSSQL dry-run output report.
MssqlDryRunWorkflowReport: Dry-run planning report for a multi-output MSSQL workflow.
MssqlLifecyclePlan: Planned SQL Server table lifecycle behavior for one selected output.
MssqlOutputBatchValidationReport: Redacted report for a successful planned-output schema validation.
MssqlOutputFieldReport: Output schema field included in an MSSQL execute report.
MssqlOutputTarget: MSSQL output target selected from a lazy table.
MssqlOutputWriteJob: One deferred SQL Server output write job.
MssqlPreparedTarget: Prepared SQL Server target identity and redacted report.
MssqlPreparedTargetReport: Redacted report for a prepared SQL Server target.
MssqlSchemaDiagnostic: DeltaFunnel-owned schema planning diagnostic entry.
MssqlSchemaDiagnosticField: Field context for a DeltaFunnel-owned schema planning diagnostic entry.
MssqlSchemaPlan: Planned SQL Server schema mapping for one selected output.
MssqlSchemaPlanOptions: Planning options for Arrow-to-SQL Server conversion.
MssqlTableName: SQL Server table name.
MssqlTargetConfig: Target configuration for writing one selected output to SQL Server.
MssqlTargetOutputPlan: Complete SQL Server target planning artifact for one selected output.
MssqlTargetResolutionContext: Context defaults available while resolving one selected output target.
MssqlTargetSummary: Redacted report for one resolved SQL Server target.
MssqlTargetTable: SQL Server table identity before arrow-tiberius identifier validation.
MssqlWorkflowWriteOptions: SQL Server multi-output workflow options.
MssqlWorkflowWriteReport: Structured report for a multi-output SQL Server write workflow.
MssqlWriteFailureContext: Structured context for a one-output SQL Server write failure.
MssqlWriteFailureReport: Structured report for the first failed SQL Server output.
MssqlWriteOptions: Execution-time write options.
MssqlWriteReport: Redacted per-output SQL Server write report.
MssqlWriteSkippedReport: Structured report for a skipped SQL Server output.
MssqlWriteStats: Per-output SQL Server write statistics.
OutputWritePlan: Planned output write request before schema planning or execution.
PhaseTimingReport: Timing evidence for one stable workflow phase.
PlannedDeltaSource: Loaded named Delta source state.
PlannedMssqlOutput: Planned MSSQL output request for one selected lazy table.
ProtocolPreflight: Successful protocol preflight for one source.
QueryOptions: Query execution options applied before DataFusion planning and execution.
RegisteredDeltaSource: One registered Delta source.
RegisteredDeltaSources: Registered Delta sources visible to a DataFusion session.
RegisteredDerivedTable: Registered SQL-derived table alias tracked by a query-load session.
RegisteredSessionSource: Registered Delta source tracked by a query-load session.
ResolvedMssqlTarget: Resolved SQL Server target for one selected output.
SessionOptions: Session-wide options for lazy query-load orchestration.
ValidationOptions: Validation and scan-summary options checked before workflow side effects.
WriteAllCacheAliasReport: Selected registered derived alias cache metadata.
WriteAllCacheCandidateSkip: Registered derived alias skipped during conservative cache selection.
WriteAllOptions: Execution options for one multi-output write_all call.
WriteAllReport: Report for one write_all call that reached the sequential workflow.

Enums§

BatchHandoffError: Terminal failure from a query-output handoff.
BatchPipelinePhase: Phase for batch pipeline setup and configuration failures.
DeltaProviderReaderBackend: Provider file-reader backend selected for Delta scan execution.
DryRunScanSummaryMode: Controls how much source scan metadata a dry-run report should collect.
FileCount: File count evidence for a report field.
FileCountKind: Classification for a reported file count.
LazyTableKind: Kind of lazy table represented by a session handle.
LoadMode: SQL Server table lifecycle mode requested for one selected output.
MssqlBinaryPolicy: Binary conversion policy.
MssqlConnectionSource: Where an effective SQL Server connection came from.
MssqlDate64Policy: Date64 conversion policy.
MssqlDecimal256Policy: Decimal256 conversion policy.
MssqlDecimalPolicy: Decimal conversion policy.
MssqlDryRunSqlIdentityState: SQL identity state included in an MSSQL dry-run output report.
MssqlFloatPolicy: Floating-point conversion policy.
MssqlLifecycleExecutionGuardrail: An execution operation guarded by successful lifecycle planning.
MssqlLifecycleGuardrailPolicy: Whether guarded execution may proceed after lifecycle planning.
MssqlNanosecondPolicy: Nanosecond timestamp precision policy.
MssqlOutputWriteStatus: Final write status for one selected SQL Server output.
MssqlPreparedTargetAction: Side effect completed while preparing a SQL Server target for loading.
MssqlStringPolicy: String conversion policy.
MssqlTargetCleanupStatus: Cleanup reporting state for a SQL Server target owned by create-and-load.
MssqlTargetTableState: Expected live state of the target table before loading starts.
MssqlTimestampPolicy: Timezone-free timestamp target policy.
MssqlTimezonePolicy: Timezone-aware timestamp conversion policy.
MssqlUInt64Policy: Unsigned 64-bit integer conversion policy.
MssqlWritePhase: Phase of one-output SQL Server write execution.
MssqlWriteSkippedReason: Reason one selected SQL Server output was skipped.
OutputStatus: Output-level workflow status.
OutputStatusKind: Classification for an output-level workflow status.
PhaseStatus: Status for a workflow phase such as planning, loading, writing, or validation.
PhaseStatusKind: Classification for a workflow phase status.
ReportReasonCode: Stable reason codes for skipped, unavailable, and not-executed report states.
RowCount: Row count evidence for a report field.
RowCountKind: Classification for a reported row count.
RunMode: Query-load action mode requested by a caller.
SourceUsageStatus: Conservative source usage status for a workflow report.
TargetValidationMode: Controls whether target-side validation should run when a workflow supports it.
ValidationStatus: Target-side validation status for a report scope.
ValidationStatusKind: Classification for target-side validation status.
WorkflowStatus: Workflow-level status for a report.
WorkflowStatusKind: Classification for a workflow-level status.
WriteAllCacheAliasStatus: Cache lifecycle status for one selected alias in a write_all report.
WriteAllCacheCandidateSkipReason: Reason a cache candidate was skipped by conservative write_all planning.
WriteAllCacheMode: Cache policy for one multi-output write_all call.
WriteAllCacheReport: Cache metadata for one write_all call.
WriteAllNoCacheReason: Conservative reason no cache alias was selected for write_all.

Constants§

VERSION: Current crate version.

Traits§

RecordBatchConsumer: Downstream consumer for one query-output RecordBatch.

Functions§

collect_delta_provider_read_stats: Collects provider-owned Delta read stats snapshots from a DataFusion physical plan.
connect_mssql_client_from_ado_string: Connects to SQL Server from an ADO-style connection string.
datafusion_query_output_stream: Executes one selected DataFusion query output as a single merged stream.
datafusion_session_config: Builds a DataFusion session config from DeltaFunnel query options.
datafusion_session_context: Builds a DataFusion session context from DeltaFunnel query options.
default_mssql_write_options: Returns Delta Funnel’s default SQL Server write options.
duration_to_micros_saturating: Saturates a Duration into the public microsecond timing report shape.
handoff_datafusion_query_output: Executes one selected DataFusion query output and hands it to a consumer.
handoff_record_batch_stream: Hands one DataFusion RecordBatch stream to one downstream consumer.
load_delta_source: Loads one named Delta source.
load_delta_sources: Loads configured Delta sources after validating all names.
mssql_schema_diagnostic_reports: Converts arrow-tiberius diagnostics into DeltaFunnel-owned report entries.
mssql_write_options_for_output_plan: Builds write options from a planned SQL Server output target.
plan_mssql_create_table_ddl: Plans optional create-table DDL for one selected output schema plan.
plan_mssql_lifecycle: Plans lifecycle behavior for one selected output.
plan_mssql_output_schema: Plans one selected output Arrow schema for the resolved SQL Server target.
plan_mssql_target_for_output: Plans one selected output schema for SQL Server writing.
plan_mssql_target_for_resolved_output: Plans one selected output schema for an already resolved SQL Server target.
plan_mssql_target_output: Builds a complete target output plan from an existing schema plan.
preflight_delta_protocol: Runs conservative Delta protocol preflight for one loaded source.
preflight_delta_sources: Runs conservative Delta protocol preflight for loaded sources.
register_delta_sources: Registers preflighted Delta sources into a DataFusion session.
register_delta_sources_with_scan_execution_options: Registers preflighted Delta sources with explicit provider execution bounds.
sanitize_uri_for_display: Sanitizes a URI for display in logs, errors, and reports.
u128_to_u64_saturating: Saturates a wide count into the public u64 report shape.
usize_to_u64_saturating: Saturates a platform-sized count into the public u64 report shape.
validate_mssql_output_record_batch: Validates a runtime RecordBatch schema against a planned SQL Server output.
validate_mssql_output_schema: Validates a runtime Arrow schema against a planned SQL Server output.
version: Returns the current crate version.
write_mssql_outputs_to_mssql: Writes multiple SQL Server outputs sequentially.
write_output_batches_to_mssql: Writes one resolved output to SQL Server from an Arrow record batch stream.

Type Aliases§

DeltaStorageOptions: Storage options for one Delta source.
MssqlOutputBatchStream: Lazy stream produced only when a SQL Server output is attempted.
MssqlOutputBatchStreamFactory: Async factory that constructs a direct batch stream for one attempted output.