Crate krishiv_sql 

SQL planning and local execution seam for Krishiv.

This crate owns the DataFusion integration for R1 while keeping DataFusion out of the long-term public API exposed by krishiv-api.

Re-exports

pub use cep_sql::MatchRecognizeStatement;

pub use cep_sql::execute_streaming_match_recognize;

pub use cep_sql::parse_match_recognize;

pub use lakehouse::AsOfTableRef;

pub use lakehouse::MergeResult;

pub use lakehouse::MergeTargetUnsupportedError;

pub use lakehouse::preprocess_as_of_sql;

pub use grammar::FeatureEntry;

pub use grammar::FeatureStatus;

pub use grammar::feature_matrix;

pub use grammar::features_by_status;

pub use grammar::features_for_category;

pub use sqlstate::SqlStateError;

pub use sqlstate::sqlstate_for;

pub use streaming::ContinuousInputError;

pub use streaming::ContinuousTableInput;

Modules

analyze

ANALYZE TABLE — collect column statistics from a batch.

catalog

Catalog abstractions for Krishiv.

cep_sql

MATCH_RECOGNIZE SQL extension planning and execution (R16 S2).

connector_table

DataFusion TableProviderFactory implementations backed by krishiv_connectors::ConnectorRegistry.

coop_amplifiers

Cooperative yielding for input-amplifying operators (#217).

coverage

Phase 60 coverage harness — the Spark-reference SQL surface as a measured number, not a vibe.

create_function_ddl

Pre-processor for CREATE FUNCTION … RETURNS TABLE DDL.

distributed_plan

Distributed physical-plan fragments and the stage builder (ADR-0003).

grace_hash_join

A hash join that spills.

grammar

SQL grammar and engine-dimensioned feature matrix for Krishiv.

incremental_view

CREATE INCREMENTAL VIEW and DECLARE RECURSIVE VIEW SQL extensions.

introspection_sql

DESCRIBE, SHOW COLUMNS, and EXPLAIN SQL intercepts.

kafka_table

lakehouse

R18 lakehouse SQL extensions: delta/hudi providers, AS OF, MERGE INTO.

late_materialize

Late materialisation of a bounded top-N aggregate.

live_table

CREATE LIVE TABLE SQL extensions (R14 S1.1).

object_store_registry

An object-store registry that builds cloud stores on first use.

pipeline_ddl

Declarative pipeline DDL: CREATE SOURCE / CREATE SINK / START PIPELINE.

pivot_sql

E5.4 — PIVOT / UNPIVOT SQL macro rewrite layer.

python_udf

Distributed Python UDF execution.

runtime_filter_exec

Cross-stage runtime filter: the two plan nodes that carry a bloom filter from a join’s build side into the probe side’s map stage.

scalar_udf

Scalar SQL-expression user functions, expanded (inlined) into native SQL before planning.

semi_join_reduction

Semi-join reduction through an aggregate.

spark_sql_ext

Spark SQL extensions: LATERAL VIEW, TABLESAMPLE, TRANSFORM, DESCRIBE EXTENDED, etc. Spark SQL feature extensions — pre-processors for SQL constructs that DataFusion doesn’t parse natively.

spillable_join

Per-join selection of a spillable algorithm under a memory cap.

sqlstate

SQLSTATE code mapping for Krishiv SQL errors.

statement_completion

Spark-reference session/navigation statements that DataFusion’s planner does not handle natively (Phase 60 statement completion).

streaming

streaming_table_ddl

CREATE STREAMING TABLE <name> AS <select> — the SQL front door for a continuous streaming job (Phase 60 “SQL DDL for the other two engines”).

streaming_tvf

Streaming window table-valued functions: TUMBLE, HOP, SESSION.

streaming_window_plan

Compile a windowed streaming SQL query into a WindowExecutionSpec.

subquery

E5.1 — Correlated subquery decorrelation: EXISTS/IN/scalar subquery analysis.

unnest_sql

E5.2 — LATERAL / UNNEST SQL pre-processing.

unspillable_headroom

A memory pool that keeps a slice of the budget out of reach of spillable consumers, so an operator that cannot spill can still make progress.

Structs

CsvReaderOptions

Typed options for CSV reads (propagated into DataFusion).

CsvWriterOptions

Typed options for CSV writes.

MemoryConsumer

The consumer/reservation half of the same pool API.

MemoryReservation

The consumer/reservation half of the same pool API.

OperationRegistry

Registry of cancelled operation IDs and optional progress snapshots.

ParquetReaderOptions

Typed options for Parquet reads (propagated into DataFusion).

ParquetWriterOptions

Typed options for Parquet writes (propagated into the ArrowWriter).

SqlDataFrame

Krishiv-owned wrapper around a DataFusion DataFrame.

SqlEngine

SqlExecutionStats

Lightweight execution statistics collected from a DataFusion physical plan.

SqlPlan

SQL planning output.

SqlStatsHandle

Handle onto a streamed execution’s physical plan; reads runtime metrics (output rows, CPU time, spill totals) after the stream is drained.

TaggedQueryResult

A query result annotated with the operation ID that produced it.

Enums

EngineMemory

Where an engine’s DataFusion execution memory comes from.

GroupingMode

Engine-agnostic interface over a prepared query result.

SqlError

SQL-layer errors.

Traits

KrishivDataFrameOps

MemoryPool

DataFusion’s memory-pool trait, re-exported so crates that only build engines (the executor) can hold a shared pool without depending on DataFusion directly. Tracks and potentially limits memory use across operators during execution.

RecordBatchStream

The record-batch stream trait, re-exported for the same reason as MemoryPool.

Functions

batch_size_from_env

Resolve the batch size from KRISHIV_BATCH_SIZE env var.

cgroup_memory_limit_bytes

Read this process’s cgroup memory limit in bytes (v2 first, then v1).

default_parallelism_from_env

Resolve the default parallelism from KRISHIV_TARGET_PARALLELISM env var.

explain_sql

Create bootstrap EXPLAIN text for a SQL query.

explain_sql_optimized

Explain a SQL query including optimizer rule decisions.

explain_sql_with_cost

Explain a SQL query and append a cost estimate from the provided cost model.

parse_public_expression

Parse the stable SQL-expression subset into the same engine-owned AST used by Rust and Python.

plan_sql

Create a Krishiv logical plan wrapper for a SQL query without executing it.

pretty_batches

Format Arrow batches for CLI and tests.

process_query_pool

The one query memory pool for this process, sized by krishiv_common::ExecutorCapacity from the cgroup limit.

query_memory_limit_from_env

Resolve the default per-engine memory limit from the KRISHIV_QUERY_MEMORY_LIMIT_BYTES environment variable, falling back to a cgroup-derived default when the variable is unset.

referenced_table_names

Return all base table/relation names referenced by query.

resolve_query_memory_limit_bytes

Resolve a per-engine DataFusion memory limit from a raw env var value. None, unparseable, and zero values all mean “no limit” (the engine runs with DataFusion’s default unbounded pool).

resolve_streaming_match_recognize_limit

Resolve the streaming MATCH_RECOGNIZE row cap from a raw env var value. None and unparseable values fall back to the documented default of 100_000. Zero is rejected because it would mean “scan zero rows”.

runtime_filters_enabled_from_env

Phase 54: DataFusion’s native dynamic (“runtime”) filters — TopK, join, and aggregate predicates pushed sideways into probe-side file scans at execution time. On by default; KRISHIV_RUNTIME_FILTERS=off disables them all (the AQE dual-run switch for result-identity verification).

streaming_match_recognize_limit_from_env

Resolve the streaming MATCH_RECOGNIZE row cap from the KRISHIV_MATCH_RECOGNIZE_STREAMING_LIMIT environment variable.

with_krishiv_optimizer_rules

Build the DataFusion session config with a configurable parallelism level.

with_krishiv_optimizer_rules_with_join_threshold

As with_krishiv_optimizer_rules, with the spillable-join build-side threshold given explicitly; None derives it from this process’s capacity.

Type Aliases

SqlResult

SQL result alias.

SqlStream

Pinned stream of record batches with typed SqlError items.