Module plans 

Available on crate feature declarative-plans only.

Declarative plans: the kernel describes data work as a relational plan; the engine executes it.

Kernel does no I/O or data processing itself. When an operation needs data work, kernel builds a plan and hands it to the engine’s PlanExecutor, which compiles it into the engine’s own representation (a Spark or DataFusion logical plan, an iterator pipeline) and runs it. The engine therefore applies its own optimizer, parallelism, and async I/O to all of kernel’s data work, not just leaf scans.

What a plan is

A Plan is a DAG of relational operators (Operator): sources, transforms, and set combinators. Most map one-to-one onto a SQL operator, so a plan reads like a query. The live-add metadata plan built in scan::scan_plan, for example, is roughly:

-- commits: keep the newest action per file, then keep only the live adds
SELECT add FROM (
    SELECT max_by(action, version) AS add FROM commits GROUP BY file_key
) WHERE add IS NOT NULL
UNION ALL
-- checkpoint adds that no newer commit superseded
SELECT c.add FROM checkpoint c
LEFT ANTI JOIN commit_keys k ON c.file_key = k.file_key

Writing an executor

An executor implements PlanExecutor::execute_op, dispatching on the Operation it receives and returning the matching PlanResult variant:

• Operation::IoOperation is a single I/O request; each IoOperation variant documents the PlanResult it must return.
• Operation::QueryPlan is a Plan, returning PlanResult::Data. Either evaluate Plan::nodes in slice order, which is topologically sorted so a node’s inputs are already evaluated, or compile the DAG into the engine’s own plan.

Every operator, expression, and predicate a plan contains must be handled; returning an error for an unsupported one is fine, and kernel surfaces it to the caller. The sync engine’s SyncPlanExecutor is a complete reference implementation.

Consuming terminal results

PlanExecutor::execute_op and PlanResult::Data provide the generic result contract for operations whose output is consumed by kernel.

Some kernel APIs instead return a Plan whose terminal rows belong to the connector. The connector may execute the plan through its ordinary query engine and keep the result in the engine’s native representation instead of adapting it through EngineData only to pass it back to itself. Scan::declarative_metadata_scan_plan is one example: the connector may consume the live add rows from the returned plan itself.

Connectors should use this native path when they own the result and adapting it through EngineData adds no semantic value. The native path must preserve the plan’s relational semantics and declared output schema. Schema validation, streaming behavior, error propagation, and cancellation must work the same way as they do on the generic path.

The generic PlanExecutor path remains required for operations whose results kernel consumes.

Where to look

• PlanBuilder builds plans through a fluent, schema-validating API, each method documenting its operator with a runnable example.
• ir::nodes is the operator catalog: each Operator variant’s payload struct carries its semantics, invariants, and worked examples.
• crate::expressions defines the expressions and predicates operators evaluate, including the type and null semantics an executor must match.

This module is opt-in behind the declarative-plans feature flag.

Re-exports

pub use ir::IoOperation;

pub use ir::Operation;

Modules

ir

Plan intermediate representation.

proto

Protobuf wire format mirroring the kernel’s plan / schema / expression IR.

Structs

PlanBuilder

A Plan under construction. See the module docs.

Enums

PlanResult

The result of executing an Operation.

Traits

PlanExecutor

Provides the ability to execute declarative plans to the Delta Kernel.