Expand description
Streaming execution engine for pull-based iterator model
This module implements a pull-based chunked streaming executor system that efficiently processes data through an operator pipeline.
Key components:
- DataChunk: Fixed-size batches of rows
- StreamingExecutor: Enum-based pull operators
- StreamingExecutionEngine: Orchestration layer
- ExecutionRuntime: Per-query runtime (cancel, memory, profile, resources)
- ResultStream: Streaming result handle
Re-exports§
pub use chunk::DataChunk;pub use chunk::LocalChunkCollector;pub use chunk::RowBufferPool;pub use context::BorrowedRowContext;pub use engine::StreamingExecutionEngine;pub use executor::StreamingExecutor;pub use operators::base::OperatorBase;pub use operators::spec::BlockingSpec;pub use operators::spec::ExchangeSpec;pub use operators::spec::JoinSpec;pub use operators::spec::SourceSpec;pub use operators::spec::UnarySpec;pub use operators::state::BlockingState;pub use operators::state::ExchangeState;pub use operators::state::JoinState;pub use operators::state::SourceState;pub use partition::PartitionView;pub use result_utils::chunks_to_execution_result;pub use result_utils::chunks_to_execution_result_with_spill;pub use result_utils::convert_chunks_to_dataset;pub use result_utils::convert_chunks_to_dataset_with_spill;pub use runtime::ExecutionRuntime;pub use runtime::OperatorProfile;pub use runtime::OperatorProfileKey;pub use runtime::ProfileBoard;pub use runtime::ProfileCollector;pub use runtime::ProfileEntry;pub use runtime::QueryFinishGuard;pub use runtime::QueryIdentity;pub use runtime::ResourceOwner;pub use slot::combine_layouts;pub use slot::combine_layouts_with_dedup;pub use slot::SlotId;pub use slot::SlotInfo;pub use slot::SlotLayout;pub use slot::SlotOrigin;pub use stream::ResultStream;pub use stream_result::StreamingQueryResult;pub use plan::context::PhysicalPlanBuildContext;pub use plan::types::CapabilitySet;pub use plan::types::FragmentGraph;pub use plan::types::FragmentId;pub use plan::types::FragmentKind;pub use plan::types::FragmentSpec;pub use plan::types::LogicalNodeId;pub use plan::types::OperatorKindSpec;pub use plan::types::OutputContract;pub use plan::types::PhysicalOperatorId;pub use plan::types::PhysicalOperatorIdAllocator;pub use plan::types::PhysicalOperatorSpec;pub use plan::types::PhysicalPlan;pub use plan::types::PlanCompatibility;pub use plan::types::PlanFingerprint;pub use plan::types::SortOrder;pub use plan::validator::PhysicalPlanValidator;pub use plan::validator::ValidationResult;pub use plan::validator::ValidationTier;pub use spill::finalize_partitions_with_runtime;pub use spill::hash_bytes_partition;pub use spill::hash_row_partition;pub use spill::schema_fingerprint;pub use spill::DiskQuota;pub use spill::HashPartitionConfig;pub use spill::HashPartitionSpiller;pub use spill::RunHeader;pub use spill::RunReader;pub use spill::RunWriter;pub use spill::SpillConfig;pub use spill::SpillManager;pub use spill::SpilledRun;pub use spill::COLLECTOR_RUN_ROWS_MAX;pub use spill::COLLECTOR_SPILL_ROWS_DEFAULT;pub use spill::HASH_JOIN_MAX_DEPTH;pub use spill::HASH_JOIN_PARTITIONS_DEFAULT;pub use instance::QueryBindings;pub use instance::QueryExecutionInstance;pub use instance::ResultSink;pub use query_registry::CancelToken;pub use query_registry::QueryGuard;pub use query_registry::QueryId;pub use query_registry::QueryMetadata;pub use query_registry::QueryRegistry;pub use state::GlobalState;pub use state::GlobalStateArena;pub use state::GlobalStateKey;pub use state::LocalState;pub use state::LocalStateArena;pub use state::LocalStateKey;pub use state::StateArenaSet;pub use state::TaskId;pub use transaction_scope::CancelReason;pub use transaction_scope::SessionTransactionController;pub use transaction_scope::TransactionCommandResult;pub use transaction_scope::TransactionScope;pub use transaction_scope::TransactionState;pub use memory_pool::DatabaseMemoryPool;pub use memory_pool::FragmentPool;pub use memory_pool::MemoryPoolError;pub use memory_pool::MemoryPoolReservation;pub use memory_pool::OperatorPool;pub use memory_pool::PoolHandle;pub use memory_pool::PooledChunk;pub use memory_pool::QueryPool;pub use memory_pool::TaskPool;
Modules§
- chunk
- DataChunk: Basic unit of streaming execution
- context
- ValueRowContext: Expression evaluation context for streaming rows
- engine
- Streaming execution engine
- executor
- StreamingExecutor: Thin dispatch layer over domain-specific operator enums
- helpers
- Helper modules for streaming executor
- instance
- QueryExecutionInstance: single-entry instantiation point for physical plans.
- interner
- join_
helpers - memory_
pool - Hierarchical MemoryPool for query execution.
- operators
- Domain-specific operator enums
- parameters
- Parameter binding and expression types for physical plans.
- partition
- Data partitioning abstractions for parallel execution
- plan
- pool
- Morsel-style query-level bounded worker pool for dynamic partition execution.
- query_
registry - Query registry — process-level query identity and cancellation.
- result_
utils - Utility functions for converting streaming results to standard formats.
- runtime
- Per-query execution runtime.
- slot
- spill
- Spill-to-disk infrastructure for blocking operators.
- state
- Typed GlobalState / LocalState arenas for per-execution operator state.
- stream
- stream_
result - subquery
- Expression-level subquery execution.
- transaction_
scope - Transaction scope for query execution instances.
Structs§
- Transaction
Id - Transaction ID type. Defined once here; do NOT duplicate in other modules.