Struct DataChunk 

pub struct DataChunk {
    pub rows: Vec<Vec<Value>>,
    pub typed_columns: Option<Vec<TypedColumn>>,
    pub selection: Option<Vec<usize>>,
    pub multiplicity: u64,
    pub schema: Arc<Schema>,
    pub layout: Arc<SlotLayout>,
    pub memory_reservation: Option<MemoryReservation>,
    pub columnar_stats: Option<Arc<ColumnarStats>>,
    /* private fields */
}

A chunk of rows processed in streaming execution

Fields

rows: Vec<Vec<Value>>

Row data with Value types

typed_columns: Option<Vec<TypedColumn>>

Optional typed column layout.

selection: Option<Vec<usize>>

Selection vector.

multiplicity: u64

Symbolic row multiplicity (Ladybug ResultSet::multiplicity analogue).

Transparent operators may bump this instead of physically duplicating rows. 1 means every visible row appears once. The expanded (flat) row count is visible_count * multiplicity.

schema: Arc<Schema>

Schema information (column names and types)

layout: Arc<SlotLayout>

Slot layout for slot-based value access.

memory_reservation: Option<MemoryReservation>

Memory reservation for this chunk’s data.

columnar_stats: Option<Arc<ColumnarStats>>

Query-level columnar fast-path counters (observability).

Implementations

impl DataChunk

pub fn new(rows: Vec<Vec<Value>>, schema: Arc<Schema>) -> Self

pub fn with_memory_reservation(self, reservation: MemoryReservation) -> Self

pub fn with_columnar_stats(self, stats: Arc<ColumnarStats>) -> Self

pub fn take_memory_reservation(&mut self) -> Option<MemoryReservation>

pub fn new_with_layout(rows: Vec<Vec<Value>>, layout: Arc<SlotLayout>) -> Self

pub fn try_new_with_layout( rows: Vec<Vec<Value>>, layout: Arc<SlotLayout>, ) -> Result<Self, QueryError>

pub fn from_rows(rows: Vec<Vec<Value>>) -> Self

pub fn from_rows_with_col_names( rows: Vec<Vec<Value>>, col_names: Option<Vec<String>>, ) -> Self

pub fn project_columns( columns: Vec<Vec<Value>>, layout: Arc<SlotLayout>, ) -> Self

Columnar projection assembly: build an output chunk from evaluated output columns without a row intermediate.

This is the designated constructor for the streaming Project fast path (passthrough / constant columns gathered or broadcast upstream). Row storage stays authoritative downstream; the transpose into rows is the single compatibility edge until the chunk itself is columnar. The typed rebuild here is unconditional by design: projected outputs are bounded by the input chunk size, unlike join outputs which honor the shared columnar gate.

pub fn from_batch(batch: &MaterializedBatch, layout: Arc<SlotLayout>) -> Self

Build a row chunk from a columnar batch slice.

Coexistence API: the batch stays the operator state authority while the chunk carries the row view downstream. A transpose is still performed internally until the chunk itself becomes columnar.

pub fn slice_from_batch( batch: &MaterializedBatch, offset: usize, len: usize, layout: Arc<SlotLayout>, ) -> Self

Build a row chunk from a row range of a columnar batch.

pub fn to_batch(&self) -> MaterializedBatch

Convert this chunk back into a columnar batch (single transpose edge).

pub fn estimated_bytes(&self) -> usize

Memory estimate for the visible expanded rows (visible_count * multiplicity), used for pool/tracker accounting.

pub fn len(&self) -> usize

pub fn is_empty(&self) -> bool

pub fn num_columns(&self) -> usize

pub fn col_names(&self) -> Vec<String>

pub fn col_name(&self, index: usize) -> Option<String>

pub fn col_name_index(&self) -> HashMap<String, usize>

pub fn get_layout(&self) -> Arc<SlotLayout>

pub fn get_by_slot(&self, row_idx: usize, slot: SlotId) -> Option<Value>

pub fn get_column(&mut self, slot: SlotId) -> Option<Vec<Value>>

Column view over Values, derived on demand from the typed layout.

New expression code should prefer typed_column; this stays for join keys and fallback paths that require owned Value columns. The column is rebuilt from the typed layout (or cloned from rows) on every call and never cached: row storage is authoritative.

pub fn column_ref(&self, slot: SlotId) -> Option<Vec<&Value>>

pub fn get_typed_by_slot(&self, row_idx: usize, slot: SlotId) -> Option<Value>

pub fn row_at(&self, i: usize) -> &[Value]

pub fn take_rows_for_reuse(&mut self) -> Vec<Vec<Value>>

Take the physical rows out for buffer reuse, resetting selection, multiplicity, and derived typed state.

Unlike Clone (documented deep copy), this moves the allocation out so the caller can return the drained buffers to super::pool::RowBufferPool via release_rows instead of dropping them. Prefer move-first construction plus this pool over cloning.

pub fn build_typed_columns(&mut self, use_columnar: bool) -> usize

Build the typed column layout for this chunk.

use_columnar carries the adaptive [ColumnarPolicy] decision from the producing operator: when the learned hit rate falls below the threshold the chunk stays row-based. The typed layout is otherwise always attempted (there is no global off switch). Returns the number of extra typed bytes allocated.

pub fn typed_column(&self, slot: SlotId) -> Option<&TypedColumn>

pub fn typed_len(&self) -> Option<usize>

Row count served by the typed layout when the row view was skipped.

Rowless chunks (storage scans whose planner annotation proved the direct consumer never reads rows) carry data only in typed_columns; every typed arm implements len, and assembly keeps the arms in lockstep exactly like rows. Returns None when there is no typed layout, so row-based consumers keep their existing empty semantics.

pub fn ensure_typed_columns(&mut self) -> usize

Rebuild the typed layout when construction deferred it.

Join outputs skip the eager build and set the deferred flag; the first typed consumer calls this to pay the build cost lazily. The columnar policy gate already ran at construction, so the rebuild is unconditional. Returns the extra typed bytes allocated.

impl DataChunk

pub fn evaluate_expressions( &mut self, expressions: &[Expression], env: Option<&EvalEnv>, ) -> Result<Vec<Vec<Value>>, ExpressionError>

pub fn try_evaluate_expressions_typed( &mut self, expressions: &[Expression], env: Option<&EvalEnv>, ) -> Result<Option<Vec<Vec<Value>>>, ExpressionError>

Batch-evaluate a list of output expressions, preferring the typed columnar path.

Returns Ok(None) as soon as any expression is not covered by the typed fast path (no typed_columns, a kind mismatch, or an operator outside the typed batch set), letting the caller fall back to the compiled-over-rows path with unchanged semantics. This is the entry point the Project/Filter hot path uses so a chunk that carries a typed layout is evaluated on raw scalars instead of per-row Values.

pub fn evaluate_expression( &mut self, expression: &Expression, env: Option<&EvalEnv>, ) -> Result<Vec<Value>, ExpressionError>

pub fn evaluate_expression_visible( &mut self, expression: &Expression, env: Option<&EvalEnv>, ) -> Result<Vec<Value>, ExpressionError>

impl DataChunk

pub fn multiplicity(&self) -> u64

Symbolic multiplicity: how many times each visible row occurs.

pub fn set_multiplicity(&mut self, multiplicity: u64)

pub fn with_multiplicity(self, multiplicity: u64) -> Self

pub fn logical_len(&self) -> u64

Expanded (flat) visible row count: visible_count * multiplicity. Saturates instead of overflowing so metrics paths stay total.

pub fn visible_rows(&self) -> impl Iterator<Item = &Vec<Value>>

Borrow visible rows in upstream order (respects selection).

pub fn expand_visible_rows(&mut self) -> Vec<Vec<Value>>

Move visible rows out, repeating each row multiplicity times.

This is the single terminal move: rows are moved out with mem::take with no per-Value clone on the move itself (multiplicity == 1 returns the moved rows directly; larger factors repeat them). After the call the chunk is empty with selection=None and derived column caches cleared.

Opaque order-preserving operators assume multiplicity == 1; a future multiplicity producer must expand at those boundaries with this call first (see the module-level multiplicity contract).

pub fn with_selection(self, indices: Vec<usize>) -> Self

pub fn selection(&self) -> Option<&[usize]>

pub fn visible_count(&self) -> usize

pub fn visible_indices(&self) -> Vec<usize>

pub fn is_visible(&self, idx: usize) -> bool

pub fn take_selection(&mut self) -> Option<Vec<usize>>

pub fn expand_multiplicity_in_place(&mut self) -> bool

Expand a symbolic multiplicity > 1 into physical rows in place.

Opaque operators (sort/window/join/aggregate) cannot carry the factor symbolically, so they must call this (directly or via DataChunk::normalize_for_opaque) before consuming rows. Returns true when an expansion happened. Visible selection is preserved: only visible rows are replicated, hidden rows are dropped. Derived column caches are cleared, matching expand_visible_rows.

pub fn normalize_for_opaque(&mut self, op: &'static str) -> bool

Unified opaque-operator boundary: expand multiplicity, then materialize any attached selection. Returns true when either step changed the chunk. op must be one of SELECTION_BOUNDARY_OPS.

This is the single entry point opaque operators should use instead of calling DataChunk::materialize_selection_by directly.

pub fn materialize_selection_by(&mut self, op: &'static str) -> bool

Contract: materialize any attached selection (opaque-operator boundary). Returns true when a selection was actually materialized.

Prefer DataChunk::normalize_for_opaque: this method alone does not expand multiplicity, so opaque operators calling it directly must guarantee multiplicity == 1 (debug-asserted by callers via the normalize path).

This is the single boundary-materialization entry point; op must be one of SELECTION_BOUNDARY_OPS so per-operator counters stay exact (ad-hoc names fall back to the unattributed counter).

pub fn take_indices(&mut self, indices: &[usize]) -> Self

pub fn slice(&mut self, start: usize, end: usize) -> Self

Trait Implementations

impl Clone for DataChunk

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DataChunk

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.