Struct NodeGroup 

pub struct NodeGroup {
    pub columns: Vec<ColumnChunk>,
    pub start_offset: u64,
    pub num_nodes: u64,
    pub version_info: Option<VersionInfo>,
    /* private fields */
}

A node group stores up to NODE_GROUP_SIZE rows in columnar format.

start_offset is the global row index within the owning table where this group’s data begins. num_nodes counts how many rows have been appended so far (≤ NODE_GROUP_SIZE).

version_info tracks MVCC insert/delete visibility for concurrent writers. It is None for single-writer mode (backward compat).

Disk Spilling

When spiller is set and the memory threshold is exceeded, the group automatically spills its contents to temp files during append_row(). After ingestion is complete, call flush_with_spiller() instead of flush() to merge all spills + final in-memory data into the columns.

Fields

columns: Vec<ColumnChunk>

One in-memory ColumnChunk per column of the table.

start_offset: u64

Global row offset within the owning table.

num_nodes: u64

Number of rows currently stored in this group.

version_info: Option<VersionInfo>

Optional MVCC version tracker for this group.

Implementations

impl NodeGroup

pub fn new(num_columns: usize, start_offset: u64) -> Self

Create a new empty node group for num_columns columns.

All columns start with the default NODE_GROUP_SIZE capacity. start_offset is the global row index in the owning table where this group begins.

pub fn with_capacity( num_columns: usize, start_offset: u64, capacity: usize, ) -> Self

Create a new node group with a custom chunk capacity per column.

pub fn with_spiller(self, spiller: Arc<Spiller>) -> Self

Attach a spiller to this node group for disk-based memory management.

When a spiller is attached, append_row() automatically spills the current buffer to disk when the memory threshold is exceeded, then continues appending. Call flush_with_spiller() instead of flush() to merge all spill files + final in-memory data.

pub fn set_spiller(&mut self, spiller: Arc<Spiller>)

Set the spiller on an existing node group.

pub fn enable_version_info(&mut self)

Enable MVCC version tracking for this node group. Must be called before any inserts if concurrent writes are expected.

pub fn append_row(&mut self, row: Vec<Value>) -> Result<(), StorageError>

Append a single row (one value per column) to the group.

Returns an error if the number of values does not match the number of columns, or if the group is already full.

If txn_id is Some(...), the insert is recorded in the version info for MVCC visibility tracking.

pub fn append_row_with_txn( &mut self, row: Vec<Value>, txn_id: Option<u64>, ) -> Result<(), StorageError>

Append a row with an optional transaction ID for MVCC tracking.

If a spiller is attached and the in-memory data exceeds the configured memory threshold, the current buffer is automatically spilled to disk before appending the new row. This keeps memory usage bounded during large batch operations like COPY FROM.

pub fn spill_and_clear(&mut self) -> Result<(), StorageError>

Spill all column chunks to disk and reset the group to empty.

The spill file is tracked so that flush_with_spiller() can later merge all spilled data back into the persistent columns.

Version info is reset together with the buffer: the records reference local row offsets that are about to be reused, so carrying them over would mis-label rows appended after the spill at the same offsets.

pub fn restore_spilled(&mut self) -> Result<(), StorageError>

Restore all spilled rows back into the in-memory columns.

Merges every tracked spill file (in creation order) followed by the rows appended since the last spill, so the group’s columns again hold the complete row set. Spill files are cleaned up on success. This is the ingest-time counterpart of flush_with_spiller(): it keeps the in-memory node group authoritative for scans and the column mirror after a memory-bounded bulk ingest (P51.44).

pub fn flush_with_spiller( &mut self, columns: &mut [Column], sort_key_column: Option<usize>, dedup: bool, ) -> Result<usize>

Flush all data to persistent columns, merging any spilled data.

This is the spill-aware alternative to flush(). It merges all previously spilled files + the current in-memory buffer into the target columns using a streaming merge. If no spilling occurred, this falls back to the regular flush().

The optional sort_key_column is the column index to use for merge ordering and PK deduplication. Pass None for unordered append (no dedup).

pub fn is_full(&self) -> bool

Whether the group has reached capacity.

pub fn is_empty(&self) -> bool

Whether the group is empty.

pub fn num_columns(&self) -> usize

Number of columns in this group.

pub fn has_spill_files(&self) -> bool

Whether any spill files are still pending merge-back into memory.

pub fn remaining(&self) -> usize

Remaining capacity (number of additional rows that can be appended).

pub fn flush(&mut self, columns: &mut [Column]) -> Result<usize>

Flush all buffered data to persistent Column instances.

Each ColumnChunk is flushed to the corresponding Column in the slice via flush_to_column(). After flushing, the chunks are cleared and ready for reuse.

Returns the total number of rows flushed.

Panics

Panics if columns.len() != self.columns.len().

pub fn flush_copy(&self, columns: &mut [Column]) -> Result<usize>

Flush data to columns but keep the in-memory buffer intact.

pub fn scan(&self) -> Vec<Vec<Value>>

Scan all rows currently buffered in the group.

Returns a Vec<Vec<Value>> where result[row][col] is the value at the given row and column.

pub fn scan_range(&self, start: usize, count: usize) -> Vec<Vec<Value>>

Scan a range of buffered rows [start, start + count).

Returns Vec<Vec<Value>> in row-major order.

pub fn get_value(&self, local_row: usize, col_idx: usize) -> Option<&Value>

Access a single value at the given local row and column index.

pub fn get_value_with_snapshot( &self, local_row: usize, col_idx: usize, snapshot_ts: Option<u64>, commit_history: &HashMap<u64, u64>, ) -> Option<&Value>

Access a single value with MVCC snapshot isolation.

Checks VersionInfo for insert/delete visibility first. If the row is not visible at snapshot_ts, returns None. Then checks UpdateInfo version chain on the column chunk for versioned updates.

pub fn get_value_owned_with_snapshot( &self, local_row: usize, col_idx: usize, snapshot_ts: Option<u64>, commit_history: &HashMap<u64, u64>, ) -> Option<Value>

Access a single value with MVCC snapshot isolation (owned variant).

Like get_value_with_snapshot but returns Option<Value> instead of Option<&Value>, enabling proper version chain traversal with deserialized old values from UpdateInfo.

pub fn is_row_visible( &self, local_row: usize, snapshot_ts: u64, commit_history: &HashMap<u64, u64>, ) -> bool

Check whether a row is visible at the given snapshot timestamp. Returns true if no version tracking is active (backward compat).

pub fn clear(&mut self)

Reset the group to empty without flushing.

Trait Implementations

impl Clone for NodeGroup

fn clone(&self) -> NodeGroup

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for NodeGroup

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.