Struct Table 

pub struct Table { /* private fields */ }

An open MongrelDB table.

Implementations

impl Table

pub fn compact(&mut self) -> Result<()>

Merge all runs into a single level-1 run, dropping superseded versions and tombstones — but preserving the version each pinned snapshot still needs. No-op if there are fewer than two runs.

impl Table

pub fn create( dir: impl AsRef<Path>, schema: Schema, table_id: u64, ) -> Result<Self>

pub fn open(dir: impl AsRef<Path>) -> Result<Self>

Open an existing table: load the manifest, replay the active WAL segment into the memtable, and rebuild the HOT + secondary indexes from the runs and replayed rows.

pub fn put(&mut self, columns: Vec<(u16, Value)>) -> Result<RowId>

Upsert a row. Allocates a RowId, appends a (non-fsynced) WAL record, and updates the memtable + indexes. Returns the new row id. Durability arrives at the next Table::commit (or Table::flush).

For an AUTO_INCREMENT primary key, omit the column (or pass Value::Null) and the engine assigns the next counter value; use Table::put_returning to learn that assigned value.

pub fn put_returning( &mut self, columns: Vec<(u16, Value)>, ) -> Result<(RowId, Option<i64>)>

Like Table::put but also returns the engine-assigned AUTO_INCREMENT value (Some only when the column was omitted/null and the engine filled it; None when the table has no auto-increment column or the caller supplied an explicit value).

pub fn put_batch(&mut self, batch: Vec<Vec<(u16, Value)>>) -> Result<Vec<RowId>>

Bulk upsert: many rows under a single WAL record + one index pass. Far cheaper than put in a loop for batch ingest.

pub fn put_batch_returning( &mut self, batch: Vec<Vec<(u16, Value)>>, ) -> Result<Vec<(RowId, Option<i64>)>>

Like Table::put_batch but each entry is paired with the engine- assigned AUTO_INCREMENT value (Some only when filled by the engine).

pub fn fill_auto_inc( &mut self, columns: &mut Vec<(u16, Value)>, ) -> Result<Option<i64>>

Fill the AUTO_INCREMENT column for an upcoming row. When the column is omitted or Value::Null the next counter value is allocated and the cell is appended/replaced in columns; an explicit Int64 is honored and advances the counter past it. Returns Some(value) when the engine allocated (so the caller can surface it), None otherwise.

pub fn reserve_auto_inc(&mut self) -> Result<Option<i64>>

Reserve (but do not insert) the next AUTO_INCREMENT value, advancing the in-memory counter. Returns None when the table has no auto-increment column.

This is the escape hatch for callers that stage the row with an explicit id inside a cross-table [crate::Transaction] — where the engine cannot fill the column on the put path (the row id + cells are only assembled at commit). Unlike the old Kit __kit_sequences sequence row, the reservation is a pure in-memory counter bump: no hot row, no second commit. It becomes durable when a row carrying the reserved id commits (the counter is checkpointed to the manifest in the same commit); an aborted reservation simply leaves a gap, which the never-reuse rule permits.

pub fn delete(&mut self, row_id: RowId) -> Result<()>

Logically delete row_id (effective at the next commit).

pub fn delete_returning(&mut self, row_id: RowId) -> Result<Option<OwnedRow>>

pub fn truncate(&mut self) -> Result<()>

Durably remove every row in the table once the current write span commits.

pub fn commit(&mut self) -> Result<Epoch>

Group-commit: make all pending writes durable, advance the epoch so they become visible, and persist the manifest. Dispatches on the WAL sink: a standalone table fsyncs its private WAL; a mounted table seals into the shared WAL and defers the fsync to the group-commit coordinator (B1).

pub fn flush(&mut self) -> Result<Epoch>

Commit, then drain the memtable into the mutable-run LSM tier (Phase 11.1). The tier absorbs flushes in place and only spills to an immutable .sr sorted run once it crosses the spill watermark — coalescing many small flushes into fewer, larger runs. While the tier holds un-spilled data the WAL is not rotated: the Flush marker / WAL rotation is deferred until the data is durably in a run, so crash recovery replays those rows back into the memtable (the tier rebuilds from replay).

pub fn set_mutable_run_spill_bytes(&mut self, bytes: u64)

Tune the mutable-run spill watermark (bytes). A smaller threshold spills sooner (more, smaller runs — closer to the pre-Phase-11.1 behavior); a larger one coalesces more flushes in memory.

pub fn set_compaction_zstd_level(&mut self, level: i32)

Set the zstd compression level for compaction output (Phase 18.1). Default 3; higher values give better compression ratio at the cost of slower compaction.

pub fn set_result_cache_max_bytes(&mut self, max_bytes: u64)

Set the result-cache byte budget (Phase 19.1 hardening (a)). Entries are evicted in access-order LRU past this limit. Takes effect immediately (may evict entries if the new limit is smaller than the current footprint).

pub fn mutable_run_len(&self) -> usize

Number of versions currently held in the mutable-run tier.

pub fn bulk_load(&mut self, batch: Vec<Vec<(u16, Value)>>) -> Result<Epoch>

Bulk-load: write batch directly to a new sorted run, bypassing the WAL and the memtable entirely (no per-row bincode, no skip-list inserts). The run + a rotated WAL + the manifest are fsynced once — the fast ingest path for large analytical loads. Indexes are still maintained.

pub fn get(&self, row_id: RowId, snapshot: Snapshot) -> Option<Row>

Read the row at row_id visible to snapshot, merging the newest version across the memtable and all sorted runs.

pub fn visible_rows(&self, snapshot: Snapshot) -> Result<Vec<Row>>

All rows visible at snapshot (newest version per RowId, tombstones dropped), merged across the memtable, the mutable-run tier, and all runs. Ascending RowId.

pub fn visible_columns( &self, snapshot: Snapshot, ) -> Result<Vec<(u16, Vec<Value>)>>

Visible data as columns (column_id → values) rather than rows — the vectorized scan path. Fast path: when the memtable is empty and there is exactly one run (the common post-flush analytical case), it computes the visible index set once and gathers each column, with no per-row HashMap/Row materialization. Falls back to Self::visible_rows pivoted to columns when the memtable is live or runs overlap.

pub fn lookup_pk(&self, key: &[u8]) -> Option<RowId>

Resolve a primary-key value to a row id (latest version).

pub fn query(&mut self, q: &Query) -> Result<Vec<Row>>

Run a conjunctive query over the shared row-id space: each condition yields a candidate row-id set, the sets are intersected, and the survivors are materialized at the current snapshot. This is the AI-native “compose primitives” surface (semsearch ∩ fm_contains ∩ cat_in).

pub fn rows_for_rids( &self, rids: &[u64], snapshot: Snapshot, ) -> Result<Vec<Row>>

Materialize the MVCC-visible, non-deleted rows for rids at snapshot, preserving the input order. Rows whose newest visible version is a tombstone, or that no longer exist, are omitted. Shared by index-served [query] and the Phase 8.1 FK-join path.

pub fn indexes_complete(&self) -> bool

Resolve the referencing (FK) side of a primary-key ↔ foreign-key join as a row-id set (Phase 8.1): union the roaring-bitmap entries of fk_column_id for every value in pk_values — the surviving primary-key values — then intersect with fk_conditions, i.e. any FK-side predicates (ann_search ∩ fm_contains, bitmap equality, range, …). Returns the survivor row-ids ascending. Requires a bitmap index on fk_column_id; returns an empty set when there is none. Whether live indexes are complete (Phase 14.7 + 17.2: the broadcast join path checks this before using the HOT index).

pub fn broadcast_join_values( &self, column_id: u16, pk_db: &Table, ) -> Option<Vec<Vec<u8>>>

Phase 17.2: broadcast join — return the distinct values in this table’s bitmap index for column_id that also exist as a key in pk_db’s HOT index. Avoids loading the entire PK table when the FK column has low cardinality. Returns None if no bitmap index exists for the column.

pub fn fk_join_row_ids( &self, fk_column_id: u16, pk_values: &[Vec<u8>], fk_conditions: &[Condition], snapshot: Snapshot, ) -> Result<Vec<u64>>

pub fn fk_join_count( &self, fk_column_id: u16, pk_values: &[Vec<u8>], fk_conditions: &[Condition], snapshot: Snapshot, ) -> Result<u64>

Like [fk_join_row_ids] but returns only the cardinality of the FK survivor set — without materializing or sorting it. For a bare COUNT(*) join with no FK-side filter this is O(1) on the bitmap union (Phase 17.4): the prior path built a HashSet<u64> + Vec<u64> + sort_unstable over up to N rows only to read .len().

pub fn snapshot(&self) -> Snapshot

pub fn pin_snapshot(&mut self) -> Snapshot

Pin the current epoch as a read snapshot; compaction will preserve the versions it needs until Table::unpin_snapshot is called.

pub fn unpin_snapshot(&mut self, snap: Snapshot)

Release a pinned snapshot.

pub fn current_epoch(&self) -> Epoch

pub fn memtable_len(&self) -> usize

pub fn count(&self) -> u64

Live (non-deleted) row count, O(1) from a manifest-maintained counter — the metadata COUNT(*) fast path (no scan).

pub fn count_conditions( &mut self, conditions: &[Condition], snapshot: Snapshot, ) -> Result<Option<u64>>

Count rows matching an index-backed conjunctive predicate without materializing projected columns. Returns None when a condition cannot be served by the native predicate resolver.

pub fn bulk_load_columns( &mut self, user_columns: Vec<(u16, NativeColumn)>, ) -> Result<Epoch>

Bulk-load typed columns straight to a new run — the fast ingest path. Bypasses the WAL, the memtable, and the Value enum entirely; writes one compressed run (delta for sorted Int64, dictionary for low-card Bytes) with LZ4 (Phase 15.3 — fast decode for scan-heavy analytical runs), rotates the WAL, and persists the manifest in a single fsync group. Indexes are bulk-built from the typed columns (Phase 14.2).

pub fn bulk_load_fast( &mut self, user_columns: Vec<(u16, NativeColumn)>, ) -> Result<Epoch>

Maximal-throughput bulk ingest (Phase 14.4): skip zstd entirely and write raw ALGO_PLAIN pages. ~3–4× the encode throughput of Self::bulk_load_columns at ~3–4× the on-disk size — the right choice when ingest latency dominates and a background compaction will re-compress later. Indexing, WAL rotation, and the manifest are identical to Self::bulk_load_columns.

pub fn visible_columns_native( &self, snapshot: Snapshot, projection: Option<&[u16]>, ) -> Result<Vec<(u16, NativeColumn)>>

no Value). Fast path: empty memtable + single run decodes columns directly and gathers visible indices; falls back to the Value path pivoted to native columns otherwise. projection (a set of column ids) limits decoding to the requested columns — None ⇒ all user columns.

pub fn run_count(&self) -> usize

pub fn memtable_is_empty(&self) -> bool

Whether the memtable is empty (no unflushed puts).

pub fn page_cache_stats(&self) -> CacheStats

Cumulative raw-page-cache hit/miss counts (Priority 14: hit visibility). Useful for confirming a repeat scan is served from cache or measuring a query’s locality after reset_page_cache_stats.

pub fn reset_page_cache_stats(&self)

Zero the raw-page-cache hit/miss counters.

pub fn run_ids(&self) -> Vec<u128>

The run IDs in level order (Phase 15.5: used by the Arrow IPC shadow to key shadow files and detect stale shadows).

pub fn single_run_is_clean(&self) -> bool

Whether the single run (if exactly one) is clean — i.e. has RUN_FLAG_CLEAN set (Phase 15.5: the shadow is zero-copy only for clean runs).

pub fn query_columns_native_cached( &mut self, conditions: &[Condition], projection: Option<&[u16]>, snapshot: Snapshot, ) -> Result<Option<Vec<(u16, NativeColumn)>>>

Phase 19.1 + hardening (c): a cached form of Table::query_columns_native. The cache key embeds the snapshot epoch so two queries at different pinned snapshots never share an entry; invalidation is fine-grained — a commit() drops only entries whose footprint intersects a deleted RowId or whose condition-columns intersect a mutated column. On a miss the underlying query_columns_native runs and the result is cached as typed NativeColumns. Returns None exactly when the non-cached path would (conditions not pushdown-served). Strictly additive — callers wanting fresh results keep using query_columns_native.

pub fn query_cached(&mut self, q: &Query) -> Result<Vec<Row>>

Phase 19.1 + hardening (c): a cached form of Table::query. The cache key is epoch-independent; invalidation is fine-grained (see Table::query_columns_native_cached). On a hit returns the cached rows (no re-resolve, no re-decode).

pub fn query_columns_native_traced( &mut self, conditions: &[Condition], projection: Option<&[u16]>, snapshot: Snapshot, ) -> Result<(Option<Vec<(u16, NativeColumn)>>, QueryTrace)>

Self::query_columns_native with a captured crate::trace::QueryTrace.

pub fn query_columns_native_cached_traced( &mut self, conditions: &[Condition], projection: Option<&[u16]>, snapshot: Snapshot, ) -> Result<(Option<Vec<(u16, NativeColumn)>>, QueryTrace)>

Self::query_columns_native_cached with a captured crate::trace::QueryTrace (records result-cache hits too).

pub fn native_page_cursor_traced( &self, snapshot: Snapshot, projection: Vec<(u16, TypeId)>, conditions: &[Condition], ) -> Result<(Option<NativePageCursor>, QueryTrace)>

Self::native_page_cursor with a captured crate::trace::QueryTrace.

pub fn native_multi_run_cursor_traced( &self, snapshot: Snapshot, projection: Vec<(u16, TypeId)>, conditions: &[Condition], ) -> Result<(Option<MultiRunCursor>, QueryTrace)>

Self::native_multi_run_cursor with a captured crate::trace::QueryTrace.

pub fn count_conditions_traced( &mut self, conditions: &[Condition], snapshot: Snapshot, ) -> Result<(Option<u64>, QueryTrace)>

Self::count_conditions with a captured crate::trace::QueryTrace.

pub fn query_traced(&mut self, q: &Query) -> Result<(Vec<Row>, QueryTrace)>

Self::query with a captured crate::trace::QueryTrace.

pub fn query_columns_native( &mut self, conditions: &[Condition], projection: Option<&[u16]>, snapshot: Snapshot, ) -> Result<Option<Vec<(u16, NativeColumn)>>>

Predicate pushdown: resolve conditions via indexes to find the matching row-id set, then decode only those rows’ columns — not the whole table. Returns None if the conditions can’t be served by indexes (caller falls back to a full scan). This is the fast path for WHERE col = 'value'.

pub fn native_page_cursor( &self, snapshot: Snapshot, projection: Vec<(u16, TypeId)>, conditions: &[Condition], ) -> Result<Option<NativePageCursor>>

Build a lazy, page-aware NativePageCursor for the single-run fast path. MVCC visibility and predicate survivor resolution are settled up front (so they see the live indexes under the DB lock); the cursor then owns the reader and decodes only the projected columns of pages that contain survivors, lazily. This is the fused-predicate + page-skip + late-materialization scan.

Phase 13.1: the memtable / mutable-run overlay is now handled. Rows with a newer version in the overlay are excluded from the run’s page plans (their run version is stale); the overlay rows are pre-materialized and appended as a final batch via NativePageCursor::new_with_overlay.

Returns None only for multiple sorted runs; the caller falls back to the materialize-then-stream scan for that layout.

pub fn native_multi_run_cursor( &self, snapshot: Snapshot, projection: Vec<(u16, TypeId)>, conditions: &[Condition], ) -> Result<Option<MultiRunCursor>>

Generalizes Self::native_page_cursor (single-run) to arbitrary run counts via a k-way merge by RowId. Cross-run MVCC resolution (newest visible version per RowId) and predicate survivor resolution are settled up front from the cheap system columns + global indexes; the cursor then lazily decodes the projected data columns of just the pages that own survivors, each page at most once. The memtable / mutable-run overlay is materialized and yielded as a final batch (mirroring the single-run path).

Returns None only when there are no runs at all (caller falls back).

pub fn scan_cursor( &self, snapshot: Snapshot, projection: Vec<(u16, TypeId)>, conditions: &[Condition], ) -> Result<Option<Box<dyn Cursor>>>

Native vectorized aggregate over a (possibly filtered) column on the single-run fast path (Phase 7.2). Resolves survivors via the same page-pruned cursor as the scan, then accumulates the aggregate in one pass over the typed buffer — no Value, no Arrow RecordBatch.

column is None for COUNT(*). Returns Ok(None) when the fast path does not apply (multi-run / non-empty memtable); the caller scans. Open the streaming Cursor matching the current run layout: the single-run page cursor when there is exactly one sorted run, otherwise the multi-run k-way merge cursor. Both fuse the predicate, skip non-surviving pages, and fold the memtable / mutable-run overlay, so callers stay columnar end-to-end and never materialize Rows. Returns None when no cursor applies (e.g. an overlay-only table with no sorted run), leaving the caller to fall back.

This is the single source of truth for layout-aware cursor selection, shared by the column scan (Self::query_columns_native / the SQL provider) and the aggregate path (Self::aggregate_native). New streaming consumers should build on this rather than re-deciding the cursor by run count.

pub fn aggregate_native( &self, snapshot: Snapshot, column: Option<u16>, conditions: &[Condition], agg: NativeAgg, ) -> Result<Option<NativeAggResult>>

Native vectorized aggregate over a (possibly filtered) column, in one pass over the typed buffers — no Value, no Arrow batch. Layout-agnostic: survivors stream through Self::scan_cursor (single- or multi-run, overlay-folded), so the same path serves every sorted-run layout.

column is None for COUNT(*). Order of attempts:

1. Single clean run + no WHERE ⇒ MIN/MAX/COUNT(col) straight from page min/max/null_count (no decode).
2. COUNT(*) ⇒ survivor cardinality from the cursor’s page plans.
3. Otherwise accumulate the projected column over the cursor.

Returns Ok(None) (caller scans) when no native path applies: an overlay-only table with no sorted run, or a non-numeric column.

pub fn count_distinct_from_bitmap(&self, column_id: u16) -> Result<Option<u64>>

Phase 7.1c: exact COUNT(DISTINCT col) from the bitmap index’s partition cardinality — the number of distinct indexed values — with no scan. Each distinct value is one bitmap key; under the insert-only invariant (empty overlay, single run, live_count == row_count) every key has at least one live row, so the key count is exact. NULL is excluded from COUNT(DISTINCT), so a null key (from an explicit Value::Null put) is discounted. Returns None (caller scans) without a bitmap index on the column or when the invariant does not hold.

pub fn aggregate_incremental( &mut self, cache_key: u64, conditions: &[Condition], column: Option<u16>, agg: NativeAgg, ) -> Result<IncrementalAggResult>

Incremental aggregate over the live table (Phase 8.3). For an append-only table, a warm cache entry (same cache_key) lets the result be refreshed by aggregating only the newly inserted rows (row-id watermark delta) and merging, instead of a full recompute. The caller supplies a stable cache_key (e.g. a hash of the SQL + projection); distinct queries must use distinct keys.

Returns IncrementalAggResult with the merged state and whether the delta path was taken. A single delete (ever) disables the incremental path for the table, so correctness never relies on append-only behavior that deletes invalidate.

pub fn approx_aggregate( &self, conditions: &[Condition], column: Option<u16>, agg: ApproxAgg, z: f64, ) -> Result<Option<ApproxResult>>

Approximate COUNT/SUM/AVG over a filtered set, computed from the in-memory reservoir sample (Phase 8.2). Returns a point estimate plus a normal-theory confidence interval at the supplied z-score (1.96 ≈ 95 %).

The WHERE predicates are evaluated exactly on each sampled row (so LIKE/FM and equality/range contribute no index bias); Ann/SparseMatch are index-defined and resolved once to a row-id set that sampled rows are tested against. Ok(None) when there is no usable sample.

pub fn exact_column_stats( &self, _snapshot: Snapshot, projection: &[u16], ) -> Result<Option<HashMap<u16, ColumnStat>>>

Exact per-column statistics for the analytical aggregate fast path (Phase 7.1: MIN/MAX/COUNT(col) from page stats). Returns None unless the table is effectively insert-only at snapshot — empty memtable, a single sorted run, and live_count == run.row_count() — so the run’s page min/max/null_count are exact (no tombstoned or superseded versions skew them). Under deletes/updates the caller falls back to scanning.

pub fn dir(&self) -> &Path

pub fn schema(&self) -> &Schema

pub fn alter_column( &mut self, column_name: &str, change: AlterColumn, ) -> Result<ColumnDef>

pub fn add_column( &mut self, name: &str, ty: TypeId, flags: ColumnFlags, ) -> Result<u16>

Add a column to the schema (schema evolution). Existing runs simply read back as null for the new column until re-written. Persists the new schema and manifest. The caller supplies the full ColumnFlags so migrations can add PRIMARY KEY / AUTO_INCREMENT columns correctly.

pub fn add_learned_range_index(&mut self, column_name: &str) -> Result<()>

Declare a LearnedRange (PGM) index on an existing numeric column and build it immediately from the current sorted run (Phase 13.3). After this, Condition::Range / Condition::RangeF64 on that column resolve survivors sub-linearly (O(log segments + log ε)) instead of scanning the full column.

Requires exactly one sorted run (call after flush). The index is rebuilt automatically on subsequent flushes.

pub fn set_sync_byte_threshold(&mut self, threshold: u64)

Tuning knob for the WAL auto-sync threshold. A no-op on a mounted table (the shared WAL’s durability is governed by the group-commit coordinator).

pub fn page_cache_flush(&self)

Flush all live page-cache entries to the persistent _cache/ backing directory (best-effort). Useful before a clean shutdown so hot pages survive restart.

pub fn page_cache_len(&self) -> usize

Number of entries currently in the shared page cache (diagnostic).

pub fn decoded_cache_len(&self) -> usize

Number of entries currently in the shared decoded-page cache (Phase 15.4 diagnostic).

pub fn drain_memtable_sorted(&mut self) -> Vec<Row>

Drain the live memtable (prototype/testing helper used by the flush path demos). Prefer Table::flush for the durable path.

impl Table

pub fn gc(&self) -> Result<GcReport>

Remove orphan run files (not in the manifest) and all but the active WAL segment. Safe to run any time; readers pin snapshots, not files.

pub fn check(&self) -> Result<CheckReport>

Verify every manifest-referenced run’s footer checksum.

pub fn doctor(&mut self) -> Result<DoctorReport>

Best-effort repair: drop any manifest-referenced run that fails its checksum, so the table reopens in a consistent (if lossy) state.