Struct CommutativeLog 

pub struct CommutativeLog {
    pub write_offset: AtomicU64,
    pub capacity: u64,
    /* private fields */
}

Fields

write_offset: AtomicU64

Byte offset one past the last reserved (but possibly not yet written) record.

Advanced atomically by put_versioned() via fetch_add. May be ahead of committed_offset while in-flight writes are still in progress.

capacity: u64

Total capacity of the mmap file in bytes. Writes that would exceed this return DbError::LogFull.

Implementations

impl CommutativeLog

pub fn open(path: &Path, size: u64) -> DbResult<Self>

Open (or create) a log file at path with a maximum size of size bytes.

path — the file is created if it does not exist. On restart the same path must be passed to recover the previous committed state.

size — the mmap (and underlying file) is pre-allocated to exactly this many bytes. Choose a value large enough to hold one full block’s worth of writes.

Crash recovery

The first 8 bytes of the file always hold the last persisted committed_offset as a little-endian u64. On open, if that value is within (8, size] both write_offset and committed_offset are initialised to it, so the next commit_fold resumes from where the previous run left off. Any unreachable bytes between the persisted offset and the physical end of the file are simply ignored.

pub fn put_versioned( &self, key: [u8; 32], value: &[u8], prev_off: u64, height: u64, ) -> DbResult<u64>

Append a key-value record with an MVCC chain pointer and block height.

This is the main hot-path write operation.

Atomicity guarantee

A single fetch_add(total, AcqRel) on write_offset reserves the byte range [off, off + total) atomically. Because fetch_add is unconditional (no CAS loop), there is zero contention between concurrent writers — each thread gets its own disjoint slice of the log. On capacity overflow the reservation is reversed with a compensating fetch_sub and DbError::LogFull is returned.

In-flight tracking

Between reserving the slot and finishing the write, inflight is incremented. commit_fold_until() and reset() spin-wait on inflight == 0 before reading or rewinding the log, ensuring they never observe a partially-written record.

Record layout (packed Hdr struct)

The 24-byte header is written as a single repr(C, packed) struct in one write_unaligned call — one cache-line store instead of four separate writes. The key (32 bytes) and value (N bytes) follow immediately.

• prev_off — mmap offset of the most recently committed record for this key, or 0 for the first write. Patched after commit if a prior record is discovered during commit_fold_until.
• height — the block height at write time, used for point-in-time queries.

pub fn del_versioned( &self, key: [u8; 32], prev_off: u64, height: u64, ) -> DbResult<u64>

Append a tombstone record for key at block height.

A tombstone uses the tombstone magic number (CTOMB) and vlen = 0 so it occupies the minimum possible space in the log (header + key = 56 bytes). The fold thread recognises the tombstone magic, XORs out the key’s existing accumulator contribution, and marks the index entry as deleted.

After commit() + ack.wait(), get(key) returns NotFound and the key is excluded from scan_prefix and scan_from_reverse results.

Errors

Returns crate::DbError::LogFull if the active segment has no room.

pub fn commit_fold( &self, index: &ShardIndex, prev_acc: [u8; 32], ) -> DbResult<([u8; 32], [u8; 32])>

Fold the entire dirty range into the index and return the new state root.

Equivalent to commit_fold_until(index, prev_acc, write_offset, 0). The dirty range is [committed_offset, write_offset) at the moment of the call — any writes that land after the snapshot of write_offset are deferred to the next commit.

Returns (new_accumulator, blake3(new_accumulator)).

pub fn commit_fold_until( &self, index: &ShardIndex, prev_acc: [u8; 32], end_offset: u64, log_id: u64, ) -> DbResult<([u8; 32], [u8; 32])>

Fold the dirty range [committed_offset, end_offset) into the index.

The end_offset parameter lets the caller pin a snapshot of the write frontier so that writes that race in after the block boundary are not accidentally included in this commit.

log_id is passed through to the shard index for bookkeeping (e.g. tracking which segment last wrote each key).

Step-by-step

1. Acquire commit lock — only one commit may run at a time.

2. Drain in-flight writes — spin on inflight == 0 so every put_versioned() that reserved an offset inside [start, end) has finished writing its data.

3. Parse dirty slice — walk the mmap from start to end, checking the magic bytes of each record. Collect (offset, key, vlen) tuples into a Vec without copying any value data (zero-copy parse).

4. Parallel blake3 hashing — rayon::par_iter maps each record to (key, blake3(value), offset). The value bytes are read via a raw pointer into the mmap (safe because step 2 ensures no concurrent writes).

5. Sequential dedup — sort by offset; build a per_block_count map to track how many times each key appeared; keep only the last (key → value_hash, offset) entry so the index always holds the highest-offset write for each key.

6. Accumulator update + index upsert — for each surviving key:

   • If the key already exists in the committed index, read its old value from the mmap and XOR out its key XOR blake3(old_value) contribution from prev_acc.
   • Patch the prev_offset field of the new record with the old offset (completing the MVCC chain).
   • XOR the new key XOR blake3(new_value) contribution into new_acc.
   • Batch updates by shard index (key[0]) for parallelism.

7. Parallel shard upserts — rayon::par_iter over the 256 shard batches; each shard acquires only its own lock.

8. Advance committed pointer — store end_offset into committed_offset and persist it to the first 8 bytes of the mmap for crash recovery, then issue an async flush.

Returns (new_accumulator, blake3(new_accumulator)).

pub fn flush(&self)

Issue an asynchronous flush of the mmap to the underlying file.

The OS will write dirty pages back to disk in the background. Use this after commit_fold_until to reduce the window of data loss on a crash.

pub fn write_offset(&self) -> u64

Return the current write frontier (one past the last reserved byte).

May be ahead of committed_offset while in-flight writes are landing.

pub fn committed_offset(&self) -> u64

Return the current committed frontier (one past the last folded byte).

Everything below this offset is reflected in the index.

pub fn mmap_ptr(&self) -> *const u8

Return the raw base pointer of the mmap region.

Used for zero-copy reads during log replay and by commit_fold_until to hash value bytes without copying them.

pub fn reset(&self)

Reset the log for reuse at the start of the next block.

Spins until all in-flight put_versioned() calls have finished writing, then rewinds both write_offset and committed_offset to 8 (just past the 8-byte crash-recovery header) and zeroes the header bytes so the next open() sees a clean starting offset.

pub fn set_committed(&self, off: u64)

Forcibly advance committed_offset to off.

Also advances write_offset if it is currently behind off, ensuring the invariant write_offset >= committed_offset always holds. Used during segment replay to restore the committed frontier without re-running a fold.

impl CommutativeLog

pub fn value_ref(self: &Arc<Self>, off: usize) -> Option<MmapRef>

Return a zero-copy MmapRef for the value at off, or None if the record is invalid or is a tombstone.

The returned reference borrows the mmap through the Arc — no heap allocation is needed. Use this on the hot read path when the caller only needs to inspect or hash the value without keeping a copy.