Struct WriteEngine 

pub struct WriteEngine { /* private fields */ }

Write engine coordinator

Orchestrates WAL, memtable, and SSTable flushing for write operations. This is the primary public API for all write operations in CQLite.

Durability contract: you MUST call close

Drop is not a flush. Rows written with write / execute live in the in-memory memtable (and the WAL) until a flush turns them into an SSTable. Only close (or an explicit flush) guarantees the memtable is persisted to a Data.db. Because Tokio has no async drop, Drop CANNOT flush — doing so would require a block_on inside drop, which is forbidden (issue #1693/AG3). An engine dropped with a non-empty memtable logs a warn! and leaves those rows recoverable only via WAL replay on the next startup.

Embedders (and every long-lived writer) MUST therefore call engine.close().await for a graceful shutdown — e.g. from a SIGINT handler — before the process exits.

Thread Safety

WriteEngine follows a single-writer model. It is NOT thread-safe and should be used from a single thread or protected by external locking. The closed flag uses atomic operations for safe concurrent access checking.

Example

use cqlite_core::storage::write_engine::{WriteEngine, WriteEngineConfig, Mutation};
use std::path::PathBuf;

// Create configuration
let config = WriteEngineConfig::new(
    PathBuf::from("data"),
    PathBuf::from("wal"),
    schema
);

// Create engine
let mut engine = WriteEngine::new(config)?;

// Write a mutation
engine.write(mutation)?;

// Execute CQL statement
engine.execute("INSERT INTO users (id, name) VALUES (1, 'Alice')")?;

// Flush to SSTable
engine.flush()?;

// Close cleanly
engine.close()?;

Implementations

impl WriteEngine

Export implementation methods (added to WriteEngine)

pub async fn export_sstable( &mut self, output_dir: &Path, options: ExportOptions, ) -> Result<ExportReport>

Export an SSTable suitable for distribution

This method performs the following steps:

1. Flushes the memtable if not empty
2. Performs full compaction (if enabled) to merge all L0 files
3. Copies the resulting SSTable to the output directory with Cassandra naming
4. Validates the exported SSTable (if enabled)

Arguments

• output_dir - Directory where exported files will be written
• options - Export configuration

Returns

An ExportReport containing metadata about the export operation.

Errors

Returns an error if:

• Engine has been closed
• Flush fails
• Compaction fails
• File copy fails
• Validation fails

Example

let options = ExportOptions::new("test_ks", "users", 1);
let report = engine.export_sstable(Path::new("/export"), options).await?;
println!("Exported {} partitions ({} bytes)", report.partition_count, report.total_size());

impl WriteEngine

pub fn set_merge_policy(&mut self, policy: Box<dyn MergePolicy>) -> Result<()>

Set the merge policy for background compaction (M5.2, Issue #383)

Arguments

• policy - Merge policy implementation (e.g., STCS, LCS, TWCS)

pub fn maintenance_stats(&self) -> CompactionStats

Return cumulative compaction statistics (M5.2, Issue #474)

Returns a snapshot of the lifetime totals accumulated across all compaction cycles that have completed since the WriteEngine was created. The snapshot is cheaply cloneable and safe to inspect from any thread (no lock required, because WriteEngine itself is not Sync).

Example

let stats = engine.maintenance_stats();
println!(
    "Completed {} compactions, merged {} rows, wrote {} bytes",
    stats.compactions_completed,
    stats.rows_merged,
    stats.bytes_written,
);

pub fn maintenance_step( &mut self, budget: Duration, ) -> Result<MaintenanceReport>

Perform incremental maintenance work (M5.2, Issue #384)

This method performs background compaction work within a time budget. It can be called repeatedly from a background thread or task scheduler to make incremental progress on compaction.

Runtime contexts

This is a synchronous method, but its internal async-to-sync bridge is runtime-aware (see [merge::block_on_async]), so it is safe to call from either a plain synchronous context or from within an active Tokio runtime — including #[tokio::main]/#[tokio::test] worker threads and async fn callers. Prior to Issue #587 calling it from inside a runtime panicked with “Cannot start a runtime from within a runtime” once a merge had input SSTables to read. The sync signature is preserved so the CLI and Python bindings can keep calling it directly. (The Node binding wraps it in spawn_blocking, which remains correct.)

Behavior

1. If no active merge exists, consult the merge policy for work
2. If merge work is available, start a new merge
3. Process the active merge until budget is exhausted
4. Return progress report

Invariants

• Budget is honored within 10% tolerance
• At least one CLUSTER GROUP is processed per call (minimum progress guarantee; issue #1668 stage 4 loosened this from “at least one partition” — a single oversized partition no longer blocks the budget for its entire duration)
• Merge state is preserved across calls for resumption, INCLUDING a partition whose cluster-group drain was paused mid-way (ActiveMerge::pending_partition, issue #1668 stage 4) — resuming never re-computes or loses a row, and the writer always still receives one partition’s mutations in one write_partition call.

Budget Enforcement — the HONEST contract (issue #1667)

The budget is a target checked at boundaries, not a hard cap. It is honored within approximately 10% tolerance, checked BETWEEN CLUSTER GROUPS (issue #1668 stage 4) rather than only between whole partitions — a fat partition can yield control back to the budget check partway through its own drain instead of running to completion regardless of elapsed time. The tolerance ensures forward progress on each call while remaining responsive to time constraints.

Because the check fires only at those boundaries, a step can OVERSHOOT the budget honestly (time_spent reports the real elapsed time, never a value clamped to budget). Three residual sources of overshoot are NOT bounded by this issue and each is deferred to Q5’s fully-streaming step():

1. A single indivisible cluster group. The budget is checked between cluster groups, so one very large cluster group (e.g. a single wide clustering row, or an unclustered partition’s sole row) runs to completion once started.
2. The per-partition WRITE. On the buffered path the whole partition is written in one PartitionEnd pass; that write is not sub-divided by the budget.
3. The dropped-column survivor pre-pass (issue #1667). When the table has dropped columns, start_merge runs a FULL, one-shot merge scan (compute_surviving_dropped_columns) over every input BEFORE the first partition is emitted. It is not incremental, so it cannot be bounded by budget. This issue makes it HONEST rather than silent: its cost is measured, counted inside time_spent, reported distinctly as MaintenanceReport::pre_pass_time, and — critically — when the pre-pass alone already exhausts the budget, the partition loop is SHORT-CIRCUITED for that step (the merge stays pending and resumes on the next call). The pre-pass can therefore never silently precede an unbudgeted partition loop within the same step. Making the pre-pass itself incremental is Q5-adjacent follow-up work, explicitly out of scope here.

This issue (#1667) is documentation + accounting only: it does NOT change WHAT is compacted, so compaction output stays byte-identical (#921). Mid-partition bounding of sources 1 and 2 lands with Q5’s streaming step() and is NOT claimed to be fixed here.

Arguments

• budget - Maximum time to spend in this call

Returns

A report containing progress metrics and whether more work is pending.

Errors

Returns an error if:

• Engine has been closed
• Merge policy returns an error
• SSTable reading or writing fails

Example

use std::time::Duration;

// Background compaction loop
loop {
    let report = engine.maintenance_step(Duration::from_millis(100))?;

    if !report.pending_compaction {
        // No more work, sleep or exit
        break;
    }

    // Log progress
    println!("Merged {} rows in {:?}", report.rows_merged, report.time_spent);
}

impl WriteEngine

pub fn memtable_size(&self) -> usize

Get the current memtable size in bytes

pub fn memtable_row_count(&self) -> usize

Get the current memtable row count

pub fn wal_size(&self) -> u64

Get the current WAL size in bytes

pub fn generation(&self) -> u64

Get the current generation number

pub fn total_written(&self) -> u64

Return the cumulative number of rows written since the engine was opened (Issue #486).

This counter is incremented for every row that is successfully inserted into the memtable and is NOT reset on flush. It therefore represents the total write throughput for the current session.

Note: This counter is in-process only and resets to zero when the engine is re-opened. WAL replay rows (recovered from a previous crash) are NOT counted; only rows written through write() / write_async() during the current session are counted.

pub fn l0_count(&self) -> u64

Return the number of L0 SSTables successfully flushed since the engine was opened (Issue #486).

Incremented once per successful flush() call that produces a non-empty SSTable. This is an in-process counter and resets to zero when the engine is re-opened.

pub fn total_flushed_bytes(&self) -> u64

Return the cumulative bytes written to flushed L0 SSTables (Data.db plus all sibling components) since the engine was opened (issue #1620).

Incremented on every successful flush — including the automatic flushes the binding write path now performs via execute_flushing — so binding write stats stay accurate for automatic flushes, not only explicit flush() calls. In-process counter; resets to zero on re-open.

impl WriteEngine

pub fn new(config: WriteEngineConfig) -> Result<Self>

Create a new write engine

This initializes the WAL and memtable. If a WAL exists in the wal_dir, it will be replayed to recover in-flight writes.

Arguments

• config - Write engine configuration

Returns

A new WriteEngine ready to accept writes.

Errors

Returns an error if:

• WAL directory doesn’t exist
• Data directory doesn’t exist
• WAL replay fails

pub fn wal_recovery(&self) -> &RecoveryReport

Summary of the WAL crash-recovery replay performed when this engine was opened (issue #1391).

The mutations field has been drained into the memtable, so only the lossiness metadata remains. Use RecoveryReport::is_clean to detect a lossy recovery. A non-clean report means the raw WAL segment was preserved aside (as commitlog.wal.corrupt.<nanos>) so a subsequent flush cannot destroy the evidence.

pub fn write(&mut self, mutation: Mutation) -> Result<()>

Write a mutation to the write engine

This appends the mutation to the WAL for durability, then inserts it into the memtable. If the memtable exceeds the flush threshold, an automatic flush is triggered.

Note: Automatic flush is disabled when called from an async context. Use write_async() for async contexts with automatic flush support.

Arguments

• mutation - The mutation to write

Returns

Ok(()) on success, or an error if the write fails.

Errors

Returns an error if:

• Engine has been closed
• WAL append fails
• Memtable insert fails
• Automatic flush fails (sync context only)

pub async fn write_async(&mut self, mutation: Mutation) -> Result<()>

Write a mutation with async automatic flush support

This is the async version of write() that supports automatic flushing in async contexts. Use this method when calling from async code.

Arguments

• mutation - The mutation to write

Returns

Ok(()) on success, or an error if the write fails.

Errors

Returns an error if:

• Engine has been closed
• WAL append fails
• Memtable insert fails
• Automatic flush fails

pub fn execute(&mut self, statement: &str) -> Result<()>

Execute a CQL statement (INSERT, UPDATE, DELETE)

This parses the CQL statement and converts it to a mutation, then writes it using the write() method.

Arguments

• statement - CQL statement string

Returns

Ok(()) on success, or an error if parsing or writing fails.

Errors

Returns an error if:

• CQL parsing fails
• Statement is not a mutation (INSERT/UPDATE/DELETE)
• Mutation conversion fails
• Write fails

Example

engine.execute("INSERT INTO users (id, name) VALUES (1, 'Alice')")?;
engine.execute("UPDATE users SET name = 'Bob' WHERE id = 1")?;
engine.execute("DELETE FROM users WHERE id = 1")?;

pub async fn execute_flushing(&mut self, statement: &str) -> Result<u64>

Execute a DML statement and, if the memtable has crossed the flush threshold, await a REAL async flush (issue #1620, DECIDED: write_async).

This is the entry point for the Node/Python binding write path, which runs inside a Tokio runtime where the sync auto-flush in write() is intentionally skipped. It restores auto-flush there WITHOUT the surprise inline-flush latency the plain sync write()/execute() path avoids. Returns the number of mutations applied (N for BATCH, else 1).

pub async fn flush(&mut self) -> Result<Option<SSTableInfo>>

Force a flush of the memtable to SSTable

This writes all data in the memtable to a new SSTable generation, then truncates the WAL. The memtable is cleared after a successful flush.

Returns

Returns Some(SSTableInfo) if data was flushed, or None if the memtable was empty.

Errors

Returns an error if:

• Engine has been closed
• SSTable write fails
• WAL truncate fails

pub async fn close(&mut self) -> Result<()>

Close the write engine

This flushes any remaining data in the memtable to SSTable, syncs the WAL, then marks the engine as closed. After calling close(), the engine cannot be used for further writes.

This is the durability boundary. Drop does not (and cannot — no async drop in Tokio) flush; callers MUST close().await before exit to guarantee written rows reach an SSTable rather than relying on WAL replay (issue #1693). See the type-level docs on WriteEngine.

This method is idempotent - calling it multiple times is safe.

Returns

Ok(()) on success.

Errors

Returns an error if the final flush fails.

WAL-truncate handling during that flush is phase-aware (issue #1392):

• A truncate failure that leaves the WAL intact (it faulted before mutating the WAL) is logged and swallowed — the WAL stays a valid, idempotent replay marker, so no error is surfaced.
• A truncate failure after set_len(0) has already zeroed the WAL (WalTruncateFailedAfterCommit) is propagated. By then flush state has already been committed — the SSTable is durable and the generation has advanced — so the data is safe, but the error is surfaced so the caller knows the WAL is no longer a replay marker.

When the WAL is already empty (e.g. Durability::Disabled) the truncate phase is skipped, so no truncate-phase error can arise.

Trait Implementations

impl Debug for WriteEngine

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Drop for WriteEngine

Available on crate feature write-support only.

Safety-net Drop implementation for WriteEngine.

When a WriteEngine is dropped without calling close() (e.g. due to an early return or a panic), the OS would release the advisory lock anyway once the file descriptor is closed. This explicit Drop makes the release deterministic and logs a warning so callers can distinguish a normal shutdown from an ungraceful one.

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.