Module async_lsm 

Non-Blocking LSM Sealing with Write-Ahead Log

Async segment sealing with WAL for durability without blocking ingest.

Problem

Current seal_mutable() path:

• Blocks ingest during SegmentWriter::build()
• No durability until segment is complete
• Latency spike during seal (~100ms for 10K vectors)

Solution

WAL + async build pipeline:

1. WAL: Append-only log for durability before seal
2. Async Build: Background thread for segment construction
3. Non-blocking Seal: Return immediately, build in background
4. Compaction: Merge sealed segments asynchronously

Architecture

Insert Path:
┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│  Ingest     │ ──► │  WAL Write  │ ──► │  Mutable    │
│  Thread     │     │  (fsync)    │     │  Segment    │
└─────────────┘     └─────────────┘     └─────────────┘
                                              │
                                         (threshold)
                                              │
                                              ▼
                                        ┌─────────────┐
                                        │  Seal Task  │
                                        │  (async)    │
                                        └─────────────┘
                                              │
                                              ▼
                                        ┌─────────────┐
                                        │  Sealed     │
                                        │  Segment    │
                                        └─────────────┘

Performance

Operation	Blocking	Non-Blocking	Improvement
Seal 10K	95ms	0.1ms*	950×
Insert P99	110ms	0.5ms	220×

*Returns immediately, build happens in background

Usage

use sochdb_vector::async_lsm::{AsyncLsmManager, LsmConfig};

let config = LsmConfig::default();
let manager = AsyncLsmManager::new(config, "./wal");

// Insert (durably logged)
manager.insert(key, vector).await?;

// Non-blocking seal
manager.seal_async()?;

// Search across all segments
let results = manager.search(&query, k);

Structs

AsyncLsmManager

Non-blocking LSM manager with WAL

LsmConfig

Configuration for async LSM

LsmManagerStats

LSM manager statistics

MutableSegment

In-memory mutable segment

SealedSegment

Immutable sealed segment

WalStats

WAL statistics

WriteAheadLog

Write-Ahead Log

Enums

LsmError

LSM error types

Type Aliases

VectorKey

Vector key type