ToonDB Storage Layer
Log-Structured Column Store (LSCS) with transaction-aware WAL for TOON-native data.
Runtime Modes
This crate supports two runtime modes:
Embedded Sync Mode (like SQLite)
For embedded deployments without async runtime:
= { = "...", = false, = ["embedded-sync"] }
Benefits:
- ~500KB smaller binary
- No async runtime overhead
- Simpler embedded integration
Async Mode (default, for servers)
For server deployments with async I/O:
= { = "..." } # async enabled by default
Benefits:
- Better scalability for concurrent connections
- Non-blocking I/O for server workloads
Novel Components
-
LSCS (
lscs): Log-Structured Column Store - columnar variant of LSM with schema-aware compression and column-aware compaction for reduced write amplification. -
Transaction WAL (
txn_wal): ACID-compliant Write-Ahead Log with transaction boundaries, commit/abort markers, and crash recovery. -
StorageEngine Trait (
storage_engine): Pluggable storage backend abstraction enabling 80% I/O reduction for columnar projections (Task 1). -
Page Manager (
page_manager): TOON file format with magic header and O(1) page allocation (Task 8). -
Columnar Compression (
columnar_compression): Type-aware encoding with dictionary, RLE, and delta compression for 2-4× storage reduction (Task 9).
Utility Components
- Bloom Filters (
bloom): Probabilistic existence checks - Block Checksums (
block_checksum): Data integrity validation - Compression (
compression): LZ4/Zstd compression - Sketches (
sketches): Approximate algorithms (HyperLogLog, CountMin, DDSketch)