solistore

An embedded, ordered key-value store built as an LSM tree, in pure Rust. It fsyncs every commit by default and makes concurrent commits share that fsync. It is written to replace RocksDB under SoliDB.

   commits ──► group commit ──► one WAL write + one fsync per group
                    │
                    ▼
   memtable (lock-free skiplist, nodes in an arena) ──seal──► immutable memtables
                                                                  │ flush
                                                                  ▼
   L0  ▢ ▢ ▢ ▢      overlapping files, newest first
   L1  ▢▢▢▢▢▢▢▢     512 MiB of sorted, non-overlapping files ─┐
   L2  ▢▢▢▢▢▢▢▢▢…   ×10 per level                             ├─ leveled compaction,
   …                                                          ┘  parallel subcompactions

Status — not used in production yet. solistore passed its go/no-go gates against RocksDB 11.8.1 on 2026-10-14. SoliDB now runs on it behind a feature flag, where it is ahead of RocksDB or level on every phase measured (see Benchmarks). The on-disk format may still change before 1.0. See the integration plan.

Why another storage engine

SoliDB used RocksDB, which has three costs:

• Build time. Its C++ was most of SoliDB's build: CI measured 95 minutes of compilation for 18 seconds of tests. It also needs a cross g++ toolchain for musl and aarch64, and Windows was limited to a compile-only check.
• Memory. It is hard to account for and harder to bound, with per-column-family structures that grow with the number of collections.
• Durability. SoliDB's ordinary writes were not fsynced, because an fsync per write was too slow.

solistore answers each of these with a design choice:

• Pure Rust, no -sys crate. Nothing in the dependency graph needs a C compiler, and CI enforces it (scripts/check_no_c_toolchain.sh).
• One memory budget. A shared block cache holds data blocks. Index and filter memory is charged to the same cache.
• Durable commits that stay fast. Writers queue behind a leader, which writes the whole group to the WAL and fsyncs once for all of them.
• Cheap column families. All of them share one LSM. Creating or dropping one is a single MANIFEST record, and opening a database with thousands of them reads no table file.

Features

• Column families created and dropped at run time.
• Atomic write batches across column families: readers see a batch entirely or not at all.
• get returns Bytes, so a block-cache hit is served without a copy. multi_get is also available.
• Bounded iterators in both directions, with RocksDB's Start, End and From(key, direction) modes; From(k, Reverse) is seek-for-prev.
• Range deletes:
  • Kept at the version level rather than in tables.
  • A file wholly covered by a delete is dropped without being read.
  • Files that straddle the deleted range are compacted first.
• Leveled compaction with parallel subcompactions and trivial moves, and level targets set from the bottom level (RocksDB's level_compaction_dynamic_level_bytes): a small database is one sorted run, and obsolete versions stay at about a tenth of the data. Writers are slowed and then stalled on level-0 backlog.
• Point-in-time checkpoints made of hard links.
• Metrics:
  • Memtable bytes, table-reader bytes, block-cache usage and files per level.
  • Approximate size of a key range.
  • Group-commit sizes, stall times and compaction write volume.
• Two durability modes:

In either mode, a write with WriteOptions { sync: true } waits for its own fsync. An fsync failure poisons the database, which then refuses further writes rather than retrying (see "fsyncgate").

Quick start

use std::time::Duration;
use solistore::{Db, Direction, Durability, IteratorMode, Options, WriteBatch};

let db = Db::open(dir, Options {
    durability: Durability::Eventual { sync_interval: Duration::from_millis(100) },
    ..Options::default()
})?;

let users = db.create_cf("users")?;
let index = db.create_cf("users_by_email")?;

// Atomic across column families.
let mut batch = WriteBatch::new();
batch.put(&users, b"u:1", br#"{"name":"Ada","email":"ada@example.com"}"#);
batch.put(&index, b"ada@example.com", b"u:1");
db.write(batch)?;

let id = db.get(&index, b"ada@example.com")?;          // Option<Bytes>

for item in db.iterator(&users, IteratorMode::From(b"u:", Direction::Forward))? {
    let (key, value) = item?;
}

let mut batch = WriteBatch::new();
batch.delete_range(&users, b"u:", b"u;");               // [start, end)
db.write(batch)?;

db.checkpoint(dir.join("backup"))?;

The full program is crates/solistore/examples/quickstart.rs. Run it with cargo run -p solistore --example quickstart.

How it is tested

A storage engine is only as good as its crash behaviour, so most of the test code exercises failure paths:

• Model-based property tests. These run random sequences against a BTreeMap model: puts, deletes, range deletes, batches, gets, bounded scans in both directions, flushes, compactions, reopens, column-family create and drop, and opening a checkpoint. Tiny memtables and files force every background path to run. Set PROPTEST_CASES to go deeper; 20,000 cases found a WAL-recycling race that 256 did not.
• A deterministic crash sweep. A fault-injecting filesystem (FaultFs) crashes the database at every I/O operation of a workload and leaves torn, unsynced tails. After each crash, the reopened database must equal the model at the last acknowledged commit.
• Fault injection. FaultFs also makes fsyncs fail or run slow, to exercise poisoning and group commit.
• Process kills. The benchmark's crash test SIGKILLs a writer 100 times mid-load and checks every acknowledged batch. Its visibility test checks that no reader ever sees part of a batch.
• Mutation testing. Bugs are injected on purpose to check that the tests above catch them.
• Fuzzing. cargo-fuzz targets feed garbage to every decoder: log framing, write batches, MANIFEST edits, blocks, bloom filters, table files, and a database opened from well-framed records saying anything (fuzz/). Corrupt input must be an error, never a panic or a huge allocation; CI runs each target briefly.
• loom. The commit path's two protocols, leader election in the commit queue and publication of finished groups, are model-checked under every interleaving of a few writers: one leader at a time, every batch handled once, nothing visible before it is inserted, a memtable switch waiting for the groups handed out, no lost wake-up. Mutations of each are caught.
• Miri. CI runs the tests of the memtable's skiplist and arena under Miri, the only unsafe in the crate besides one sync_file_range call (an FFI call Miri cannot run, which passes a file descriptor and two offsets and no pointer). They include concurrent inserts with readers, at sizes scaled down for Miri.

Benchmarks

All figures are against RocksDB 11.8.1 with SoliDB's production profile. Ratios are solistore throughput over RocksDB's; above 1.0 is faster. solistore runs in Eventual mode (100 ms), and RocksDB with SoliDB's sync=false writes, unless stated otherwise.

Inside SoliDB, whole server

The same SoliDB binary ran on each engine, with 1M documents and two indexes. The load came over SoliDB's native driver with the query cache off; the server had 8 cores, the client 8 others. Figures are medians of 3 runs.

SDBQL queries, 64 connections (8 for the scan and aggregate):

solistore also uses less CPU per operation, for example 27 µs per update against 32 µs. Peak RSS is 1.02–1.08× RocksDB's at 1M documents; at 5M, with 64 connections each reading 5,000 documents per query, it is 0.66× (1,355 MB against 2,042 MB), because point reads hand out a copy of a small value rather than a slice that keeps its 4 KiB block alive (bench/results/2026-10-15-point-read-copies). With 256 connections, RocksDB's p99 update latency is 4.4 ms and solistore's 3.0 ms.

Storage engines alone

bench/ runs a SoliDB-shaped workload on each engine directly: a write is the document plus three index entries, and the phases are gets, scans, updates, fsynced transactions and a ~1,232-collection catalog.

At 5M documents, medians of 3, the final series of 2026-11-01 (bench/results/2026-11-01-beat-rocksdb/final, both engines alternating on the same machine):

Ratios move with RocksDB's own spread from run to run. Its bulk-load median went from 428k to 512k documents a second between that day's series, which is why the load reads 1.31× in 2026-11-01-all and 1.15× here. The A/B runs of these changes left solistore's own load within 2%.

Each fsynced-commit phase is 2,000 commits, and one or two stalls a run decide it, for both engines: table writes from flush and compaction share the disks with the WAL's fsync. Over every comparable run of the day (16 of solistore, 9 of RocksDB) the medians are 1.43× with one thread and 1.62× with eight.

The single-thread phase was the one behind at b29426d (0.89×, bench/results/2026-11-01-all). The first fsync after the unsynced load flushed up to 100 ms of WAL, about 30 MB. The eventual mode's background syncer now also fsyncs as soon as wal_bytes_per_sync (1 MiB) waits.

In full durability mode (2026-11-01-all, b29426d) solistore fsyncs every commit, where RocksDB's sync=false writes fsync none. That alone makes the load 0.49× and the updates 0.48×. Its fsynced commits run at 2–3× RocksDB's all the same.

On the go/no-go series of 2026-10-14 (bench/results/2026-10-14-m7), before the commit pipeline, the arena memtable and dynamic level targets, the bulk load was 0.81× and the updates 0.82×.

The gates that block integration all passed:

• 100 SIGKILL crashes with 14.98M acknowledged writes: 0 lost, 0 torn batches.
• 0 visibility violations in 47.7M checks.
• Reopen under 0.005 s.
• 202 MB reclaimed after a drop, ending 10% smaller than RocksDB.
• Peak RSS 0.98× RocksDB's, disk 0.95×.

Against SurrealKV

SurrealKV 1.0.0 is in the same bench, with the budget the other engines get: 512 MiB block cache, 64 MiB memtable, 4 KiB blocks and Snappy. At 1M documents (single runs, bench/results/2026-10-14-surrealkv):

With every commit fsynced, solistore writes about twice as fast as SurrealKV (load 170k against 86k, updates 174k against 90k).

With its factory settings (32 MiB cache, no compression), SurrealKV's uniform gets fall to 307k a second. Its published benchmark used 500,000 keys, which fit in its memtable and cache.

Raw results are in bench/results/, as JSON lines per phase.

Building

cargo build --release
cargo test --profile ci          # optimised, no LTO: crash and property tests run in seconds
cargo clippy --all-targets -- -D warnings

The benchmark is a separate workspace, so RocksDB's C++ never enters the engine's build:

cargo build --release --manifest-path bench/Cargo.toml        # both engines
bench/target/release/engine-bench run --engine solistore --dir /fast/disk/bench
bench/gate.sh                                                  # the full go/no-go run

MSRV is Rust 1.91, SoliDB's floor. CI checks Linux, macOS, Windows and musl.

Repository layout

crates/solistore/src/
  db.rs           open, commit pipeline, reads, column families, checkpoints
  batch.rs        WriteBatch and its WAL encoding
  log.rs          record framing shared by the WAL and the MANIFEST
  memtable.rs     memtable over the skiplist, range tombstones
  skiplist.rs     concurrent skiplist with keys, values and links in one arena
  sst/            table format: blocks, bloom filters, block cache, reader/builder
  version.rs      versions, super-versions (arc-swap), level iterators
  manifest.rs     version edits and recovery
  flush.rs        memtable → level 0
  compaction.rs   pickers, subcompactions, trivial moves
  iter.rs         merging iterator and the user-facing iterator
  fs.rs           filesystem trait: StdFs, and FaultFs for crash tests
bench/            SoliDB-shaped benchmark against RocksDB
docs/             integration plan for SoliDB

Roadmap

• M0–M7: WAL and group commit, tables, compaction, range tombstones, block cache, checkpoints and metrics, then the go/no-go benchmark.
• SoliDB on solistore behind a feature, with the engine recorded per data directory. Ahead of RocksDB or level on every measured phase.
• A migration tool from RocksDB data directories (solidb-migrate-engine, SoliDB 2.4).
• A trial on real workloads, then removing RocksDB from SoliDB.
• Crash tests on NTFS.

Licence

O'Saasy License © 2024-2026 SoliSoft SARL, the same as SoliDB. It is MIT-style, except that you may not offer the software itself as a competing hosted service.