FlowDB

A high-performance embedded storage engine written in Rust, powered by an LSM-tree architecture with WAL, SSTables, and Bloom filters. Now includes JsonDB — a built-in IndexedDB-compatible JSON document database layer with ACID transactions and secondary indexes.

Fully synchronous API — no async runtime required. FlowDB uses plain OS threads for background maintenance, making it runtime-agnostic. Use it from Tokio, async-std, smol, or plain synchronous code without any wrappers.

Quick Start

[dependencies]
flowdb = "0.6"
serde_json = "1"  # optional, for JsonDB

LSM Engine

use flowdb::{Engine, Config, Record, Query};

let engine = Engine::open(Config::default())?;
engine.write_batch(&[Record::new("sensor:temp", 1_700_000_000_000, b"22.5".to_vec())])?;
for result in engine.query(Query::prefix("sensor:"))? {
    println!("{}", result.key_str());
}
engine.shutdown()?;

JsonDB Document Store

use flowdb::jsondb::{JsonDB, StoreSchema};
use serde_json::json;

let db = JsonDB::open(Default::default())?;
db.apply_store(&StoreSchema::new("users", "id")
    .with_index("by_email", &["email"], true)
)?;
db.put("users", json!({"id": "u1", "email": "a@b.com"}))?;
let doc = db.get("users", &json!("u1"))?;

Or with the derive macro:

use flowdb::ObjectStore;

#[derive(ObjectStore)]
#[store(name = "users", key_path = "id")]
struct User {
    id: String,
    #[index(unique)]
    email: String,
}

db.apply_schema::<User>()?;

Node.js (npm)

npm install @restsend/flowdb

const { FlowDB } = require('@restsend/flowdb')

const db = FlowDB.open({ dataDir: './data' })
await db.createObjectStore('users', 'id')
await db.createIndex('users', 'byEmail', ['email'], true)

await db.put('users', { id: 'u1', email: 'a@b.com' })
const doc = await db.get('users', 'u1')

const tx = db.transaction(['users'], 'readwrite')
tx.put('users', { id: 'u2', email: 'b@b.com' })
await tx.commit()
await db.close()

Zero Rust toolchain required. Full IndexedDB-compatible API with ACID transactions.

Tutorial → 05-nodejs

Full API reference → docs/api.md

Tutorials → docs/tutorials/

English

中文

Run the Examples

cargo run --example basic_engine
cargo run --example basic_jsondb
cargo run --example supabase_example
cargo run --example supabase-server   # then open http://localhost:3000

Benchmark Results (v0.4.0)

LSM Engine vs RocksDB (100K records, 128B values, batch=100, release, Apple M-series)

JsonDB Document Layer (release, Apple M-series, SyncMode::Always)

Note: Write throughput is bottlenecked by WAL fsync (SyncMode::Always). For higher throughput, use batch writes (transaction) or SyncMode::IntervalMs.

Features

LSM Engine

• LSM-tree storage with WAL (write-ahead log) for crash recovery
• Fully synchronous API — no async, no Tokio dependency, runtime-agnostic
• Background maintenance on OS threads — flush, compaction, GC via std::thread
• Per-record WAL checksums for corruption detection on replay
• Config validation — invalid configs rejected at startup
• Frozen memtable backpressure — writes stall when flush can't keep up
• Lazy scan iterator (RocksDB-style ScanIterator) for bounded-memory range scans
• Bloom filters for fast point query negative checks
• lz4 compression for all SST blocks (flush + compaction)
• Buffered WAL writes (256KB buffer) for reduced syscall overhead
• WAL pre-encoding outside the write lock for better concurrency
• Time-bucketed block index with binary search
• LRU block cache (64 shards) with true LRU eviction
• Vec-based active memtable for O(1) writes
• Size-tiered compaction with streaming heap merge (low memory footprint)
• Range tombstones for efficient bulk key-range deletion
• Garbage collection (TTL expiry) and point deletes
• Graceful shutdown — flushes WAL + memtables before exit
• Engine stats — structured counters + Prometheus-format metrics

JsonDB (IndexedDB-compatible layer)

• ACID transactions with atomic batch commit (OCC optimistic concurrency)
• Secondary indexes with automatic maintenance on CRUD
• Unique constraint enforcement on indexed fields
• Auto-increment primary keys
• Read-your-writes consistency within transactions
• Snapshot isolation via MVCC sequence numbers
• Schema persistence across restarts (automatic recovery)
• IndexedDB-like API — create_object_store, create_index, put, get, delete, transaction
• Index queries — point lookup (get_by_index) and range queries (range_by_index)
• Multi-field indexes via dotted key paths (e.g. "address.city")
• Primary key null-byte (\x00) validation — prevents composite-key separator injection

Configuration

use flowdb::Config;

let config = Config {
    data_dir: "./data".into(),
    memtable_size_mb: 64,
    wal_sync_mode: flowdb::SyncMode::Always,
    ..Default::default()
};

Full parameter docs → docs/api.md

Architecture

LSM Engine:
  Write Path:  Client → encode → WriteWorker → WAL (fsync) + MemTable → (flush) SST
  Read Path:   Client → MemTable → Block Index → Bloom → SST (LRU cached)

JsonDB Layer (built-in):
  Document:   D\x00{store}\x00{primary_key} → serialized JSON
  Index:      I\x00{store}\x00{index}\x00{encoded_value}\x00{primary_key} → primary_key
  Schema:     S\x00{store} → serialized StoreDef

License

MIT.