moltendb-core

🌋 The Pure Engine Crate

In-memory document store · Append-only WAL · Query evaluator · Analytics (🚧 WIP)
Zero knowledge of HTTP, auth, JWT, or WASM bindings.

What is this crate?

moltendb-core is the heart of MoltenDB. It contains every piece of logic that is shared between the HTTP server (moltendb-server) and the browser WASM adapter (moltendb-wasm):

• In-memory store — DashMap-backed document collections, keyed by (collection, key).
• Append-only WAL — every write is appended to a log file (LogEntry: INSERT, DELETE, DROP, INDEX, ENC). On startup the log is replayed into memory.
• Storage backends — DiskStorage (sync/async), TieredStorage (hot + cold log), EncryptedStorage (ChaCha20-Poly1305), OpfsStorage (WASM / browser OPFS).
• Query evaluator — $eq, $ne, $gt, $gte, $lt, $lte, $in, $nin, $contains, $or, $and, field projection (include / exclude), dot-notation for nested fields, joins, sort, count, offset.
• Analytics engine — COUNT, SUM, AVG, MIN, MAX with optional WHERE filtering. ⚠️ Under active development — not ready for production use.
• Auto-indexing — query_heatmap tracks hot fields and builds indexes automatically.
• Handler pipeline — process_get, process_set, process_update, process_delete, process_analytics — the single source of truth consumed by both the server and the WASM adapter.
• Input validation — collection name, key, and field name rules enforced before any operation reaches the engine.

Crate type

[lib]

crate-type = ["rlib"]

moltendb-core compiles to a native rlib. It is not a cdylib — WASM bindings live in the separate moltendb-wasm crate. This keeps the native dependency tree clean (no wasm-bindgen, no web-sys).

WASM-specific code (OpfsStorage, Db::open_wasm) is gated behind #[cfg(target_arch = "wasm32")] and only compiled when the crate is used as a dependency of moltendb-wasm.

Add to your project

[dependencies]

moltendb-core = "0.4.0"

Minimal example

use moltendb_core::engine::Db;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Open (or create) a database log file on disk
    let db = Db::open("./my_app.log").await?;

    // Insert a document
    db.set("users", "u1", serde_json::json!({
        "name": "Alice",
        "role": "admin"
    })).await?;

    // Read it back
    let user = db.get("users", "u1").await?;
    println!("{}", user);

    Ok(())
}

Using the handler pipeline (same API as the HTTP server)

use moltendb_core::{engine::Db, handlers};
use serde_json::json;

let db = Db::open("./my_app.log").await?;

let payload = json!({
    "collection": "users",
    "where": { "role": "admin" },
    "fields": ["name", "role"],
    "sort": [{ "field": "name", "order": "asc" }]
});

let (status_code, result) = handlers::process_get::process_get(&db, &payload, 10 * 1024 * 1024);
println!("{} — {}", status_code, result);

Hybrid Bitcask Storage

MoltenDB uses a Hybrid Bitcask-inspired Storage Model. Frequently accessed data is kept in RAM (Hot) as parsed JSON for sub-microsecond reads. Less frequently used data is paged out to disk (Cold) as byte-offsets, freeing up memory. This allows MoltenDB to handle datasets much larger than the available RAM while maintaining high performance for the active working set.

By default, any collection exceeding 50,000 documents will automatically evict the oldest documents to the Cold tier (disk/OPFS). This limit is configurable when opening the database.

Module overview

Storage modes

Design constraints

• No longer limited by RAM. While MoltenDB is "Memory-First," the Hybrid Bitcask model allows it to page out documents to disk while keeping only the keys and offsets in RAM. A 10GB database can now comfortably run on a machine with 512MB of RAM.
• No HTTP, no auth, no JWT. This crate has zero knowledge of the network layer. It is safe to embed in any Rust application without pulling in Axum, Tokio TLS, or any auth dependency.
• Single writer, many readers. The DashMap store is safe for concurrent reads. Writes are serialised through the storage backend.

Testing

moltendb-core includes a comprehensive test suite to ensure engine reliability and query correctness.

Unit Tests

Unit tests are located within the source files (e.g., src/query.rs, src/engine/storage/mod.rs).

cargo test -p moltendb-core --lib

Integration Tests

Integration tests are located in the tests/ directory and verify the interaction between the engine, handlers, and storage backends.

# Run all core integration tests

cargo test -p moltendb-core --test engine_tests

cargo test -p moltendb-core --test query_tests

cargo test -p moltendb-core --test hybrid_storage_tests

Part of the MoltenDB workspace

MoltenDB/
├── moltendb-core/     ← you are here
├── moltendb-wasm/     — browser adapter (wasm-bindgen glue, WorkerDb, OPFS)
├── moltendb-auth/     — identity layer (JWT, Argon2, UserStore)
└── moltendb-server/   — network layer (Axum, TLS, CORS, CLI config)

See the root README for the full architecture overview and feature list.