MagnumDB

Open-source embedded SQL and key-value database engine written in Rust.

Project Status: Experimental / Alpha (Not recommended for production use yet).

MagnumDB is an open-source database project built to explore database internals. The database engine and core storage components (B+ Tree, Buffer Pool, WAL, SQL Executor) are implemented in Rust.

Features & Supported SQL Matrix

MagnumDB exposes a Volcano-style query engine.

Two Modes of Operation

MagnumDB can be used in two distinct ways:

1. Embedded Mode: Compile MagnumDB directly into your Rust application. You get direct access to the Database struct for Key-Value access, or the Executor for in-memory SQL execution without network overhead.
2. Server Mode: Run magnumdb as a standalone TCP process. It listens for PostgreSQL Wire Protocol connections, meaning you can connect to it using psql, pg8000, or standard ORMs.

What's New in v0.4.4

• 🔒 Authentication Handshake: Native PostgreSQL wire-protocol MD5 password authentication. (Note: MD5 is cryptographically obsolete, but required for legacy Postgres client handshakes).
• 👤 CREATE USER DDL: Provision users directly via SQL (CREATE USER admin WITH PASSWORD 'pass';).
• 🧵 Thread-Safe MVCC: Thread-safe transaction execution using parking_lot::RwLock.

Architecture & Storage Format

MagnumDB implements a standard monolithic RDBMS architecture:

• Storage: Data is stored on disk in 4KB pages. Pages are managed by a custom Disk Pager.
• B+ Tree: Table records are stored in a B+ Tree. The tree supports overflow pages for records exceeding page size, and recycles freed leaf pages using a free-list.
• Buffer Pool: An LRU (Least Recently Used) Buffer Pool manages in-memory pages, pinning them during transactions and evicting when memory limits are reached.
• Transactions & MVCC: Transactions are assigned monotonic TxIDs. Row headers contain xmin and xmax fields for Multi-Version Concurrency Control (MVCC).
• Optimizer: We currently use a heuristic, rule-based planner. (A Cost-Based Optimizer / CBO using statistics is planned for the future).
• Concurrency: RwLock is used on the core Database struct. While reads can be concurrent, writes currently take exclusive locks during commit phases.

Durability & Crash Recovery

MagnumDB uses a Write-Ahead Log (WAL) for durability. Writes are buffered and appended to the WAL. A committed transaction is only considered durable once its WAL frame (with a CRC32 checksum) is fsync'd to disk.

• Process Crash: On restart, the engine replays the WAL from the last checkpoint.
• Partial/Corrupted WAL Frame: Checksums detect partial writes; the replay stops at the first corrupted frame.
• Uncommitted Transactions: Transactions without a COMMIT record in the WAL are discarded during recovery.

Known Limitations & Security

• No TLS Support: The server does not support SSL/TLS encryption. Do not expose MagnumDB to the public internet.
• Authentication limits: Uses MD5 for Postgres compatibility. No strong cryptographic auth exists.
• Isolation Level: Currently only guarantees Read Committed. Phantom reads are possible.
• RBAC: We support user creation, but granular GRANT/REVOKE table-level permissions do not exist yet.

Benchmarks & Comparisons

MagnumDB is an educational / research project. We do not yet claim to outperform production systems like SQLite, RocksDB, or redb.

• Benchmark results will be published here in v0.5.0.

Installation & Usage

1. Embedded Mode (Rust)

Add to Cargo.toml:

[dependencies]
magnumdb = "0.4.4"

use magnumdb::{Database, Config};
use magnumdb::sql::{Executor, Parser};

fn main() -> anyhow::Result<()> {
    let config = Config::default().with_path("./sql_data");
    let mut db = Database::open(config)?;
    let mut exec = Executor::new(&mut db);

    exec.execute(Parser::parse("CREATE TABLE users(id INT, name TEXT)")?)?;
    
    // Transactions
    exec.execute(Parser::parse("BEGIN;")?)?;
    exec.execute(Parser::parse("INSERT INTO users VALUES(1, 'Alice')")?)?;
    exec.execute(Parser::parse("COMMIT;")?)?;

    Ok(())
}

2. Server Mode (PostgreSQL Protocol)

Clone and build from source:

git clone https://github.com/sohamdev77/MagnumDB.git
cd MagnumDB
cargo build --release

# Start the TCP server on 127.0.0.1:5432
cargo run --bin magnumdb --release

In a separate terminal, connect using psql (Authentication requires the default postgres user):

psql -h 127.0.0.1 -p 5432 -U postgres -d postgres

postgres=> CREATE USER admin WITH PASSWORD 'secure123';
Query OK, user 'admin' created.

postgres=> BEGIN;
postgres=> CREATE TABLE test (id INT);
postgres=> INSERT INTO test VALUES (1);
postgres=> COMMIT;

3. Concurrent Clients Example

Because of the Tokio async server and MVCC locks, multiple clients can connect simultaneously:

# Terminal 1
psql -h 127.0.0.1 -U postgres
postgres=> BEGIN;
postgres=> INSERT INTO users VALUES (10);
# (Transaction uncommitted)

# Terminal 2
psql -h 127.0.0.1 -U postgres
postgres=> SELECT * FROM users;
# (Will not see '10' until Terminal 1 commits due to Read Committed isolation)

Testing

To run the test suite locally:

cargo test
cargo test --release

We include unit tests for storage components, SQL parsing, and integration tests for crash-recovery and network concurrency.