magnumdb 0.3.0

High-performance embeddable database engine in Rust with B+ Tree storage, WAL durability, Volcano query engine, and Postgres wire protocol support.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132

# MagnumDB

> Modern Open Source Embedded Database Engine in Native Rust

[![Build Status](https://img.shields.io/github/actions/workflow/status/sohamdev77/MagnumDB/ci.yml?branch=main)](https://github.com/sohamdev77/MagnumDB/actions)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Rust Version](https://img.shields.io/badge/rust-1.75%2B-blue.svg)](https://www.rust-lang.org/)
[![Crates.io Version](https://img.shields.io/crates/v/magnumdb.svg)](https://crates.io/crates/magnumdb)
[![Documentation](https://docs.rs/magnumdb/badge.svg)](https://docs.rs/magnumdb)
[![GitHub stars](https://img.shields.io/github/stars/sohamdev77/MagnumDB.svg)](https://github.com/sohamdev77/MagnumDB/stargazers)
[![GitHub issues](https://img.shields.io/github/issues/sohamdev77/MagnumDB.svg)](https://github.com/sohamdev77/MagnumDB/issues)

MagnumDB is an open-source embedded key-value and SQL database engine written 100% from scratch in native Rust. Designed for performance, reliability, and modularity.

## Features

- **Embedded Engine**: Runs directly inside Rust binaries with zero external C/C++ dependencies.
- **WAL Durability**: Write-Ahead Logging (WAL) with TxID framing and CRC32 checksums ensures crash recovery.
- **B+ Tree Indexing**: Custom 4KB page disk pager, LRU buffer pool management, overflow pages, and leaf page recycling.
- **Relational SQL Query Engine**: Hash JOINs (`INNER JOIN`, `LEFT JOIN`), `GROUP BY` & `HAVING` analytical aggregations, range filters, and composite multi-column indexes.
- **MVCC & Transactions**: MVCC row headers (`xmin`, `xmax`) and transaction logging with `BEGIN`, `COMMIT`, and `ROLLBACK`.
- **PostgreSQL Protocol Compatibility**: Native PostgreSQL Wire Protocol (`pgwire`) handling `StartupMessage`, `RowDescription` (`T`), `DataRow` (`D`), and `CommandComplete` (`C`).
- **Multi-Client TCP Server**: Async TCP server powered by Tokio with connection limits and idle timeouts.

---

## What's New in v0.3.0

Version `0.3.0` is a milestone release introducing relational query capabilities and PostgreSQL protocol compatibility:

- 🔗 **Relational JOINs**: Support for `INNER JOIN` and `LEFT JOIN` using streaming `HashJoinExec` Volcano operators:
  ```sql
  SELECT users.name, orders.amount FROM users JOIN orders ON users.id = orders.user_id;
  ```
- 📊 **`GROUP BY` & `HAVING` Aggregations**: Streaming `HashGroupAggregateExec` operator for analytical groupings:
  ```sql
  SELECT dept, COUNT(*) FROM employees GROUP BY dept HAVING COUNT(*) > 1;
  ```
- 📑 **Composite Multi-Column Indexes**: Supports secondary indexing over multiple columns:
  ```sql
  CREATE INDEX idx_name ON users(last_name, first_name);
  ```
- 🐘 **Native PostgreSQL Wire Protocol (`pgwire`)**: Server auto-detects and serves native PostgreSQL protocol connections (`StartupMessage`, `RowDescription`, `DataRow`, `CommandComplete`, `ReadyForQuery`).
- ⏳ **MVCC Tuple Headers**: Binary row encoding incorporates `[xmin: 8B][xmax: 8B]` transaction visibility metadata.

---

## Architecture

```mermaid
graph TD
    A[Client / magnum shell / psql] -->|SQL Query| B(SQL Parser)
    B -->|AST| C(Query Executor)
    
    C -->|Reads/Writes| D[B+ Tree Index]
    C -->|Logs| E[(WAL - Write Ahead Log)]
    
    D -->|Request Page| F(Buffer Pool Manager)
    F -->|Evict/Load 4KB Pages| G[(Disk / Pager)]
    
    style A fill:#f9f,stroke:#333,stroke-width:2px
    style D fill:#bbf,stroke:#333,stroke-width:2px
    style F fill:#bbf,stroke:#333,stroke-width:2px
    style G fill:#dfd,stroke:#333,stroke-width:2px
    style E fill:#dfd,stroke:#333,stroke-width:2px
```

---

## Installation

Add MagnumDB to your `Cargo.toml`:

```toml
[dependencies]
magnumdb = "0.3.0"
```

---

## Quick Start (Embedded Key-Value)

```rust
use magnumdb::{Database, Config};

fn main() -> anyhow::Result<()> {
    let config = Config::default().with_path("./my_database");
    let mut db = Database::open(config)?;

    // Embedded Key-Value API
    db.put(b"user:100", b"Soham")?;
    let val = db.get(b"user:100")?;
    
    if let Some(bytes) = val {
        println!("Found: {}", String::from_utf8_lossy(&bytes));
    }

    db.close()?;
    Ok(())
}
```

---

## Embedded SQL Usage

```rust
use magnumdb::{Database, Config};
use magnumdb::sql::{Executor, Parser};

fn main() -> anyhow::Result<()> {
    let config = Config::default().with_path("./sql_data");
    let mut db = Database::open(config)?;
    let mut exec = Executor::new(&mut db);

    exec.execute(Parser::parse("CREATE TABLE users(id INT, name TEXT)")?)?;
    exec.execute(Parser::parse("INSERT INTO users VALUES(1, 'Alice')")?)?;
    
    let res = exec.execute(Parser::parse("SELECT * FROM users")?)?;
    println!("{}", res);

    Ok(())
}
```

---

## Contributing & License

We welcome contributions! Please review [CONTRIBUTING.md](CONTRIBUTING.md).

Licensed under the [MIT License](LICENSE).