magnumdb 0.2.0

High-performance embeddable database engine in Rust with B+ Tree storage, WAL durability, Volcano query engine, and Postgres wire protocol support.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145

# MagnumDB

> Modern Open Source Embedded Database Engine in Native Rust

[![Build Status](https://img.shields.io/github/actions/workflow/status/sohamdev77/MagnumDB/ci.yml?branch=main)](https://github.com/sohamdev77/MagnumDB/actions)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Rust Version](https://img.shields.io/badge/rust-1.75%2B-blue.svg)](https://www.rust-lang.org/)
[![Crates.io Version](https://img.shields.io/crates/v/magnumdb.svg)](https://crates.io/crates/magnumdb)
[![Documentation](https://docs.rs/magnumdb/badge.svg)](https://docs.rs/magnumdb)
[![GitHub stars](https://img.shields.io/github/stars/sohamdev77/MagnumDB.svg)](https://github.com/sohamdev77/MagnumDB/stargazers)
[![GitHub issues](https://img.shields.io/github/issues/sohamdev77/MagnumDB.svg)](https://github.com/sohamdev77/MagnumDB/issues)

MagnumDB is an open-source embedded key-value and SQL database engine written 100% from scratch in native Rust. Designed for performance, reliability, and modularity.

## Features

- **Embedded Engine**: Runs directly inside Rust binaries with zero external C/C++ dependencies.
- **WAL Durability**: Write-Ahead Logging (WAL) with TxID framing and CRC32 checksums ensures crash recovery.
- **B+ Tree Indexing**: Custom 4KB page disk pager, LRU buffer pool management, overflow pages, and leaf page recycling.
- **SQL Execution Engine**: Built-in SQL AST parser, Volcano-style streaming query execution, range filters, and secondary indexes.
- **Multi-Client TCP Server**: Async TCP server powered by Tokio with connection limits and idle timeouts.
- **ACID Transactions**: Transaction logging with `BEGIN`, `COMMIT`, and `ROLLBACK` support.

---

## What's New in v0.2.0

Version `0.2.0` represents a major production stability release resolving 14 core database engine issues:

- 🛡️ **Durable Transactions**: Transaction writes properly track `tx_id` in WAL records, and committed pages are flushed and synced on `COMMIT`.
- 🔒 **Page Serialization Protection**: B+ Tree page serialization includes bounds-checking guards against 4KB overflows.
- 📐 **Documented Metadata Page Layout**: Page 0 layout standardized with magic bytes (`MGDB`), root page ID, free-list head, and checkpoint LSN.
- ⚡ **Optimized Crash Recovery**: Startup WAL recovery uses checkpoint LSN filtering to avoid full file replay.
- 🧹 **Secondary Index Maintenance**: `DELETE` and `UPDATE` SQL queries automatically update and clean up secondary indexes.
- 👥 **Multi-PK Secondary Indexes**: Secondary indexes now support multiple primary keys per indexed value (1:N mapping).
- 💬 **Enhanced SQL Parsing**: Supports string literals with spaces, commas (`'hello, world'`), and escaped quotes (`''`).
- 🛑 **Identifier Validation**: Reserved system namespace (`__`) protects catalog tables from SQL injection.
- 🌐 **Async TCP Server Hardening**: Connection semaphore limits (`max_connections`) and idle read timeouts.

---

## Architecture

```mermaid
graph TD
    A[Client / magnum shell] -->|SQL Query| B(SQL Parser)
    B -->|AST| C(Query Executor)
    
    C -->|Reads/Writes| D[B+ Tree Index]
    C -->|Logs| E[(WAL - Write Ahead Log)]
    
    D -->|Request Page| F(Buffer Pool Manager)
    F -->|Evict/Load 4KB Pages| G[(Disk / Pager)]
    
    style A fill:#f9f,stroke:#333,stroke-width:2px
    style D fill:#bbf,stroke:#333,stroke-width:2px
    style F fill:#bbf,stroke:#333,stroke-width:2px
    style G fill:#dfd,stroke:#333,stroke-width:2px
    style E fill:#dfd,stroke:#333,stroke-width:2px
```

---

## Installation

Add MagnumDB to your `Cargo.toml`:

```toml
[dependencies]
magnumdb = "0.2.0"
```

---

## Quick Start (Embedded Key-Value)

```rust
use magnumdb::{Database, Config};

fn main() -> anyhow::Result<()> {
    let config = Config::default().with_path("./my_database");
    let mut db = Database::open(config)?;

    // Embedded Key-Value API
    db.put(b"user:100", b"Soham")?;
    let val = db.get(b"user:100")?;
    
    if let Some(bytes) = val {
        println!("Found: {}", String::from_utf8_lossy(&bytes));
    }

    db.close()?;
    Ok(())
}
```

---

## Embedded SQL Usage

```rust
use magnumdb::{Database, Config};
use magnumdb::sql::{Executor, Parser};

fn main() -> anyhow::Result<()> {
    let config = Config::default().with_path("./sql_data");
    let mut db = Database::open(config)?;
    let mut exec = Executor::new(&mut db);

    exec.execute(Parser::parse("CREATE TABLE users(id INT, name TEXT)")?)?;
    exec.execute(Parser::parse("INSERT INTO users VALUES(1, 'Alice')")?)?;
    
    let res = exec.execute(Parser::parse("SELECT * FROM users")?)?;
    println!("{}", res);

    Ok(())
}
```

---

## Publishing & Updating Crates.io

To update or publish new versions to [crates.io](https://crates.io/crates/magnumdb):

1. Login to crates.io via Cargo token:
   ```bash
   cargo login <YOUR_CRATES_IO_TOKEN>
   ```
2. Package and verify dry-run build:
   ```bash
   cargo package
   ```
3. Publish to Crates.io:
   ```bash
   cargo publish
   ```

---

## Contributing & License

We welcome contributions! Please review [CONTRIBUTING.md](CONTRIBUTING.md).

Licensed under the [MIT License](LICENSE).