<div align="center">
<pre>
███████████ ████ █████ █████ ████ █████ █████
░░███░░░░░░█░░███ ░░███ ░░███ ███░ ░░███ ░░███
░███ █ ░ ░███ ██████ █████ ░███████ ░███ ███ ░███ ░███
░███████ ░███ ░░░░░███ ███░░ ░███░░███ ░███████ ░███ ░███
░███░░░█ ░███ ███████ ░░█████ ░███ ░███ ░███░░███ ░░███ ███
░███ ░ ░███ ███░░███ ░░░░███ ░███ ░███ ░███ ░░███ ░░░█████░
█████ █████░░████████ ██████ ████ █████ █████ ░░████ ░░███
░░░░░ ░░░░░ ░░░░░░░░ ░░░░░░ ░░░░ ░░░░░ ░░░░░ ░░░░ ░░░
</pre>
### A High-Performance, Redis-Compatible In-Memory Database Built from Scratch in Rust
[](https://www.rust-lang.org/)
[](LICENSE)
[]()
[]()
<p align="center">
<strong>A learning-focused systems programming project demonstrating database internals, network protocols, and concurrent programming in Rust</strong>
</p>
[Features](#features) •
[Architecture](#architecture) •
[Getting Started](#getting-started) •
[Commands](#supported-commands) •
[What I Learned](#what-i-learned)
---
</div>
## About This Project
**FlashKV** is a Redis-compatible, in-memory key-value database that I built from scratch to deeply understand systems programming concepts. This isn't just another Redis clone—it's a comprehensive learning journey through database internals, network programming, protocol design, and concurrent systems.
### Why I Built This
I wanted to go beyond tutorials and actually implement the core systems that power modern databases:
- **How do databases handle thousands of concurrent connections?** → Built an async TCP server with Tokio
- **How do protocols like Redis communicate efficiently?** → Implemented a zero-copy RESP parser
- **How do you make data structures thread-safe without killing performance?** → Designed a sharded storage engine
- **How do databases handle key expiration?** → Created a dual-strategy expiry system
> 💡 **This project comes with 15 detailed documentation files** explaining every design decision, from Rust fundamentals to advanced concurrency patterns.
---
## Features
### Core Database Features
| **Redis Protocol Compatible** | Works with `redis-cli`, Telnet, and any Redis client library |
| **Thread-Safe Concurrent Access** | 64-shard architecture allowing parallel reads/writes |
| **TTL & Auto-Expiry** | Keys can expire automatically with lazy + active cleanup |
| **Multiple Data Types** | Strings and Lists with full Redis-compatible operations |
| **Pattern Matching** | KEYS command with glob-style pattern support (`*`, `?`, `[abc]`) |
| **Built-in Statistics** | Real-time metrics for ops/second, memory usage, and more |
### Technical Highlights
| **Async Runtime** | Tokio for handling 10,000+ concurrent connections |
| **Zero-Copy Parsing** | `bytes::Bytes` for efficient memory handling |
| **Protocol** | Full RESP (Redis Serialization Protocol) + inline commands |
| **Concurrency** | `Arc<RwLock>` with sharding to minimize contention |
| **Background Tasks** | Adaptive expiry sweeper that adjusts based on load |
---
## Architecture
```
┌────────────────────────────────────────────────────────────────────────────┐
│ FlashKV │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ TCP Server │────▶│ Connection │────▶│ Command │ │
│ │ (Listener) │ │ Handler │ │ Handler │ │
│ │ │ │ │ │ │ │
│ │ • Accepts │ │ • Per-client │ │ • Parses │ │
│ │ connections│ │ read loop │ │ commands │ │
│ │ • Spawns │ │ • Buffered │ │ • Dispatches │ │
│ │ tasks │ │ I/O │ │ to storage │ │
│ └──────────────┘ └──────────────┘ └──────┬───────┘ │
│ │ │
│ ┌─────────────────────────────────────────┼────────────────┐ │
│ │ ▼ │ │
│ │ ┌────────────────────────────────────────────────────┐ │ │
│ │ │ StorageEngine │ │ │
│ │ │ │ │ │
│ │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │
│ │ │ │ Shard 0 │ │ Shard 1 │ │ Shard 2 │ │ ... │ │ │ │
│ │ │ │ ─────── │ │ ─────── │ │ ─────── │ │ 64 │ │ │ │
│ │ │ │ RwLock │ │ RwLock │ │ RwLock │ │ shards │ │ │ │
│ │ │ │ HashMap │ │ HashMap │ │ HashMap │ │ │ │ │ │
│ │ │ │(strings)│ │(strings)│ │(strings)│ │ │ │ │ │
│ │ │ │ HashMap │ │ HashMap │ │ HashMap │ │ │ │ │ │
│ │ │ │ (lists) │ │ (lists) │ │ (lists) │ │ │ │ │ │
│ │ │ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │ │ │
│ │ │ │ │ │
│ │ └────────────────────────────────────────────────────┘ │ │
│ │ ▲ │ │
│ │ │ │ │
│ │ ┌────────────────────────┴───────────────────────────┐ │ │
│ │ │ ExpirySweeper (Background) │ │ │
│ │ │ • Adaptive interval (100ms - 1s based on load) │ │ │
│ │ │ • Scans shards for expired keys │ │ │
│ │ │ • Graceful shutdown support │ │ │
│ │ └────────────────────────────────────────────────────┘ │ │
│ └──────────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────────────────┐ │
│ │ RESP Protocol Parser │ │
│ │ • Zero-copy parsing with bytes::Bytes │ │
│ │ • Supports: Simple Strings, Errors, Integers, Bulk Strings, Arrays │ │
│ │ • Inline command support (plain text like "SET key value") │ │
│ └──────────────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────────────┘
```
### Key Design Decisions
| **64 Shards** | Reduces lock contention—keys are distributed by hash, allowing parallel access |
| **RwLock per Shard** | Multiple readers can access data simultaneously; writers get exclusive access |
| **Separate List Storage** | Type safety—prevents accidental string operations on lists |
| **Lazy + Active Expiry** | Lazy catches expired keys on access; active reclaims memory for untouched keys |
| **VecDeque for Lists** | O(1) push/pop on both ends, perfect for LPUSH/RPUSH/LPOP/RPOP |
---
## Getting Started
### Prerequisites
- Rust 1.75+ ([Install Rust](https://rustup.rs/))
### Building
```bash
# Clone the repository
git clone https://github.com/yourusername/flashkv.git
cd flashkv
# Build in release mode (optimized)
cargo build --release
# Run tests (69 tests covering all functionality)
cargo test
```
### Running the Server
```bash
# Start with defaults (127.0.0.1:6379)
cargo run --release
# Or with custom settings
./target/release/flashkv --host 0.0.0.0 --port 6380
```
### Connecting
**Option 1: Using redis-cli**
```bash
redis-cli -p 6379
127.0.0.1:6379> PING
PONG
127.0.0.1:6379> SET greeting "Hello, World!"
OK
127.0.0.1:6379> GET greeting
"Hello, World!"
```
**Option 2: Using Telnet (inline commands)**
```bash
telnet localhost 6379
set name Ariz
+OK
get name
$4
Ariz
```
---
## Supported Commands
### String Commands (17 commands)
| `SET` | `SET key value [EX s] [PX ms] [NX\|XX] [GET]` | Set a key with optional expiry and conditions |
| `GET` | `GET key` | Get value by key |
| `DEL` | `DEL key [key ...]` | Delete one or more keys |
| `EXISTS` | `EXISTS key [key ...]` | Check if keys exist |
| `INCR` | `INCR key` | Increment integer value by 1 |
| `INCRBY` | `INCRBY key delta` | Increment by specified amount |
| `DECR` | `DECR key` | Decrement integer value by 1 |
| `DECRBY` | `DECRBY key delta` | Decrement by specified amount |
| `APPEND` | `APPEND key value` | Append to existing string |
| `STRLEN` | `STRLEN key` | Get string length |
| `MSET` | `MSET k1 v1 [k2 v2 ...]` | Set multiple keys atomically |
| `MGET` | `MGET k1 [k2 ...]` | Get multiple values |
| `SETNX` | `SETNX key value` | Set only if key doesn't exist |
| `SETEX` | `SETEX key seconds value` | Set with expiry in seconds |
| `PSETEX` | `PSETEX key ms value` | Set with expiry in milliseconds |
| `GETSET` | `GETSET key value` | Set new value, return old |
| `GETDEL` | `GETDEL key` | Get value and delete key |
### List Commands (9 commands)
| `LPUSH` | `LPUSH key val [val ...]` | Push to head of list |
| `RPUSH` | `RPUSH key val [val ...]` | Push to tail of list |
| `LPOP` | `LPOP key` | Remove and return from head |
| `RPOP` | `RPOP key` | Remove and return from tail |
| `LLEN` | `LLEN key` | Get list length |
| `LINDEX` | `LINDEX key index` | Get element by index (supports negative) |
| `LRANGE` | `LRANGE key start stop` | Get range of elements |
| `LSET` | `LSET key index value` | Set element at index |
| `LREM` | `LREM key count value` | Remove elements by value |
### Key Commands (10 commands)
| `EXPIRE` | `EXPIRE key seconds` | Set TTL in seconds |
| `PEXPIRE` | `PEXPIRE key ms` | Set TTL in milliseconds |
| `EXPIREAT` | `EXPIREAT key timestamp` | Set expiry at Unix timestamp |
| `TTL` | `TTL key` | Get remaining TTL in seconds |
| `PTTL` | `PTTL key` | Get remaining TTL in milliseconds |
| `PERSIST` | `PERSIST key` | Remove expiry from key |
| `KEYS` | `KEYS pattern` | Find keys matching pattern |
| `TYPE` | `TYPE key` | Get type (string/list/none) |
| `RENAME` | `RENAME key newkey` | Rename a key |
| `RENAMENX` | `RENAMENX key newkey` | Rename only if new key doesn't exist |
### Server Commands (10 commands)
| `PING` | `PING [message]` | Test connection |
| `ECHO` | `ECHO message` | Echo message back |
| `INFO` | `INFO [section]` | Server information |
| `DBSIZE` | `DBSIZE` | Number of keys |
| `FLUSHDB` | `FLUSHDB` | Clear entire database |
| `FLUSHALL` | `FLUSHALL` | Clear entire database |
| `COMMAND` | `COMMAND` | List available commands |
| `CONFIG` | `CONFIG GET param` | Get configuration |
| `TIME` | `TIME` | Server time |
| `DEBUG` | `DEBUG SLEEP seconds` | Debug utilities |
---
## Project Structure
```
flashkv/
├── src/
│ ├── main.rs # Entry point, CLI parsing, TCP server setup
│ ├── lib.rs # Public API exports
│ │
│ ├── protocol/ # RESP Protocol Implementation
│ │ ├── mod.rs # Module exports
│ │ ├── types.rs # RespValue enum, serialization
│ │ └── parser.rs # Zero-copy parser, inline command support
│ │
│ ├── storage/ # Storage Engine
│ │ ├── mod.rs # Module exports
│ │ ├── engine.rs # Sharded HashMap, Entry/ListEntry, all operations
│ │ └── expiry.rs # Background sweeper task
│ │
│ ├── commands/ # Command Handlers
│ │ ├── mod.rs # Module exports
│ │ └── handler.rs # 46 command implementations
│ │
│ └── connection/ # Connection Management
│ ├── mod.rs # Module exports
│ └── handler.rs # Per-client read loop, stats
│
├── docs/ # Comprehensive Documentation (15 files)
│ ├── 00_INDEX.md # Documentation index
│ ├── 01_RUST_FUNDAMENTALS.md # Rust concepts used in the project
│ ├── 02_ASYNC_PROGRAMMING.md # Tokio, async/await patterns
│ ├── 03_NETWORKING_BASICS.md # TCP, sockets, buffered I/O
│ ├── 04_RESP_PROTOCOL.md # Redis protocol specification
│ ├── 05_PROTOCOL_TYPES.md # RespValue implementation
│ ├── 06_PROTOCOL_PARSER.md # Parser design and zero-copy
│ ├── 07_CONCURRENCY.md # Arc, RwLock, sharding strategies
│ ├── 08_STORAGE_ENGINE.md # Storage design and operations
│ ├── 09_EXPIRY_SWEEPER.md # Background task implementation
│ ├── 10_COMMAND_HANDLER.md # Command dispatch and handlers
│ ├── 11_CONNECTION_HANDLER.md# Client connection lifecycle
│ ├── 12_MAIN_SERVER.md # Server bootstrap and shutdown
│ ├── 13_LIB_EXPORTS.md # Public API design
│ ├── 14_BENCHMARKING.md # Performance testing guide
│ └── 15_EXERCISES.md # Extension exercises
│
├── benches/
│ └── throughput.rs # Criterion benchmarks
│
├── Cargo.toml # Dependencies and metadata
└── README.md # You are here!
```
---
## What I Learned
Building FlashKV taught me deep, practical knowledge in several areas:
### Systems Programming
| **Memory Management** | Zero-copy parsing with `bytes::Bytes`, avoiding allocations in hot paths |
| **Concurrency** | Sharded locks allowing parallel access without data races |
| **Background Tasks** | Graceful spawning and shutdown of async tasks |
| **Resource Cleanup** | Proper cleanup on connection drop and server shutdown |
### Network Programming
| **TCP Servers** | Async listener accepting thousands of connections |
| **Buffered I/O** | Efficient reading with `BytesMut` buffers |
| **Protocol Parsing** | Incremental parser handling partial data |
| **Connection Lifecycle** | Per-client state management |
### Database Internals
| **Storage Engines** | Thread-safe HashMap with TTL support |
| **Data Types** | Strings and Lists with type-safe operations |
| **Expiry Mechanisms** | Lazy deletion + active background sweeping |
| **Statistics Tracking** | Atomic counters for real-time metrics |
### Rust Proficiency
| **Ownership & Borrowing** | Throughout—especially in the parser and storage |
| **Lifetimes** | Parser returning references into buffers |
| **Traits** | `Debug`, `Clone`, `Default` implementations |
| **Error Handling** | `Result`, `Option`, `thiserror` for custom errors |
| **Async/Await** | Tokio-based server and background tasks |
| **Smart Pointers** | `Arc`, `Box` for shared ownership |
| **Interior Mutability** | `RwLock` for concurrent access |
---
## Performance
Run benchmarks:
```bash
cargo bench
```
Or use Redis's benchmark tool:
```bash
# Start FlashKV
cargo run --release &
# Run benchmark
redis-benchmark -t set,get,incr -n 100000 -q -p 6379
```
Expected results on modern hardware:
```
SET: ~150,000 requests/second
GET: ~200,000 requests/second
INCR: ~180,000 requests/second
```
---
## Future Enhancements
The following features could be added to extend FlashKV:
- [ ] **Persistence** - RDB snapshots and AOF logging
- [ ] **Pub/Sub** - Publish/Subscribe messaging
- [ ] **Transactions** - MULTI/EXEC command blocks
- [ ] **More Data Types** - Sets, Sorted Sets, Hashes
- [ ] **Cluster Mode** - Distributed sharding across nodes
- [ ] **Lua Scripting** - Server-side scripting support
---
## Documentation
This project includes **15 detailed documentation files** covering:
1. **Rust Fundamentals** - Ownership, borrowing, lifetimes
2. **Async Programming** - Tokio, futures, task spawning
3. **Networking** - TCP, buffered I/O, connection handling
4. **Protocol Design** - RESP specification and parsing
5. **Concurrency** - Locks, atomics, sharding strategies
6. **Storage Design** - Data structures, expiry, statistics
7. **And more...**
Each file includes code examples, diagrams, and exercises.
---
## Contributing
Contributions are welcome! Whether it's:
- Bug fixes
- New features
- Documentation improvements
- Additional tests
---
## License
MIT License - feel free to use this for learning and building!
---
<div align="center">
### Built with ❤️ and 🦀 Rust
*This project was built as a learning exercise to understand database internals, network programming, and systems design.*
**[Back to Top](#flashkv)**
</div>