minikv 0.7.0

A production-grade distributed KV store with Raft consensus in ~1,500 lines
Documentation

🦀 minikv

A distributed, multi-tenant key-value & object store written in Rust

minikv provides strong consistency (Raft + 2PC), durability (WAL), and production-grade observability, security, and multi-tenancy — all in a modern Rust codebase.

Built in public as a learning-by-doing project — now evolved into a complete, reference implementation of distributed systems in Rust.

Repo Rust License: MIT Production Grade Build Status

🚦 What's New in v0.7.0

minikv v0.7.0 brings advanced data management and query capabilities :

  • Secondary indexes : Search keys by value content with GET /search?value=<substring>
  • Multi-key transactions : Execute multiple operations atomically with POST /transaction
  • Streaming/batch import/export : Bulk data operations with POST /admin/import & GET /admin/export
  • Durable S3-backed object store : Persistent storage for S3-compatible API via pluggable backends

Previous highlights (v0.6.0) : enterprise security, multi-tenancy, encryption at rest, quotas, audit logging, persistent backends, watch/subscribe system.

📚 Table of Contents

🤔 What is minikv?

minikv is a distributed key-value store written in Rust, designed for simplicity, speed, and reliability.

Who is this for ?
minikv is for engineers learning distributed systems, teams experimenting with Rust-based infrastructure, and anyone curious about consensus, durability, and system trade-offs.

  • Clustered : Raft consensus and 2PC for transactional writes
  • Virtual Sharding : 256 vshards for elastic scaling & balancing
  • WAL : Write-ahead log for durability
  • gRPC for node communication, HTTP REST & S3 API for clients
  • Bloom filters, snapshots, watch/subscribe for performance & reactivity

🛠 Tech Stack

  • Rust – core logic
  • Shell – orchestration/automation
  • JavaScript – benchmarks, tools
  • Makefile – build flows

⚡ Quick Start

git clone https://github.com/whispem/minikv.git
cd minikv
cargo build --release

# Start a node
cargo run -- --config config.example.toml

# API examples
curl localhost:8080/health/ready   # readiness
curl localhost:8080/metrics        # Prometheus metrics
curl localhost:8080/admin/status   # admin dashboard

# Create API key (admin)
curl -X POST http://localhost:8080/admin/keys -d '{"role":"ReadWrite","tenant_id":"acme"}'

# S3 (demo)
curl -X PUT localhost:8080/s3/mybucket/mykey -d 'hello minikv!'
curl localhost:8080/s3/mybucket/mykey

For cluster setup and advanced options, see the documentation.

📐 Architecture

  • Raft: consensus and leader election
  • 2PC: atomic distributed/batch writes
  • Virtual Shards: scale and rebalance across 256 partitions
  • Pluggable Storage: in-memory, RocksDB, Sled
  • Admin API: HTTP endpoints for status, metrics and config
  • Config: via environment, file or CLI flags

🚀 Performance

  • Write throughput : over 50,000 operations/sec (single node, in-memory)
  • Sub-millisecond read latency
  • Cluster tested (3–5 nodes, commodity VMs)
  • Built-in Prometheus metrics

🌟 Features

Distributed Core

  • Raft consensus (multi-node, strong consistency)
  • Two-phase commit (2PC) for atomic multi-key transactions
  • 256 virtual shards for cluster scaling and rebalancing
  • Write-ahead log (WAL) for durability
  • Auto-rebalancing, graceful leader failover, hot-join and node removal

Data Management

  • Time-To-Live keys (TTL)
  • LZ4 compression (configurable)
  • Bloom filters and index snapshots
  • Pluggable and persistent storage: in-memory, RocksDB, Sled
  • Batch & range operations, prefix queries

API

  • HTTP REST (CRUD, batch, range, admin)
  • S3-compatible API (with TTL extensions)
  • gRPC (internal)
  • WebSocket and SSE endpoints for real-time watch/subscribe events

Security & Multi-tenancy

  • API keys (Argon2) and JWT authentication
  • Role-based access control (RBAC) and audit logging
  • Multi-tenant isolation
  • AES-256-GCM encryption at rest
  • Per-tenant quotas (storage, requests, rate limits)
  • TLS (HTTP & gRPC)

Observability

  • Admin dashboard
  • Prometheus metrics (counters, histograms)
  • Request and endpoint statistics
  • Structured logging and tracing spans
  • Kubernetes health probes

Production-grade Design

  • Memory-safe Rust
  • Test suite, automated CI
  • Documentation and sample config
  • Single static binary

🗺️ Roadmap

v0.7.0 (latest)

  • Secondary indexes
  • Multi-key transactions
  • Durable S3-backed object store
  • Batch import/export

Next (v0.8.0+)

  • Cross-datacenter replication
  • Change Data Capture (CDC)
  • Admin Web UI
  • Backup & Restore
  • Plugin system

📖 Story

minikv started as a 24-hour challenge by a Rust learner (82 days into the language!). It now serves as both a playground and a reference for distributed systems, demonstrating curiosity, learning-by-doing, and robust engineering.

📚 Documentation

🛠️ Development

cargo test           # Run all tests
cargo clippy --fix   # Lint and fix
cargo fmt            # Format code

Continuous Integration runs on push & PR via .github/workflows/ci.yml.

🤝 Contributing

Issues and PRs welcome! See CONTRIBUTING.md.

📬 Contact