# Hero Redis
A high-performance, Redis-compatible server built on [redb](https://github.com/cberner/redb) with ChaCha20-Poly1305 encryption and dual authentication modes.
## Features
- **Redis Protocol Compatible** - Works with any Redis client including Redis Desktop Manager
- **Dual Authentication**:
- **Simple**: Secret-based auth with `AUTH <secret>` (recommended for most users)
- **Advanced**: Ed25519 signature-based auth (for high-security environments)
- **Multi-Database with ACL** - Per-database Read/Write/Admin permissions
- **Persistent Storage** - Data stored in redb (pure Rust embedded database)
- **Encryption** - All values encrypted with ChaCha20-Poly1305
- **Multiple Databases** - Up to 1000 databases with lazy loading
- **Low Memory** - ~3MB at startup, databases loaded on demand
- **Auto-cleanup** - Idle databases automatically closed after 5 minutes
- **Unix Socket & TCP** - Both connection methods supported
- **Vector Search** - HNSW-based similarity search with hannoy library
- **Cross-platform** - Linux (x86_64, aarch64) and macOS (x86_64, aarch64)
## Quick Start
### Install Pre-built Binaries
The easiest way to get started is to install pre-built binaries:
```bash
This installs all tools to `~/hero/bin/`:
- `hero_redis` - The Redis-compatible server
- `hero_redis_client` - Client with Rhai scripting support
- `hero_redis_login` - Ed25519 keypair generator
- `hero_redis_tester` - Integration test tool
Add to your PATH (if not already):
```bash
export PATH="$HOME/hero/bin:$PATH"
```
Supported platforms:
- Linux x86_64 (amd64)
- Linux aarch64 (arm64)
- macOS Apple Silicon (darwin-arm64)
### Build from Source
```bash
git clone https://forge.ourworld.tf/geomind_code/hero_redis.git
cd hero_redis
cargo build --release --features server
```
### Start Server (Simple Auth)
```bash
# Start with admin secret (simplest way)
hero_redis \
--encryption-key "your-encryption-key" \
--admin-secret "your-admin-secret" \
--data-dir ~/.hero_redis
```
Or if built from source:
```bash
./target/release/hero_redis \
--encryption-key "your-encryption-key" \
--admin-secret "your-admin-secret"
```
### Connect and Use
```bash
# Using redis-cli (default port is 6666)
redis-cli -p 6666
# Authenticate with your secret
> AUTH your-admin-secret
OK
# Now use any Redis command
> SET mykey myvalue
OK
> GET mykey
"myvalue"
```
### Run Rhai Scripts
```bash
# Run a single script
hero_redis_client my_script.rhai
# Run all scripts in a directory
hero_redis_client scripts/
```
### Generate Ed25519 Keypair
```bash
# Generate a new keypair for Ed25519 authentication
hero_redis_login --generate
```
### Using with Redis Desktop Manager
1. Start server with `--admin-secret`
2. In Redis Desktop Manager, configure connection:
- Host: 127.0.0.1
- Port: 6666
- Auth: your-admin-secret
3. Connect and use normally
## Authentication Modes
### 1. Simple Auth (Recommended)
Start the server with `--admin-secret`:
```bash
./target/release/hero_redis \
--encryption-key "encryption-key" \
--admin-secret "my-secret-password"
```
Connect with `AUTH`:
```
AUTH my-secret-password
```
Create users with secrets:
```
USER.CREATESECRET alice alice-secret-123
USER.GRANT 1 alice write
```
Users authenticate with:
```
AUTH alice-secret-123
```
### 2. Advanced Auth (Ed25519)
For high-security environments, use Ed25519 signature-based authentication:
```bash
# Generate keypair
./target/release/hero_redis_login --generate
# Start server with pubkey
./target/release/hero_redis \
--encryption-key "encryption-key" \
--admin-pubkey "a1b2c3d4..."
```
Authentication flow:
```
CHALLENGE → returns random challenge
TOKEN <pubkey> <signature> → returns session token
AUTH <token> → authenticates session
```
## Command Line Options
### Server (hero_redis)
| `-d, --data-dir` | `~/.hero_redis` | Database directory |
| `-s, --socket` | `~/.hero_redis/redis.sock` | Unix socket path |
| `-p, --port` | `6666` | TCP port (0 to disable) |
| `--encryption-key` | _required_ | Encryption key for DB 0 |
| `--admin-secret` | - | Admin secret for simple auth |
| `--admin-pubkey` | - | Admin public key for Ed25519 auth |
| `-v, --verbose` | false | Enable debug logging |
Note: At least one of `--admin-secret` or `--admin-pubkey` must be provided.
## User Management
### Create Users (Secret-based)
```
# As admin, create a user with a secret
USER.CREATESECRET <username> <secret>
# Example
USER.CREATESECRET alice mysecretpassword
```
### Create Users (Ed25519-based)
```
# As admin, register a user's public key
USER.CREATE <pubkey>
# Example
USER.CREATE a1b2c3d4e5f6...
```
### Grant Permissions
```
# Grant access to a database
USER.GRANT <db_number> <username_or_pubkey> <read|write|admin>
# Examples
USER.GRANT 1 alice write
USER.GRANT 2 bob read
USER.GRANT 1 a1b2c3d4e5f6... admin
```
### Revoke Permissions
```
USER.REVOKE <db_number> <username_or_pubkey>
# Example
USER.REVOKE 1 alice
```
## Permission Levels
| **read** | Yes | No | No | No | No |
| **write** | Yes | Yes | No | No | No |
| **admin** | Yes | Yes | Yes | Yes (same db) | No |
| **server admin** | Yes | Yes | Yes | Yes | Yes |
## Admin Commands
### Server Admin Commands
| `DATABASE.CREATE <encryption_key>` | Create new database, returns db number |
| `DATABASE.STATUS [db\|all]` | Show database info |
| `DATABASE.PUBLIC <db> <on\|off>` | Enable/disable public read-only access |
| `USER.CREATE <pubkey>` | Register Ed25519 user |
| `USER.CREATESECRET <username> <secret>` | Create secret-based user |
| `USER.DELETE <username_or_pubkey>` | Delete a user |
| `ADMIN.ADD <pubkey>` | Add server admin (Ed25519) |
| `ADMIN.REMOVE <pubkey>` | Remove server admin |
| `ADMIN.LIST` | List server admins |
### Public Read-Only Access
You can make a database publicly readable (no authentication required for reads):
```bash
# As admin, make database 1 public
DATABASE.PUBLIC 1 on
# Now anyone can read without AUTH
SELECT 1
GET publickey # Works without authentication
SET key value # Fails - writes still require auth
```
### Database Admin Commands
| `USER.GRANT <db> <user> <perm>` | Grant permission (read/write/admin) |
| `USER.REVOKE <db> <user>` | Revoke permission |
| `FLUSHDB` | Clear current database |
## Supported Redis Commands
### Authentication
`AUTH`, `SAUTH`, `CHALLENGE`, `TOKEN`
### String Commands
`GET`, `SET`, `MGET`, `MSET`, `DEL`, `EXISTS`, `EXPIRE`, `TTL`, `KEYS`, `SCAN`, `INCR`, `INCRBY`, `DECR`, `DECRBY`, `APPEND`, `STRLEN`, `GETRANGE`, `SETNX`, `SETEX`, `GETSET`, `TYPE`
### Hash Commands
`HSET`, `HGET`, `HMSET`, `HMGET`, `HGETALL`, `HDEL`, `HEXISTS`, `HLEN`, `HKEYS`, `HVALS`, `HINCRBY`, `HSETNX`, `HSCAN`
### List Commands
`LPUSH`, `RPUSH`, `LPOP`, `RPOP`, `LRANGE`, `LLEN`, `LINDEX`, `LSET`, `LREM`
### Set Commands
`SADD`, `SREM`, `SMEMBERS`, `SISMEMBER`, `SCARD`, `SPOP`, `SUNION`, `SINTER`, `SDIFF`, `SSCAN`
### Stream Commands
`XADD`, `XLEN`, `XRANGE`, `XREVRANGE`, `XREAD`, `XINFO`, `XTRIM`, `XGROUP`, `XREADGROUP`, `XACK`, `XPENDING`
### Vector Commands
`VECTOR.CREATE`, `VECTOR.ADD`, `VECTOR.SEARCH`, `VECTOR.SEARCHBYID`, `VECTOR.GET`, `VECTOR.DEL`, `VECTOR.BUILD`, `VECTOR.INFO`, `VECTOR.LIST`, `VECTOR.DROP`, `VECTOR.CLEAR`, `VECTOR.EXISTS`, `VECTOR.LEN`
### Connection Commands
`PING`, `AUTH`, `SAUTH`, `SELECT`, `QUIT`, `COMMAND`
### Management Commands
`COMPACT`, `SHUTDOWN`, `MEMORY USAGE`, `MEMORY STATS`, `DBSIZE`, `FLUSHDB`, `INFO`, `CONFIG`, `CLIENT`
## Vector Search
Hero Redis includes HNSW-based vector similarity search using the [hannoy](https://crates.io/crates/hannoy) library. Perfect for AI embeddings, semantic search, and RAG applications.
### Quick Start
```bash
# Create a vector index (128 dimensions, cosine similarity)
VECTOR.CREATE embeddings 128 METRIC cosine
# Add vectors (JSON format)
VECTOR.ADD embeddings 1 [0.1, 0.2, 0.3, ...]
VECTOR.ADD embeddings 2 [0.4, 0.5, 0.6, ...]
# Build the HNSW index (required before searching)
VECTOR.BUILD embeddings
# Search for similar vectors (returns top 10)
VECTOR.SEARCH embeddings [0.1, 0.2, 0.3, ...] 10
# Search by existing vector ID
VECTOR.SEARCHBYID embeddings 1 10
```
### High-Performance Bulk Insert
For bulk loading, use `VECTOR.ADDBATCH` with binary format - **~97x faster** than individual adds:
| Individual VECTOR.ADD | ~220 vec/s | ~45 sec |
| Binary VECTOR.ADDBATCH | ~21,000 vec/s | ~0.5 sec |
Binary format (little-endian):
```
Header (8 bytes):
- num_vectors: u32 (4 bytes)
- dimensions: u32 (4 bytes)
Per vector:
- id: u32 (4 bytes)
- vector: f32 × dimensions
```
### Rhai Script Example
```rhai
// Create index
vector_create(r, "embeddings", 128, "cosine");
// Batch add (fast!)
let batch = [];
for id in 1..=1000 {
batch.push([id, generate_embedding(id)]);
}
vector_add_batch(r, "embeddings", 128, batch);
// Build and search
vector_build(r, "embeddings");
let results = vector_search(r, "embeddings", query_vector, 10);
```
### Rust Client Example
```rust
use hero_redis::HeroRedisClient;
let mut client = HeroRedisClient::new("127.0.0.1", 6666, private_key)?;
client.select(1)?;
// Create index
client.vector_create("embeddings", 128, "cosine")?;
// Bulk add with binary format (fast!)
let items: Vec<(u32, Vec<f32>)> = embeddings.into_iter().enumerate()
.map(|(i, v)| (i as u32, v))
.collect();
client.vector_add_batch("embeddings", 128, &items)?;
// Build and search
client.vector_build("embeddings")?;
let results = client.vector_search("embeddings", &query, 10)?;
```
### Vector Commands Reference
| `VECTOR.CREATE <index> <dims> METRIC <cosine\|euclidean\|manhattan>` | Create index |
| `VECTOR.ADD <index> <id> <json_vector>` | Add single vector |
| `VECTOR.ADDBATCH <index> <binary_data>` | Bulk add (binary format, ~97x faster) |
| `VECTOR.BUILD <index>` | Build HNSW graph (required before search) |
| `VECTOR.SEARCH <index> <json_vector> <k>` | Find k nearest neighbors |
| `VECTOR.SEARCHBYID <index> <id> <k>` | Find neighbors of existing vector |
| `VECTOR.GET <index> <id>` | Get vector by ID |
| `VECTOR.DEL <index> <id>` | Delete vector |
| `VECTOR.INFO <index>` | Get index statistics |
| `VECTOR.LEN <index>` | Get vector count |
| `VECTOR.LIST` | List all indexes |
| `VECTOR.DROP <index>` | Delete index |
| `VECTOR.CLEAR <index>` | Remove all vectors |
### Distance Metrics
| `cosine` | Cosine similarity | Text embeddings (OpenAI, etc.) |
| `euclidean` | L2 distance | Image embeddings |
| `manhattan` | L1 distance | Sparse vectors |
### Common Embedding Dimensions
| OpenAI text-embedding-ada-002 | 1536 |
| OpenAI text-embedding-3-small | 1536 |
| Cohere embed-english-v3.0 | 1024 |
| all-MiniLM-L6-v2 | 384 |
| BGE-base-en | 768 |
See [docs/ai_embeddings.md](docs/ai_embeddings.md) for complete documentation.
## Using as a Library
Hero Redis supports feature flags to include only what you need, reducing binary size and compile times.
### Feature Flags
| `client` | Redis client with Ed25519 authentication | Yes |
| `server` | Redis-compatible server with redb backend | No |
| `rhai` | Rhai scripting support for the client | No |
| `vector` | Vector database with HNSW similarity search | No |
| `full` | All features enabled | No |
### Usage Examples
```toml
# Client only (default) - minimal dependencies
[dependencies]
hero_redis = "0.2"
# Server only (no client)
[dependencies]
hero_redis = { version = "0.2", default-features = false, features = ["server"] }
# Client with Rhai scripting
[dependencies]
hero_redis = { version = "0.2", features = ["rhai"] }
# Client with vector support
[dependencies]
hero_redis = { version = "0.2", features = ["vector"] }
# Everything
[dependencies]
hero_redis = { version = "0.2", features = ["full"] }
```
### Client Example
```rust
use hero_redis::hero_redis_client::HeroRedisClient;
// Connect with Ed25519 private key
let mut client = HeroRedisClient::new("127.0.0.1", 6666, "your_private_key_hex")?;
// Or connect with simple password
let mut client = HeroRedisClient::new_with_password("127.0.0.1", 6666, "your_password")?;
// Or use the builder pattern
let mut client = HeroRedisClient::builder()
.host("127.0.0.1")
.port(6666)
.password("your_password") // or .private_key("hex")?
.database(1)
.connect()?;
client.select(1)?;
// String operations
client.set("name", "Alice")?;
let name = client.get("name")?.unwrap();
// Hashes
client.hset("user:1", "name", "Bob")?;
let user = client.hgetall("user:1")?;
// Lists
client.rpush("queue", "job1")?;
let job = client.lpop("queue")?;
// Sets
client.sadd("tags", "rust")?;
let tags = client.smembers("tags")?;
// Admin operations (create database, manage users)
let db_num = client.admin().create_database("my-encryption-key")?;
client.admin().grant_permission(db_num, &user_pubkey, Permission::Write)?;
```
### Rhai Scripting Example
With `features = ["rhai"]`:
```rust
use hero_redis::hero_redis_rhai::RhaiScriptRunner;
let runner = RhaiScriptRunner::builder()
.host("127.0.0.1")
.port(6666)
.private_key("your_private_key_hex")
.build()?;
runner.run_script("scripts/my_script.rhai")?;
```
Example Rhai script:
```rhai
// Select database
select(r, 1);
// Basic operations
set(r, "counter", "0");
incr(r, "counter");
let val = get(r, "counter");
print(`Counter: ${val}`);
// Hashes
hset(r, "user:1", "name", "Alice");
hset(r, "user:1", "email", "alice@example.com");
let user = hgetall(r, "user:1");
print(user);
```
## Data Storage
- Data directory: `~/.hero_redis/` (configurable)
- Database files: `db0.redb`, `db1.redb`, ... `db999.redb`
- DB 0 is reserved for admin metadata
- Each database has its own encryption key
- Values are encrypted with ChaCha20-Poly1305
## Building & Publishing
### Pre-built Binaries
Pre-built binaries are published to the Forgejo package registry for:
| `hero_redis` | linux-amd64, linux-arm64, darwin-arm64 | Redis-compatible server |
| `hero_redis_tester` | linux-amd64, linux-arm64, darwin-arm64 | Integration test tool |
| `hero_redis_login` | linux-amd64, linux-arm64, darwin-arm64 | Ed25519 keypair generator & login tool |
| `hero_redis_client` | linux-amd64, linux-arm64, darwin-arm64 | Client with Rhai scripting |
### Build Scripts
Two scripts are provided for local development and publishing:
#### `build_package.sh` - Local Build & Publish to Forgejo
Builds macOS binaries locally using `forgejo-runner` and publishes to the Forgejo package registry.
```bash
# Set your Forgejo token first
export FORGEJO_TOKEN='your-token-here'
# Run local build and publish
./build_package.sh
```
This script:
1. Verifies your Forgejo token
2. Installs `forgejo-runner` if needed
3. Runs the macOS build workflow locally
4. Publishes binaries to `forge.ourworld.tf`
#### `build_publish.sh` - Build & Publish to crates.io
Runs full CI checks and optionally publishes the library to crates.io.
```bash
./build_publish.sh
```
This script:
1. Cleans previous build artifacts
2. Checks code formatting (`cargo fmt`)
3. Runs clippy linting
4. Builds with release profile
5. Runs tests
6. Generates documentation
7. Verifies package with `cargo publish --dry-run`
8. Prompts for confirmation before publishing to crates.io
### CI/CD Workflows
The Forgejo workflows (`.forgejo/workflows/`) automatically build and publish on push to `main`:
- **build-linux.yaml**: Builds for `linux-amd64` and `linux-arm64` (with UPX compression)
- **build-macos.yaml**: Builds for `darwin-arm64` (native macOS host)
Both workflows build with `--features "server,rhai"` to include:
- `hero_redis` - The server binary
- `hero_redis_client` - The Rhai-enabled client binary
## License
Apache 2.0