# do-memory-storage-redb
[](https://crates.io/crates/do-memory-storage-redb)
[](https://docs.rs/do-memory-storage-redb)
[](https://opensource.org/licenses/MIT)
**Purpose**: High-performance embedded cache layer for do-memory-core
## Overview
`do-memory-storage-redb` implements the high-performance cache layer for AI agent episodic memory using [redb](https://www.redb.org), an embedded key-value database. It provides blazing fast reads for hot-path operations with minimal overhead.
## Features
- **Blazing Fast**: Sub-microsecond read latency for cached data
- **Embedded**: No separate server process required
- **LRU Caching**: Automatic eviction of least recently used entries
- **TTL Support**: Configurable time-to-live for cache entries
- **Safe Concurrency**: Multiple readers with single writer (MVCC)
- **Crash Safe**: ACID guarantees with automatic recovery
- **Zero Copy**: Direct memory mapping for minimal overhead
## Key Modules
| `storage` | Core storage operations and cache management |
| `tables` | Table definitions and schema management |
| `cache` | Cache-specific operations (LRU, TTL, eviction) |
## Installation
Add this to your `Cargo.toml`:
```toml
[dependencies]
do-memory-storage-redb = "0.1"
```
## Quick Start
```rust
use memory_storage_redb::RedbStorage;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// Create cache with default settings
let cache = RedbStorage::new("./data/cache.redb")?;
// Configure cache size (max episodes)
let cache = RedbStorage::with_config(
"./data/cache.redb",
1000, // max_cache_size
)?;
// Cache is ready to use
Ok(())
}
```
## ⚠️ IMPORTANT: v0.1.7+ Breaking Change (Current: v0.1.13)
### Postcard Serialization (NOT bincode)
**Version v0.1.7 introduced a breaking change**: The serialization format changed from **bincode** to **postcard** (current version: v0.1.13).
### What Changed?
- **Old (< v0.1.7)**: Used `bincode` for serialization
- **New (≥ v0.1.7)**: Uses `postcard` for serialization (v0.1.13)
- **Why**: Postcard provides better memory safety and is more Rust-idiomatic
### Migration Requirements
If you have existing cache files from v0.1.6 or earlier:
```bash
# 1. Backup your existing cache (if needed)
cp ./data/cache.redb ./data/cache.redb.backup
# 2. Clear the cache to force re-build from Turso
rm ./data/cache.redb
# 3. Restart your application
# The cache will be repopulated from Turso storage
```
### No Data Loss
The cache is a **secondary storage layer**. Your primary data is stored in Turso. Clearing the cache simply forces a rebuild from the durable storage layer.
## Configuration
### Basic Configuration
```rust
use memory_storage_redb::{RedbStorage, RedbConfig};
let config = RedbConfig {
max_cache_size: 1000, // Maximum episodes to cache
ttl_seconds: Some(3600), // 1 hour TTL
enable_compression: true, // Enable value compression
};
let cache = RedbStorage::with_full_config("./data/cache.redb", config)?;
```
### Environment Variables
```bash
# Optional: Configure cache settings
export REDB_CACHE_PATH="./data/cache.redb"
export REDB_MAX_CACHE_SIZE="1000"
export REDB_TTL_SECONDS="3600"
```
## Cache Tables
The cache maintains four tables for different data types:
- **episodes**: Full episode data for fast retrieval
- **patterns**: Extracted patterns with similarity keys
- **embeddings**: Vector embeddings (if enabled)
- **metadata**: Cache statistics and health info
## Performance
Optimized for extremely fast read operations:
| Cache Read | < 10µs | 100K+ ops/s |
| Cache Write | < 100µs | 10K+ ops/s |
| LRU Eviction | < 1ms | Background |
| TTL Cleanup | < 10ms | Background |
### Real-world Performance
In production with do-memory-core (v0.1.13):
- **Cache hit rate**: 85-95% (varies by workload)
- **Average read latency**: 5-8µs (cache hits)
- **Average write latency**: 50-80µs
- **Throughput**: 50K+ reads/s, 8K+ writes/s
## LRU Cache Strategy
The cache uses an LRU (Least Recently Used) eviction policy:
1. Cache fills up to `max_cache_size`
2. New entries trigger eviction of oldest entries
3. Access updates entry timestamp
4. TTL expiration runs in background
### LRU Behavior
```rust
// Entries are tracked by last access time
cache.get("episode-123")?; // Updates access time
// When cache is full, oldest entry is evicted
cache.insert("episode-456", data)?; // May trigger eviction
```
## TTL Management
Time-to-live (TTL) automatically expires old entries:
```rust
use memory_storage_redb::{RedbConfig, RedbStorage};
let config = RedbConfig {
ttl_seconds: Some(3600), // Expire after 1 hour
..Default::default()
};
let cache = RedbStorage::with_full_config("./data/cache.redb", config)?;
// TTL cleanup runs every 60 seconds in background
// Expired entries are automatically removed
```
### TTL Settings
| `ttl_seconds` | Time-to-live in seconds | `None` (no TTL) |
| `ttl_check_interval` | Cleanup interval in seconds | 60 |
## Usage with do-memory-core
```rust
use memory_core::SelfLearningMemory;
use memory_storage_turso::TursoStorage;
use memory_storage_redb::RedbStorage;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// Turso for durable storage
let turso = TursoStorage::new(
std::env::var("TURSO_DATABASE_URL")?,
std::env::var("TURSO_AUTH_TOKEN")?,
).await?;
// redb for fast caching
let redb = RedbStorage::new("./data/cache.redb")?;
// Hybrid storage for best performance
let memory = SelfLearningMemory::with_storage(turso, redb).await?;
Ok(())
}
```
## Cache Synchronization
The cache automatically synchronizes with Turso:
- **Write-through**: Writes go to both Turso and redb
- **Read-aside**: Reads check cache first, then Turso
- **Invalidation**: Cache can be refreshed from Turso
- **Reconciliation**: Periodic sync ensures consistency
```rust
// Manual cache refresh (force sync from Turso)
memory.sync_memories().await?;
// Get cache statistics
let stats = cache.get_stats()?;
println!("Cache hit rate: {:.2}%", stats.hit_rate * 100.0);
println!("Total entries: {}", stats.total_entries);
println!("Cache size: {} bytes", stats.cache_size_bytes);
```
## Background Tasks
The cache runs background tasks for maintenance:
- **TTL Cleanup**: Removes expired entries (every 60s)
- **LRU Eviction**: Maintains size limits (on write)
- **Compression**: Optional value compression (configurable)
- **Health Checks**: Monitors cache health (on demand)
### Background Task Configuration
```rust
use memory_storage_redb::{RedbConfig, RedbStorage};
let config = RedbConfig {
ttl_check_interval: 30, // Check TTL every 30s
enable_compression: true, // Compress values
compression_threshold: 1024, // Compress values > 1KB
..Default::default()
};
let cache = RedbStorage::with_full_config("./data/cache.redb", config)?;
```
## File Management
redb stores all data in a single file:
```bash
# Check cache file size
ls -lh ./data/cache.redb
# Backup cache
cp ./data/cache.redb ./backups/cache-$(date +%Y%m%d).redb
# Clear cache (forces rebuild from Turso)
rm ./data/cache.redb
# Monitor cache growth
watch -n 5 'ls -lh ./data/cache.redb'
```
### Cache File Structure
- **Single file database**: All tables and data in one file
- **Memory mapped**: Direct access for zero-copy reads
- **Crash recovery**: Automatic recovery from unclean shutdown
- **Incremental writes**: Only modified data is written
## Compression
Optional value compression reduces cache size:
```rust
let config = RedbConfig {
enable_compression: true, // Enable compression
compression_threshold: 1024, // Compress values > 1KB
compression_level: 6, // Compression level (0-9)
..Default::default()
};
let cache = RedbStorage::with_full_config("./data/cache.redb", config)?;
```
### Compression Benefits
- **Reduced memory usage**: 40-60% smaller cache files
- **Faster disk I/O**: Less data to read/write
- **Trade-off**: Slightly higher CPU usage
## Monitoring & Statistics
### Cache Statistics
```rust
let stats = cache.get_stats()?;
println!("=== Cache Statistics ===");
println!("Hit rate: {:.2}%", stats.hit_rate * 100.0);
println!("Total entries: {}", stats.total_entries);
println!("Cache size: {} bytes", stats.cache_size_bytes);
println!("Reads: {}", stats.total_reads);
println!("Writes: {}", stats.total_writes);
println!("Evictions: {}", stats.total_evictions);
```
### Health Check
```rust
let health = cache.health_check()?;
println!("Cache healthy: {}", health.is_healthy);
println!("Disk available: {} bytes", health.available_disk_space);
println!("Cache file: {} bytes", health.cache_file_size);
```
## Testing
Run tests with an in-memory database:
```bash
cargo test -p do-memory-storage-redb
```
For integration tests with a real database file:
```bash
cargo test -p do-memory-storage-redb -- --ignored
```
## Dependencies
### Core Dependencies
- **redb**: Embedded key-value database
- **tokio**: Async runtime
- **async-trait**: Async trait support
- **anyhow**: Error handling
- **serde**: Serialization framework
- **postcard**: Serialization format (v0.1.7+, current: v0.1.13)
### Breaking Change Dependencies
- **postcard** (v0.1.7+, current: v0.1.13): Replaces bincode for safer serialization
## Documentation
Full API documentation: [docs.rs/do-memory-storage-redb](https://docs.rs/do-memory-storage-redb)
## Best Practices
### Production Deployment
- Enable TTL for automatic cleanup
- Set appropriate `max_cache_size` based on memory constraints
- Monitor cache hit rate to optimize size
- Enable compression for large datasets
- Run periodic health checks
### Development
- Use smaller cache sizes for local development
- Disable compression for easier debugging
- Monitor cache file growth
- Clear cache frequently during development
### Performance Tuning
- Adjust `max_cache_size` to balance memory and hit rate
- Use TTL for time-sensitive data
- Enable compression for memory-constrained environments
- Monitor eviction rate to detect undersized cache
### Cache Size Guidelines
| Low traffic | 100-500 | 85-90% |
| Medium traffic | 500-2000 | 90-95% |
| High traffic | 2000-10000 | 95-98% |
## Troubleshooting
### Cache Not Populating
```rust
// Check Turso connection first
let turso = TursoStorage::new(url, token).await?;
let episodes = turso.get_all_episodes().await?;
println!("Turso has {} episodes", episodes.len());
// Force cache sync
memory.sync_memories().await?;
```
### High Eviction Rate
```rust
let stats = cache.get_stats()?;
if stats.total_evictions > stats.total_writes / 2 {
// Increase cache size
let config = RedbConfig {
max_cache_size: stats.total_entries * 2,
..Default::default()
};
}
```
### Low Cache Hit Rate
```rust
let stats = cache.get_stats()?;
if stats.hit_rate < 0.8 {
// Check if cache is being used correctly
// Verify do-memory-core is configured to use cache
println!("Consider increasing cache size or TTL");
}
```
## License
Licensed under the MIT License. See [LICENSE](../LICENSE) for details.
## Project
Part of the [rust-self-learning-memory](https://github.com/d-o-hub/rust-self-learning-memory) project.
## Version History
- **v0.1.13** (Current): Postcard serialization, improved performance
- **v0.1.7**: Postcard serialization introduced
- **v0.1.6**: Initial release with bincode serialization