do-memory-storage-redb

Purpose: High-performance embedded cache layer for do-memory-core

Overview

do-memory-storage-redb implements the high-performance cache layer for AI agent episodic memory using redb, an embedded key-value database. It provides blazing fast reads for hot-path operations with minimal overhead.

Features

• Blazing Fast: Sub-microsecond read latency for cached data
• Embedded: No separate server process required
• LRU Caching: Automatic eviction of least recently used entries
• TTL Support: Configurable time-to-live for cache entries
• Safe Concurrency: Multiple readers with single writer (MVCC)
• Crash Safe: ACID guarantees with automatic recovery
• Zero Copy: Direct memory mapping for minimal overhead

Key Modules

Module	Purpose
storage	Core storage operations and cache management
tables	Table definitions and schema management
cache	Cache-specific operations (LRU, TTL, eviction)

Installation

Add this to your Cargo.toml:

[dependencies]
do-memory-storage-redb = "0.1"

Quick Start

use memory_storage_redb::RedbStorage;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Create cache with default settings
    let cache = RedbStorage::new("./data/cache.redb")?;

    // Configure cache size (max episodes)
    let cache = RedbStorage::with_config(
        "./data/cache.redb",
        1000,  // max_cache_size
    )?;

    // Cache is ready to use
    Ok(())
}

⚠️ IMPORTANT: v0.1.7+ Breaking Change (Current: v0.1.13)

Postcard Serialization (NOT bincode)

Version v0.1.7 introduced a breaking change: The serialization format changed from bincode to postcard (current version: v0.1.13).

What Changed?

• Old (< v0.1.7): Used bincode for serialization
• New (≥ v0.1.7): Uses postcard for serialization (v0.1.13)
• Why: Postcard provides better memory safety and is more Rust-idiomatic

Migration Requirements

If you have existing cache files from v0.1.6 or earlier:

# 1. Backup your existing cache (if needed)
cp ./data/cache.redb ./data/cache.redb.backup

# 2. Clear the cache to force re-build from Turso
rm ./data/cache.redb

# 3. Restart your application
# The cache will be repopulated from Turso storage

No Data Loss

The cache is a secondary storage layer. Your primary data is stored in Turso. Clearing the cache simply forces a rebuild from the durable storage layer.

Configuration

Basic Configuration

use memory_storage_redb::{RedbStorage, RedbConfig};

let config = RedbConfig {
    max_cache_size: 1000,        // Maximum episodes to cache
    ttl_seconds: Some(3600),      // 1 hour TTL
    enable_compression: true,     // Enable value compression
};

let cache = RedbStorage::with_full_config("./data/cache.redb", config)?;

Environment Variables

# Optional: Configure cache settings
export REDB_CACHE_PATH="./data/cache.redb"
export REDB_MAX_CACHE_SIZE="1000"
export REDB_TTL_SECONDS="3600"

Cache Tables

The cache maintains four tables for different data types:

• episodes: Full episode data for fast retrieval
• patterns: Extracted patterns with similarity keys
• embeddings: Vector embeddings (if enabled)
• metadata: Cache statistics and health info

Performance

Optimized for extremely fast read operations:

Operation	Latency	Throughput
Cache Read	< 10µs	100K+ ops/s
Cache Write	< 100µs	10K+ ops/s
LRU Eviction	< 1ms	Background
TTL Cleanup	< 10ms	Background

Real-world Performance

In production with do-memory-core (v0.1.13):

• Cache hit rate: 85-95% (varies by workload)
• Average read latency: 5-8µs (cache hits)
• Average write latency: 50-80µs
• Throughput: 50K+ reads/s, 8K+ writes/s

LRU Cache Strategy

The cache uses an LRU (Least Recently Used) eviction policy:

1. Cache fills up to max_cache_size
2. New entries trigger eviction of oldest entries
3. Access updates entry timestamp
4. TTL expiration runs in background

LRU Behavior

// Entries are tracked by last access time
cache.get("episode-123")?;  // Updates access time

// When cache is full, oldest entry is evicted
cache.insert("episode-456", data)?;  // May trigger eviction

TTL Management

Time-to-live (TTL) automatically expires old entries:

use memory_storage_redb::{RedbConfig, RedbStorage};

let config = RedbConfig {
    ttl_seconds: Some(3600),  // Expire after 1 hour
    ..Default::default()
};

let cache = RedbStorage::with_full_config("./data/cache.redb", config)?;

// TTL cleanup runs every 60 seconds in background
// Expired entries are automatically removed

TTL Settings

Setting	Description	Default
ttl_seconds	Time-to-live in seconds	None (no TTL)
ttl_check_interval	Cleanup interval in seconds	60

Usage with do-memory-core

use memory_core::SelfLearningMemory;
use memory_storage_turso::TursoStorage;
use memory_storage_redb::RedbStorage;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Turso for durable storage
    let turso = TursoStorage::new(
        std::env::var("TURSO_DATABASE_URL")?,
        std::env::var("TURSO_AUTH_TOKEN")?,
    ).await?;

    // redb for fast caching
    let redb = RedbStorage::new("./data/cache.redb")?;

    // Hybrid storage for best performance
    let memory = SelfLearningMemory::with_storage(turso, redb).await?;

    Ok(())
}

Cache Synchronization

The cache automatically synchronizes with Turso:

• Write-through: Writes go to both Turso and redb
• Read-aside: Reads check cache first, then Turso
• Invalidation: Cache can be refreshed from Turso
• Reconciliation: Periodic sync ensures consistency

// Manual cache refresh (force sync from Turso)
memory.sync_memories().await?;

// Get cache statistics
let stats = cache.get_stats()?;
println!("Cache hit rate: {:.2}%", stats.hit_rate * 100.0);
println!("Total entries: {}", stats.total_entries);
println!("Cache size: {} bytes", stats.cache_size_bytes);

Background Tasks

The cache runs background tasks for maintenance:

• TTL Cleanup: Removes expired entries (every 60s)
• LRU Eviction: Maintains size limits (on write)
• Compression: Optional value compression (configurable)
• Health Checks: Monitors cache health (on demand)

Background Task Configuration

use memory_storage_redb::{RedbConfig, RedbStorage};

let config = RedbConfig {
    ttl_check_interval: 30,      // Check TTL every 30s
    enable_compression: true,     // Compress values
    compression_threshold: 1024, // Compress values > 1KB
    ..Default::default()
};

let cache = RedbStorage::with_full_config("./data/cache.redb", config)?;

File Management

redb stores all data in a single file:

# Check cache file size
ls -lh ./data/cache.redb

# Backup cache
cp ./data/cache.redb ./backups/cache-$(date +%Y%m%d).redb

# Clear cache (forces rebuild from Turso)
rm ./data/cache.redb

# Monitor cache growth
watch -n 5 'ls -lh ./data/cache.redb'

Cache File Structure

• Single file database: All tables and data in one file
• Memory mapped: Direct access for zero-copy reads
• Crash recovery: Automatic recovery from unclean shutdown
• Incremental writes: Only modified data is written

Compression

Optional value compression reduces cache size:

let config = RedbConfig {
    enable_compression: true,          // Enable compression
    compression_threshold: 1024,        // Compress values > 1KB
    compression_level: 6,               // Compression level (0-9)
    ..Default::default()
};

let cache = RedbStorage::with_full_config("./data/cache.redb", config)?;

Compression Benefits

• Reduced memory usage: 40-60% smaller cache files
• Faster disk I/O: Less data to read/write
• Trade-off: Slightly higher CPU usage

Monitoring & Statistics

Cache Statistics

let stats = cache.get_stats()?;
println!("=== Cache Statistics ===");
println!("Hit rate: {:.2}%", stats.hit_rate * 100.0);
println!("Total entries: {}", stats.total_entries);
println!("Cache size: {} bytes", stats.cache_size_bytes);
println!("Reads: {}", stats.total_reads);
println!("Writes: {}", stats.total_writes);
println!("Evictions: {}", stats.total_evictions);

Health Check

let health = cache.health_check()?;
println!("Cache healthy: {}", health.is_healthy);
println!("Disk available: {} bytes", health.available_disk_space);
println!("Cache file: {} bytes", health.cache_file_size);

Testing

Run tests with an in-memory database:

cargo test -p do-memory-storage-redb

For integration tests with a real database file:

cargo test -p do-memory-storage-redb -- --ignored

Dependencies

Core Dependencies

• redb: Embedded key-value database
• tokio: Async runtime
• async-trait: Async trait support
• anyhow: Error handling
• serde: Serialization framework
• postcard: Serialization format (v0.1.7+, current: v0.1.13)

Breaking Change Dependencies

• postcard (v0.1.7+, current: v0.1.13): Replaces bincode for safer serialization

Documentation

Full API documentation: docs.rs/do-memory-storage-redb

Best Practices

Production Deployment

• Enable TTL for automatic cleanup
• Set appropriate max_cache_size based on memory constraints
• Monitor cache hit rate to optimize size
• Enable compression for large datasets
• Run periodic health checks

Development

• Use smaller cache sizes for local development
• Disable compression for easier debugging
• Monitor cache file growth
• Clear cache frequently during development

Performance Tuning

• Adjust max_cache_size to balance memory and hit rate
• Use TTL for time-sensitive data
• Enable compression for memory-constrained environments
• Monitor eviction rate to detect undersized cache

Cache Size Guidelines

Workload	Max Cache Size	Expected Hit Rate
Low traffic	100-500	85-90%
Medium traffic	500-2000	90-95%
High traffic	2000-10000	95-98%

Troubleshooting

Cache Not Populating

// Check Turso connection first
let turso = TursoStorage::new(url, token).await?;
let episodes = turso.get_all_episodes().await?;
println!("Turso has {} episodes", episodes.len());

// Force cache sync
memory.sync_memories().await?;

High Eviction Rate

let stats = cache.get_stats()?;
if stats.total_evictions > stats.total_writes / 2 {
    // Increase cache size
    let config = RedbConfig {
        max_cache_size: stats.total_entries * 2,
        ..Default::default()
    };
}

Low Cache Hit Rate

let stats = cache.get_stats()?;
if stats.hit_rate < 0.8 {
    // Check if cache is being used correctly
    // Verify do-memory-core is configured to use cache
    println!("Consider increasing cache size or TTL");
}

License

Licensed under the MIT License. See LICENSE for details.

Project

Part of the rust-self-learning-memory project.

Version History

• v0.1.13 (Current): Postcard serialization, improved performance
• v0.1.7: Postcard serialization introduced
• v0.1.6: Initial release with bincode serialization