# Codex Memory Technical Architecture
**Version**: 3.0.0
**Last Updated**: 2025-09-01
**Document Type**: Technical Specification
## Table of Contents
1. [System Specifications](#system-specifications)
2. [API Specifications](#api-specifications)
3. [Database Specifications](#database-specifications)
4. [Performance Requirements](#performance-requirements)
5. [Integration Specifications](#integration-specifications)
6. [Build and Deployment](#build-and-deployment)
7. [Error Handling](#error-handling)
8. [Testing Requirements](#testing-requirements)
## System Specifications
### Core Requirements
| **Language** | Rust 1.70+ | Cargo.toml rust-version = "1.70" |
| **Runtime** | Tokio 1.35+ | Async runtime for all I/O operations |
| **Database** | PostgreSQL 14+ with pgvector | Version check on startup |
| **Protocol** | MCP 2024-11-05 | JSON-RPC 2.0 over stdio |
| **Binary Size** | <15MB (base), <20MB (with patterns) | Release build measurement |
| **Memory Usage** | <50MB steady state | Production monitoring |
### Module Structure
```
codex-memory/
├── Cargo.toml # Dependencies and features
├── src/
│ ├── main.rs # Entry point, CLI parsing
│ ├── lib.rs # Public API exports
│ ├── config.rs # Environment configuration
│ ├── database/
│ │ ├── mod.rs # Database module interface
│ │ ├── pool.rs # Connection pool management
│ │ ├── migrations.rs # Schema migrations
│ │ └── queries.rs # SQL query definitions
│ ├── models/
│ │ ├── mod.rs # Data structures
│ │ ├── memory.rs # Memory model
│ │ ├── processed.rs # ProcessedMemory model
│ │ └── patterns.rs # CodePattern model (optional)
│ ├── storage/
│ │ ├── mod.rs # Storage trait definition
│ │ ├── postgres.rs # PostgreSQL implementation
│ │ └── cache.rs # Optional caching layer
│ ├── mcp/
│ │ ├── mod.rs # MCP server implementation
│ │ ├── protocol.rs # JSON-RPC handling
│ │ ├── tools.rs # Tool definitions
│ │ └── handlers.rs # Request handlers
│ └── patterns/ # Optional pattern analysis
│ ├── mod.rs # Feature-gated module
│ ├── analyzer.rs # AST analysis
│ └── languages.rs # Language-specific parsers
└── tests/
├── integration/ # Integration tests
└── benchmarks/ # Performance benchmarks
```
### Feature Flags
```toml
[features]
default = []
pattern-learning = ["dep:tree-sitter", "dep:tree-sitter-rust", "dep:tree-sitter-python"]
metrics = ["dep:prometheus", "dep:opentelemetry"]
cache = ["dep:moka"]
[dependencies]
# Core dependencies (always included)
tokio = { version = "1.35", features = ["full"] }
sqlx = { version = "0.7", features = ["runtime-tokio-native-tls", "postgres", "uuid", "json", "chrono"] }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
uuid = { version = "1.6", features = ["v4", "serde"] }
chrono = { version = "0.4", features = ["serde"] }
sha2 = "0.10"
clap = { version = "4.4", features = ["derive", "env"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
anyhow = "1.0"
thiserror = "1.0"
# Optional dependencies
tree-sitter = { version = "0.20", optional = true }
tree-sitter-rust = { version = "0.20", optional = true }
tree-sitter-python = { version = "0.20", optional = true }
prometheus = { version = "0.13", optional = true }
opentelemetry = { version = "0.21", optional = true }
moka = { version = "0.12", optional = true }
```
## API Specifications
### Storage Trait
```rust
#[async_trait]
pub trait Storage: Send + Sync {
/// Store pre-enriched memory
/// Returns: UUID of stored memory
/// Errors: DatabaseError, ValidationError
async fn store(&self, memory: CreateMemory) -> Result<Uuid>;
/// Retrieve memory by ID
/// Returns: Memory if found
/// Errors: DatabaseError
async fn get(&self, id: Uuid) -> Result<Option<Memory>>;
/// Search memories
/// Returns: Paginated results
/// Errors: DatabaseError, InvalidQuery
async fn search(&self, query: SearchQuery) -> Result<SearchResults>;
/// Delete memory
/// Returns: true if deleted
/// Errors: DatabaseError
async fn delete(&self, id: Uuid) -> Result<bool>;
/// Get storage statistics
/// Returns: Storage metrics
/// Errors: DatabaseError
async fn stats(&self) -> Result<StorageStats>;
}
```
### Data Models
```rust
pub struct CreateMemory {
pub content: String, // Required, max 100MB
pub context: String, // Required, max 1000 chars
pub summary: String, // Required, max 500 chars
pub tags: Vec<String>, // Required, max 50 tags
pub insights: Option<JsonValue>, // Pre-computed, max 64KB
pub embeddings: Option<Vec<f32>>, // Pre-computed, dimension must match
pub entities: Option<JsonValue>, // Pre-computed, max 32KB
pub sentiment: Option<f32>, // Pre-computed, -1.0 to 1.0
}
pub struct Memory {
pub id: Uuid,
pub content: String,
pub content_hash: String, // SHA-256 hex
pub context: String,
pub summary: String,
pub tags: Vec<String>,
pub metadata: JsonValue,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
}
pub struct ProcessedMemory {
pub id: Uuid,
pub memory_id: Uuid,
pub insights: Option<JsonValue>,
pub embeddings: Option<Vec<f32>>,
pub entities: Option<JsonValue>,
pub sentiment: Option<f32>,
pub processed_at: DateTime<Utc>,
pub processing_version: i32,
}
pub struct SearchQuery {
pub tags: Option<Vec<String>>, // Filter by tags (OR)
pub context: Option<String>, // Full-text search in context
pub summary: Option<String>, // Full-text search in summary
pub from_date: Option<DateTime<Utc>>,
pub to_date: Option<DateTime<Utc>>,
pub limit: i32, // Max 100, default 20
pub offset: i32, // For pagination
pub order_by: OrderBy, // created_at_desc (default)
}
```
### MCP Tool Specifications
#### Base Tools (Always Available)
| `store_memory` | content, context, summary, tags?, metadata?, insights?, embeddings? | UUID | content <= 1MB |
| `get_memory` | id: UUID | Memory object or null | Valid UUID required |
| `search_memories` | query: SearchQuery | Memory[] | limit <= 100 |
| `delete_memory` | id: UUID | boolean | Returns false if not found |
| `get_statistics` | none | StorageStats | Cached 60s |
#### Pattern Tools (Feature: pattern-learning)
| `analyze_patterns` | memory_id: UUID, language: string | PatternAnalysis | Supported languages only |
| `get_patterns` | memory_id: UUID | CodePattern[] | Max 1000 patterns |
| `search_patterns` | pattern_type: string, limit?: number | CodePattern[] | limit <= 100 |
| `pattern_statistics` | from_date?: DateTime, to_date?: DateTime | PatternStats | Date range <= 90 days |
## Database Specifications
### Connection Pool Configuration
```rust
pub struct DatabaseConfig {
pub url: String, // DATABASE_URL env var
pub max_connections: u32, // Default: 20
pub min_connections: u32, // Default: 2
pub acquire_timeout: Duration, // Default: 30s
pub idle_timeout: Duration, // Default: 600s
pub max_lifetime: Duration, // Default: 1800s
pub test_before_acquire: bool, // Default: true
}
```
### Schema Definition
```sql
-- Primary schema
CREATE SCHEMA IF NOT EXISTS public;
CREATE SCHEMA IF NOT EXISTS codex_processed;
-- Enable extensions
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS "pgvector";
-- Memories table (public schema)
CREATE TABLE public.memories (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
content TEXT NOT NULL CHECK (length(content) <= 1048576),
content_hash VARCHAR(64) NOT NULL,
context TEXT NOT NULL CHECK (length(context) <= 1000),
summary TEXT NOT NULL CHECK (length(summary) <= 500),
tags TEXT[] DEFAULT '{}' CHECK (array_length(tags, 1) <= 50),
metadata JSONB DEFAULT '{}' CHECK (pg_column_size(metadata) <= 16384),
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
-- Processed memories table
CREATE TABLE codex_processed.processed_memories (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
memory_id UUID NOT NULL REFERENCES public.memories(id) ON DELETE CASCADE,
insights JSONB CHECK (pg_column_size(insights) <= 65536),
embeddings vector(1536), -- Dimension fixed at creation
entities JSONB CHECK (pg_column_size(entities) <= 32768),
sentiment REAL CHECK (sentiment >= -1.0 AND sentiment <= 1.0),
processed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
processing_version INTEGER NOT NULL DEFAULT 1,
UNIQUE(memory_id, processing_version)
);
-- Code patterns table (optional)
CREATE TABLE IF NOT EXISTS codex_processed.code_patterns (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
memory_id UUID NOT NULL REFERENCES public.memories(id) ON DELETE CASCADE,
pattern_type VARCHAR(50) NOT NULL,
pattern_signature TEXT NOT NULL,
frequency REAL NOT NULL CHECK (frequency >= 0 AND frequency <= 1),
confidence REAL NOT NULL CHECK (confidence >= 0 AND confidence <= 1),
ast_features JSONB,
language VARCHAR(20) NOT NULL,
discovered_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
UNIQUE(memory_id, pattern_type, pattern_signature)
);
-- Indexes
CREATE UNIQUE INDEX idx_memories_content_hash ON public.memories(content_hash);
CREATE INDEX idx_memories_created_at ON public.memories(created_at DESC);
CREATE INDEX idx_memories_tags ON public.memories USING GIN(tags);
CREATE INDEX idx_memories_metadata ON public.memories USING GIN(metadata);
CREATE INDEX idx_memories_context_fts ON public.memories USING GIN(to_tsvector('english', context));
CREATE INDEX idx_memories_summary_fts ON public.memories USING GIN(to_tsvector('english', summary));
CREATE INDEX idx_processed_memory_id ON codex_processed.processed_memories(memory_id);
CREATE INDEX idx_processed_embeddings ON codex_processed.processed_memories USING ivfflat(embeddings vector_cosine_ops);
CREATE INDEX idx_patterns_memory_id ON codex_processed.code_patterns(memory_id);
CREATE INDEX idx_patterns_type_lang ON codex_processed.code_patterns(pattern_type, language);
CREATE INDEX idx_patterns_frequency ON codex_processed.code_patterns(frequency DESC);
```
### Migration Strategy
```sql
-- Version tracking
CREATE TABLE IF NOT EXISTS schema_migrations (
version INTEGER PRIMARY KEY,
applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
checksum VARCHAR(64) NOT NULL
);
-- Migrations must be:
-- 1. Idempotent (IF NOT EXISTS)
-- 2. Transactional
-- 3. Forward-only (no rollbacks)
-- 4. Include checksums for verification
```
## Performance Requirements
### Latency Requirements
| Store (no patterns) | 3ms | 5ms | 8ms | 20ms |
| Store (with patterns) | 10ms | 30ms | 50ms | 100ms |
| Get by ID | 1ms | 2ms | 3ms | 10ms |
| Search (simple) | 5ms | 10ms | 15ms | 50ms |
| Search (complex) | 10ms | 20ms | 30ms | 100ms |
| Delete | 2ms | 3ms | 5ms | 20ms |
| Statistics | 10ms | 15ms | 20ms | 50ms |
### Throughput Requirements
| Concurrent connections | 20 | Connection pool size |
| Requests per second | 1000 | Single instance |
| Storage rate | 100 memories/sec | Sustained load |
| Search QPS | 500 | With caching |
| Pattern analysis | 10 files/sec | AST parsing |
### Resource Requirements
| Memory | 128MB | 256MB | 512MB |
| CPU | 0.5 core | 1 core | 2 cores |
| Disk IOPS | 100 | 500 | 2000 |
| Network | 10 Mbps | 100 Mbps | 1 Gbps |
| PostgreSQL connections | 2 | 10 | 20 |
## Integration Specifications
### MCP Protocol Implementation
```json
{
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"name": "store_memory",
"arguments": {
"content": "string",
"context": "string",
"summary": "string",
"tags": ["string"],
"insights": {},
"embeddings": [0.1, 0.2, ...],
"entities": {},
"sentiment": 0.5
}
},
"id": "unique-request-id"
}
```
### Environment Variables
| DATABASE_URL | Yes | - | PostgreSQL connection string |
| RUST_LOG | No | info | Log level (error/warn/info/debug/trace) |
| MAX_CONNECTIONS | No | 20 | Database pool maximum |
| MIN_CONNECTIONS | No | 2 | Database pool minimum |
| ACQUIRE_TIMEOUT | No | 30 | Connection acquire timeout (seconds) |
| ENABLE_CACHE | No | false | Enable in-memory cache |
| CACHE_SIZE | No | 1000 | Max cached items |
| CACHE_TTL | No | 300 | Cache TTL (seconds) |
| PATTERN_LANGUAGES | No | rust,python | Comma-separated languages |
| MCP_TIMEOUT | No | 60 | MCP request timeout (seconds) |
## Error Handling
### Error Types
```rust
#[derive(thiserror::Error, Debug)]
pub enum StorageError {
#[error("Database error: {0}")]
Database(#[from] sqlx::Error),
#[error("Validation error: {0}")]
Validation(String),
#[error("Not found: {0}")]
NotFound(String),
#[error("Duplicate content: hash {0}")]
Duplicate(String),
#[error("Size limit exceeded: {0}")]
SizeLimit(String),
#[error("Connection pool exhausted")]
PoolExhausted,
#[error("Pattern analysis failed: {0}")]
PatternAnalysis(String),
}
```
### Error Response Format
```json
{
"jsonrpc": "2.0",
"error": {
"code": -32603,
"message": "Internal error",
"data": {
"type": "StorageError::Database",
"details": "Connection timeout"
}
},
"id": "request-id"
}
```
### Error Codes
| -32700 | Parse error | Invalid JSON |
| -32600 | Invalid request | Invalid method |
| -32601 | Method not found | Unknown tool |
| -32602 | Invalid params | Invalid arguments |
| -32603 | Internal error | Storage/database error |
| -32000 | Validation error | Input validation failed |
| -32001 | Not found | Resource not found |
| -32002 | Duplicate | Content already exists |
| -32003 | Size limit | Input too large |
## Testing Requirements
### Unit Test Coverage
| storage | 90% | - |
| models | 95% | - |
| database | 85% | - |
| mcp | 80% | - |
| patterns | 75% | - |
### Integration Test Scenarios
```rust
#[tokio::test]
async fn test_store_and_retrieve()
#[tokio::test]
async fn test_deduplication()
#[tokio::test]
async fn test_search_by_tags()
#[tokio::test]
async fn test_concurrent_operations()
#[tokio::test]
async fn test_connection_pool_exhaustion()
#[tokio::test]
async fn test_large_content_handling()
#[tokio::test]
async fn test_pattern_analysis()
```
### Performance Benchmarks
```rust
#[bench]
fn bench_store_memory(b: &mut Bencher)
#[bench]
fn bench_get_memory(b: &mut Bencher)
#[bench]
fn bench_search_memories(b: &mut Bencher)
#[bench]
fn bench_pattern_analysis(b: &mut Bencher)
```
### Load Testing Requirements
| Sustained load | 1 hour | 100 req/s | 99% success, P95 < 10ms |
| Burst load | 5 minutes | 1000 req/s | 95% success, P95 < 50ms |
| Concurrent users | 30 minutes | 100 concurrent | No deadlocks |
| Memory leak test | 24 hours | 10 req/s | Memory < 500MB |
## Monitoring and Observability
### Metrics (Optional feature)
```rust
// Prometheus metrics
codex_memory_operations_total{operation="store", status="success"}
codex_memory_operation_duration_seconds{operation="store", quantile="0.95"}
codex_memory_pool_connections{state="active"}
codex_memory_pool_connections{state="idle"}
codex_memory_cache_hits_total
codex_memory_cache_misses_total
codex_memory_pattern_analysis_duration_seconds{language="rust"}
```
### Health Checks
```rust
GET /health/ready
"status": "healthy|unhealthy",
"database": "connected|disconnected",
"pool_usage": 0.35,
"version": "3.0.0"
}
GET /health/live
Response: 200 OK
{
"status": "alive",
"uptime_seconds": 3600
}
```
### Logging Standards
```rust
// Log levels and examples
ERROR: Database connection failed: timeout
WARN: Connection pool usage above 70%: 14/20
INFO: Memory stored successfully: id=uuid
DEBUG: Search query executed: 5ms, 10 results
TRACE: SQL query: SELECT * FROM memories WHERE...
```
## Security Requirements
### Input Validation
| content | Max 1MB, UTF-8 valid |
| context | Max 1000 chars, no control characters |
| summary | Max 500 chars, no control characters |
| tags | Max 50 tags, each max 50 chars, alphanumeric + dash |
| embeddings | Exactly 1536 dimensions, valid floats |
| sentiment | Range -1.0 to 1.0 |
### SQL Injection Prevention
- All queries use parameterized statements
- No string concatenation for SQL
- Input validation before query construction
- Prepared statement caching
### Rate Limiting
| store_memory | 100 | 1 minute |
| search_memories | 300 | 1 minute |
| get_memory | 1000 | 1 minute |
| analyze_patterns | 10 | 1 minute |
## Deployment Checklist
### Pre-deployment
- [ ] Database migrations applied
- [ ] Environment variables configured
- [ ] Connection string validated
- [ ] Health checks passing
- [ ] Performance benchmarks met
- [ ] Security scan completed
### Post-deployment
- [ ] Health endpoints responding
- [ ] Metrics being collected
- [ ] Logs being aggregated
- [ ] Latency within SLA
- [ ] Error rate < 0.1%
- [ ] Database connections stable