# rust-memex
[](https://crates.io/crates/rust-memex) [](LICENSE) [](https://crates.io/crates/rust-memex) [](https://github.com/Loctree/rust-memex/actions)
`rust-memex` is a custom Rust MCP kernel providing RAG and long-term memory capabilities to AI agents via LanceDB.
It exposes two explicit transport modes from a single canonical surface:
1. **`stdio` (Standard MCP)**: Native MCP integration for local agents (e.g., Claude Desktop).
2. **`HTTP/SSE` (Multi-Agent Daemon)**: A central daemon mode allowing concurrent AI agents to access the same memory pool over the network, resolving LanceDB's exclusive lock constraints.
> **Binary Name:** `rust-memex` is the only supported binary name. The GitHub installer also creates `rust_memex` as a legacy compatibility symlink for older scripts.
>
> **MCP Contract:** The current MCP surface is intentionally tools-only. `initialize` advertises `tools`, while `resources/*` is not implemented yet.
## Release Surface
- Quick install: `curl -LsSf https://raw.githubusercontent.com/Loctree/rust-memex/main/install.sh | sh`
- Prebuilt binary bundles: [GitHub Releases](https://github.com/Loctree/rust-memex/releases) uploaded from locally built and signed artifacts
- Release runbook: [docs/RELEASE.md](docs/RELEASE.md)
- Configuration guide: [docs/02_configuration.md](docs/02_configuration.md)
- HTTP/SSE reference: [docs/HTTP_API.md](docs/HTTP_API.md)
- Static launch page source: `docs/index.html` published by `.github/workflows/pages.yml`
## Quick Start
```bash
# Install from the latest GitHub Release
# Start the MCP server
rust-memex serve
# Open the local dashboard in your browser
rust-memex dashboard
# Or run the multi-agent HTTP/SSE daemon
rust-memex sse
```
## Overview
As an MCP (Model Context Protocol) server, `rust-memex` provides:
- **RAG (Retrieval-Augmented Generation)** - document indexing and semantic search
- **Hybrid Search** - BM25 keyword + semantic vector search (Tantivy-based)
- **Vector Memory** - semantic storage and retrieval of text chunks
- **Namespace Isolation** - data isolation in namespaces
- **Security** - token-based access control for protected namespaces
- **Onion Slice Architecture** - hierarchical embeddings (OUTER→MIDDLE→INNER→CORE)
- **Preprocessing** - automatic noise filtering from conversation exports (~36-40% reduction)
- **Exact-Match Deduplication** - SHA256-based dedup for overlapping exports
## Architecture
```
┌─────────────────────────────────────────────────────────────┐
│ rust-memex │
├─────────────────────────────────────────────────────────────┤
│ MCP Server (JSON-RPC over stdio) │
│ ├── handlers/mod.rs - Request routing & validation │
│ ├── security/mod.rs - Namespace access control │
│ └── rag/mod.rs - RAG pipeline │
├─────────────────────────────────────────────────────────────┤
│ Storage Layer │
│ ├── LanceDB - Vector embeddings │
│ ├── Tantivy - BM25 keyword index │
│ └── moka - In-memory cache │
├─────────────────────────────────────────────────────────────┤
│ Embeddings (External Providers) │
│ ├── Ollama - Local models (recommended) │
│ ├── MLX Bridge - Apple Silicon acceleration │
│ └── OpenAI-compatible - Any compatible endpoint │
└─────────────────────────────────────────────────────────────┘
```
## Features
### RAG Tools
| `rag_index` | Index document from file |
| `rag_index_text` | Index raw text |
| `rag_search` | Search documents semantically (supports `auto_route`) |
### Memory Tools
| `memory_upsert` | Add/update chunk in namespace |
| `memory_get` | Get chunk by ID |
| `memory_search` | Search semantically in namespace (supports `auto_route`) |
| `memory_delete` | Delete chunk |
| `memory_purge_namespace` | Delete all chunks in namespace |
| `dive` | Deep exploration with all onion layers (outer/middle/inner/core) |
### Security Tools
| `namespace_create_token` | Create access token for namespace |
| `namespace_revoke_token` | Revoke token (namespace becomes public) |
| `namespace_list_protected` | List protected namespaces |
| `namespace_security_status` | Security system status |
## Library Usage
`rust-memex` can be used as a library in your Rust applications. It provides a high-level `MemexEngine` API for vector storage operations.
### Add to Cargo.toml
```toml
# Full library with CLI
rust-memex = "0.5"
# Library only (no CLI dependencies)
rust-memex = { version = "0.5", default-features = false }
```
### Basic Usage
```rust
use rust_memex::{MemexEngine, MemexConfig, MetaFilter, StoreItem};
use serde_json::json;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// Quick setup for any application
let engine = MemexEngine::for_app("my-app", "documents").await?;
// Store a document
engine.store(
"doc-1",
"Patient presented with lethargy and decreased appetite",
json!({"patient_id": "P-123", "visit_type": "checkup"})
).await?;
// Search semantically
let results = engine.search("lethargy symptoms", 10).await?;
for r in &results {
println!("{}: {} (score: {:.2})", r.id, r.text, r.score);
}
// Get by ID
if let Some(doc) = engine.get("doc-1").await? {
println!("Found: {}", doc.text);
}
// Delete
engine.delete("doc-1").await?;
Ok(())
}
```
### Vista Integration
For Vista PIMS, use the optimized constructor:
```rust
use rust_memex::MemexEngine;
// Vista-optimized: 1024 dims, qwen3-embedding:0.6b model
let engine = MemexEngine::for_vista().await?;
// Store visit notes
engine.store(
"visit-456",
"SOAP note: Feline diabetes mellitus diagnosis...",
json!({"patient_id": "P-789", "doc_type": "soap_note"})
).await?;
```
### Batch Operations
```rust
use rust_memex::{MemexEngine, StoreItem};
use serde_json::json;
let engine = MemexEngine::for_app("my-app", "notes").await?;
let items = vec![
StoreItem::new("doc-1", "First document").with_metadata(json!({"type": "note"})),
StoreItem::new("doc-2", "Second document").with_metadata(json!({"type": "note"})),
StoreItem::new("doc-3", "Third document").with_metadata(json!({"type": "note"})),
];
let result = engine.store_batch(items).await?;
println!("Stored {} documents", result.success_count);
```
### GDPR-Compliant Deletion
```rust
use rust_memex::{MemexEngine, MetaFilter};
let engine = MemexEngine::for_app("my-app", "patients").await?;
// Delete all documents for a specific patient
let filter = MetaFilter::for_patient("P-123");
let deleted = engine.delete_by_filter(filter).await?;
println!("Deleted {} documents", deleted);
```
### Hybrid Search (BM25 + Vector)
```rust
use rust_memex::{MemexEngine, SearchMode};
let engine = MemexEngine::for_app("my-app", "documents").await?;
// Hybrid search with BM25 + vector fusion (recommended)
let results = engine.search_hybrid("dragon mac studio", 10).await?;
for r in &results {
println!("{}: {} (combined: {:.2}, vector: {:.2}, bm25: {:.2})",
r.id, r.document, r.combined_score, r.vector_score, r.bm25_score);
}
// Explicit mode selection
let results = engine.search_with_mode("exact keyword", 10, SearchMode::Keyword).await?;
let results = engine.search_with_mode("semantic concept", 10, SearchMode::Vector).await?;
let results = engine.search_with_mode("best of both", 10, SearchMode::Hybrid).await?;
```
### MCP Contract and In-Process Helpers
Use `tool_definitions()` when you need the exact MCP tool metadata exposed by
the stdio and HTTP/SSE transports. The helper functions below are for in-process
Rust callers; they are a local convenience layer, not a second public MCP
contract. Their names intentionally differ from MCP tool names so the two
surfaces do not drift together by accident.
```rust
use rust_memex::{MemexEngine, search_documents, store_document, tool_definitions};
use serde_json::json;
let engine = MemexEngine::for_app("agent", "memory").await?;
// Canonical MCP tool surface exposed by rust-memex transports
let tools = tool_definitions();
// Local Rust helpers for in-process callers
let result = store_document(
&engine,
"mem-1".to_string(),
"Important information to remember".to_string(),
json!({"source": "conversation"}),
).await?;
assert!(result.success);
let results = search_documents(&engine, "important".to_string(), 5, None).await?;
```
### Feature Flags
| `cli` | CLI binary, TUI wizard, progress bars | Yes |
| `provider-cascade` | Ollama/OpenAI-compatible embeddings | Yes |
```bash
# Build library only (no CLI)
cargo build --no-default-features
# Build with CLI
cargo build --features cli
```
---
## Configuration Guide
Complete guide for integrating rust-memex as a library in any Rust project.
### Prerequisites
**Ollama** (recommended) or any OpenAI-compatible embedding API:
```bash
# Install Ollama
# Pull an embedding model (choose based on your needs)
ollama pull qwen3-embedding:0.6b # 1024 dims, ~600MB (fast, good quality)
ollama pull qwen3-embedding:8b # 4096 dims, ~4GB (best quality)
ollama pull nomic-embed-text # 768 dims, ~274MB (lightweight)
# Verify it's running
curl http://localhost:11434/api/tags
```
### Environment Variables
Configure via `.env` or environment:
```bash
# =============================================================================
# EMBEDDING PROVIDER CONFIGURATION
# =============================================================================
# Ollama (default, recommended)
OLLAMA_BASE_URL=http://localhost:11434
EMBEDDING_MODEL=qwen3-embedding:0.6b
EMBEDDING_DIMENSION=1024
# Database storage (auto-created)
MEMEX_DB_PATH=~/.rmcp-servers/myapp/lancedb
# Optional: BM25 keyword search index
MEMEX_BM25_PATH=~/.rmcp-servers/myapp/bm25
# =============================================================================
# ADVANCED: Multiple providers (fallback cascade)
# =============================================================================
# Remote embedding server fallback
# DRAGON_BASE_URL=http://your-server.local
# DRAGON_EMBEDDER_PORT=12345
# MLX embedder for Apple Silicon
# EMBEDDER_PORT=12300
# MLX_MAX_BATCH_CHARS=32000
# MLX_MAX_BATCH_ITEMS=16
# DISABLE_MLX=1 # Set to disable MLX fallback
```
### Quick Start (Auto-config)
```rust
use rust_memex::MemexEngine;
use serde_json::json;
// Auto-configures from defaults + environment
let engine = MemexEngine::for_app("my-app", "default").await?;
engine.store("doc-1", "Document content...", json!({"type": "note"})).await?;
let results = engine.search("content", 10).await?;
```
### Custom Configuration
```rust
use rust_memex::{MemexConfig, MemexEngine};
use rust_memex::embeddings::{EmbeddingConfig, ProviderConfig};
// Read from your app's environment
let ollama_url = std::env::var("OLLAMA_BASE_URL")
.unwrap_or_else(|_| "http://localhost:11434".to_string());
let model = std::env::var("EMBEDDING_MODEL")
.unwrap_or_else(|_| "qwen3-embedding:0.6b".to_string());
let dimension: usize = std::env::var("EMBEDDING_DIMENSION")
.unwrap_or_else(|_| "1024".to_string())
.parse()
.unwrap_or(1024);
let db_path = std::env::var("MEMEX_DB_PATH")
.unwrap_or_else(|_| "~/.rmcp-servers/myapp/lancedb".to_string());
let config = MemexConfig {
app_name: "my-app".to_string(),
namespace: "default".to_string(),
db_path: Some(db_path),
dimension,
embedding_config: EmbeddingConfig {
required_dimension: dimension,
providers: vec![ProviderConfig {
name: "ollama".to_string(),
base_url: ollama_url,
model,
priority: 1,
endpoint: "/v1/embeddings".to_string(),
}],
..EmbeddingConfig::default()
},
enable_bm25: false,
bm25_config: None,
};
let engine = MemexEngine::new(config).await?;
```
### Provider Cascade (Multiple Fallbacks)
Configure multiple providers - library tries them in priority order:
```rust
use rust_memex::embeddings::{EmbeddingConfig, ProviderConfig};
let config = EmbeddingConfig {
required_dimension: 1024,
providers: vec![
// Priority 1: Local Ollama (fastest)
ProviderConfig {
name: "ollama-local".to_string(),
base_url: "http://localhost:11434".to_string(),
model: "qwen3-embedding:0.6b".to_string(),
priority: 1,
endpoint: "/v1/embeddings".to_string(),
},
// Priority 2: Remote server fallback
ProviderConfig {
name: "remote-server".to_string(),
base_url: "http://your-server:8080".to_string(),
model: "text-embedding-3-small".to_string(),
priority: 2,
endpoint: "/v1/embeddings".to_string(),
},
// Priority 3: OpenAI API fallback
ProviderConfig {
name: "openai".to_string(),
base_url: "https://api.openai.com".to_string(),
model: "text-embedding-3-small".to_string(),
priority: 3,
endpoint: "/v1/embeddings".to_string(),
},
],
..EmbeddingConfig::default()
};
```
### Namespace Strategy
**Recommended: One namespace per application, use metadata for filtering:**
```rust
// ✅ CORRECT: Single namespace, filter by user_id/entity_id in metadata
let engine = MemexEngine::for_app("my-app", "default").await?;
// Store with entity IDs in metadata
engine.store("doc-1", "Document content...", json!({
"user_id": "U-123", // For multi-tenant filtering
"project_id": "P-456", // For project-level filtering
"doc_type": "note",
"created_at": "2024-12-28"
})).await?;
// Search within user context
let filter = MetaFilter::default().with_custom("user_id", "U-123");
let results = engine.search_filtered("query", filter, 10).await?;
// GDPR deletion: remove all user data
let deleted = engine.delete_by_filter(
MetaFilter::default().with_custom("user_id", "U-123")
).await?;
```
### Embedding Models Reference
| `qwen3-embedding:0.6b` | 1024 | ~600MB | Fast, good quality (recommended) |
| `qwen3-embedding:8b` | 4096 | ~4GB | Best quality, slower |
| `nomic-embed-text` | 768 | ~274MB | Lightweight, fast |
| `mxbai-embed-large` | 1024 | ~670MB | Good multilingual |
| `all-minilm` | 384 | ~46MB | Very fast, lower quality |
### Troubleshooting
**Error: "No embedding providers available"**
```bash
# Check if Ollama is running
curl http://localhost:11434/api/tags
# Start Ollama
ollama serve
# Pull model if missing
ollama pull qwen3-embedding:0.6b
```
**Error: "Dimension mismatch"**
- LanceDB dimension is fixed per table after creation
- Use different `db_path` for different dimensions
- Delete old database to change dimensions
**Error: "Connection refused"**
```bash
# Linux
systemctl status ollama
systemctl start ollama
# macOS
brew services info ollama
brew services start ollama
# Or run manually
ollama serve
```
**Performance tuning:**
```bash
# Larger batches (requires more VRAM)
MLX_MAX_BATCH_CHARS=64000
MLX_MAX_BATCH_ITEMS=32
```
---
## Quick Start
### Installation
**Quick install (recommended):**
```bash
Prebuilt GitHub Release bundles are the canonical install path and avoid compiling
the full LanceDB-heavy Rust dependency graph on the target machine.
**From source (development only):**
```bash
cargo install --path .
```
### Running
```bash
# Canonical MCP surface
rust-memex serve
# With security enabled
rust-memex serve --security-enabled
# With HTTP/SSE server for multi-agent access
rust-memex serve --http-port 8997
# HTTP-only daemon mode (no MCP stdio)
rust-memex serve --http-port 8997 --http-only
```
`rust-memex` always exposes one canonical MCP tool surface. To narrow runtime
access, use `--allowed-paths`, HTTP auth, or namespace security rather than a
separate "memory/full" mode switch.
## HTTP/SSE Server (Multi-Agent Access)
LanceDB uses exclusive file locks - only one process can access the database at a time.
The HTTP/SSE server solves this by providing a central access point for multiple agents.
### Architecture
```
┌─────────────────────────────────────────────────────────────┐
│ rust-memex daemon │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ MCP Server │ │ HTTP/SSE │ │
│ │ (stdio) │ │ (port 8997) │ │
│ └────────┬────────┘ └────────┬────────┘ │
│ │ │ │
│ └──────────┬───────────┘ │
│ ▼ │
│ ┌─────────────┐ │
│ │ RAGPipeline │ ← Single lock holder │
│ └──────┬──────┘ │
│ ▼ │
│ ┌─────────────┐ │
│ │ LanceDB │ │
│ └─────────────┘ │
└─────────────────────────────────────────────────────────────┘
▲ ▲
│ │
Claude Desktop HTTP Agents
(MCP stdio) (curl, fetch)
```
### HTTP Endpoints
| `/health` | GET | Health check (status, db_path, embedding_provider) |
| `/search` | POST | Search with optional `project`, `layer`, and `deep` filters (`k` alias supported); response includes collapsed `clusters` and `duplicate_count` |
| `/sse/search` | GET | SSE streaming search with optional `project`, `layer`, and `deep` filters |
| `/api/context-pack` | POST | Build a markdown context pack from a query or explicit chunk IDs, with grouped evidence and rebuilt indexed source chunks |
| `/upsert` | POST | Add/update document |
| `/index` | POST | Full pipeline indexing with onion slices |
| `/expand/{ns}/{id}` | GET | Expand onion slice (get children) |
| `/parent/{ns}/{id}` | GET | Drill up to parent slice |
| `/get/{ns}/{id}` | GET | Get document by ID |
| `/delete/{ns}/{id}` | POST | Delete document |
| `/ns/{namespace}` | DELETE | Purge entire namespace |
**Diagnostic & lifecycle endpoints** (require `Bearer` auth_token):
| `/api/audit` | GET | Per-namespace quality audit (chunk completeness, hash coverage, score) |
| `/api/stats` / `/api/stats/{ns}` | GET | Database / namespace statistics |
| `/api/timeline` | GET | Indexing timeline aggregates |
| `/api/purge-quality` | POST | Purge low-quality chunks under a threshold (gated by approval key) |
| `/api/dedup` | POST | Run post-index deduplication (`group-by`, `keep`, `dry_run` body fields) |
| `/api/backfill-hashes` | POST | Spec P0 backfill: populate per-chunk `content_hash` + `source_hash` for pre-v4 namespaces |
### MCP-over-SSE Endpoints (Claude Code compatibility)
| `/sse/` | GET | SSE stream - sends `endpoint` event with messages URL |
| `/messages/` | POST | JSON-RPC messages with `?session_id=xxx` |
Configure in `~/.claude.json`:
```json
{
"mcpServers": {
"rust-memex": {
"type": "sse",
"url": "http://localhost:8997/sse/"
}
}
}
```
### Usage Examples
```bash
# Open the local dashboard
rust-memex dashboard --db-path ~/.ai-memories/lancedb
# Start daemon
rust-memex sse --db-path ~/.ai-memories/lancedb &
# Health check
curl http://localhost:8997/health
# Store document
curl -X POST http://localhost:8997/upsert \
-H "Content-Type: application/json" \
-d '{"namespace": "agent1", "id": "mem1", "content": "Important context..."}'
# Search
curl -X POST http://localhost:8997/search \
-H "Content-Type: application/json" \
-d '{"query": "context", "namespace": "agent1", "k": 10, "project": "Vista", "deep": true}'
# SSE streaming search
curl -N "http://localhost:8997/sse/search?query=context&namespace=agent1&limit=5&project=Vista&layer=1"
```
### Multi-Host Database Paths
For setups with multiple machines (e.g., dragon, mgbook16), use per-host database paths:
```bash
# Per-host paths (each machine gets own database)
rust-memex serve --db-path ~/.ai-memories/lancedb.$(hostname -s)
# Or use the wizard for machine-agnostic configuration
rust-memex wizard
```
The TUI wizard auto-detects hostname and offers:
- **Shared mode**: `~/.ai-memories/lancedb` (same path everywhere)
- **Per-host mode**: `~/.ai-memories/lancedb.dragon`, `~/.ai-memories/lancedb.mgbook16`, etc.
### Configuration (TOML)
```toml
# ~/.rmcp-servers/rust-memex/config.toml
db_path = "~/.rmcp-servers/rust-memex/lancedb"
cache_mb = 4096
log_level = "info"
# Whitelist of allowed paths
allowed_paths = [
"~",
"/Volumes/ExternalDrive/data"
]
# Security
security_enabled = true
token_store_path = "~/.rmcp-servers/rust-memex/tokens.json"
# Optional: dashboard-only OIDC for browser users.
# API / SSE / MCP still stay Bearer-authenticated via auth_token.
auth_token = "replace-me"
[dashboard_oidc]
issuer_url = "https://issuer.example"
client_id = "rust-memex-dashboard"
client_secret = "optional-confidential-client-secret"
public_base_url = "https://memex.example.com"
scopes = ["openid", "profile", "email"]
```
## Documentation
- [01_security.md](./01_security.md) - Security system (namespace tokens)
- [02_configuration.md](./02_configuration.md) - Configuration and CLI options
## Onion Slice Architecture
Instead of traditional flat chunking, rust-memex offers hierarchical "onion slices":
```
┌─────────────────────────────────────────┐
│ OUTER (~100 chars) │ ← Minimum context, maximum navigation
│ Keywords + ultra-compression │
├─────────────────────────────────────────┤
│ MIDDLE (~300 chars) │ ← Key sentences + context
├─────────────────────────────────────────┤
│ INNER (~600 chars) │ ← Expanded content
├─────────────────────────────────────────┤
│ CORE (full text) │ ← Complete document
└─────────────────────────────────────────┘
```
**Philosophy:** "Minimum info → Maximum navigation paths"
### QueryRouter & Auto-Route
Intelligent query intent detection for automatic search mode selection:
```bash
# Auto-detect query intent and select optimal mode
rust-memex search -n memories -q "when did we buy dragon" --auto-route
# Output: Query intent: temporal (confidence: 0.70)
# Selects: hybrid mode with date boosting
# Structural queries suggest loctree
rust-memex search -n code -q "who imports main.rs" --auto-route
# Output: Query intent: structural (confidence: 0.80)
# Consider: loctree query --kind who-imports --target main.rs
# Deep exploration with all onion layers
rust-memex dive -n memories -q "dragon" --verbose
```
**Intent Types:**
| Temporal | when, date, yesterday, ago, 2024 | Hybrid (date boost) |
| Structural | import, depends, module, who uses | BM25 + loctree suggestion |
| Semantic | similar, related, explain | Vector |
| Exact | "quoted strings" | BM25 |
| Hybrid | (default) | Vector + BM25 fusion |
### CLI Commands
```bash
# Index with onion slicing (default)
rust-memex index -n memories /path/to/data/ --slice-mode onion
# Index with progress bar and ETA
rust-memex index -n memories /path/to/data/ --progress
# Index with flat chunking (backward compatible)
rust-memex index -n memories /path/to/data/ --slice-mode flat
# Search in namespace
rust-memex search -n memories -q "best moments" --limit 10
# Search only in specific layer
rust-memex search -n memories -q "query" --layer outer
# Drill down in hierarchy (expand children)
rust-memex expand -n memories -i "slice_id_here"
# Get chunk by ID
rust-memex get -n memories -i "chunk_abc123"
# RAG search (cross-namespace)
rust-memex rag-search -q "search term" --limit 5
# List namespaces with stats
rust-memex namespaces --stats
# Export namespace to JSON
rust-memex export -n memories -o backup.json --include-embeddings
```
### Preprocessing (Noise Filtering)
Automatic removal of ~36-40% noise from conversation exports:
- MCP tool artifacts (`<function_calls>`, `<invoke>`, etc.)
- CLI output (git status, cargo build, npm install)
- Metadata (UUIDs, timestamps → placeholders)
- Empty/boilerplate content
```bash
# Index with preprocessing
rust-memex index -n memories /path/to/export.json --preprocess
```
### Deduplication & Hash Hygiene
Two layers of dedup work together:
**1. Pre-index source dedup** (during `index`)
The pipeline computes `sha256(file_text)` and skips files whose `source_hash`
already exists in the namespace (with a fallback to `content_hash` for pre-v4
namespaces). The skip line is logged at `info!` so it shows up in the default
operator run log:
```
Skip duplicate source: /path/to/file.md (source_hash 8ee43c1e7393b432)
```
```bash
# Dedup enabled (default)
rust-memex index -n memories /path/to/data/
# Disable dedup for this run
rust-memex index -n memories /path/to/data/ --no-dedup
# Spec P4 escape hatch: force re-index a known-duplicate source
rust-memex index -n memories /path/to/data/ --allow-duplicates
```
**2. Post-index dedup CLI** (standalone command)
```bash
# Default grouping: source-hash + layer (preserves onion structure,
# removes only true source repeats while keeping outer/middle/inner/core)
rust-memex dedup -n kb:transcripts --dry-run
# Collapse all layers per source (legacy aggressive grouping)
rust-memex dedup -n kb:transcripts --group-by source-hash
# Per-chunk content_hash grouping (legacy pre-v4 behavior)
rust-memex dedup -n kb:transcripts --group-by content-hash
# Cross-namespace dedup pool
rust-memex dedup --cross-namespace --dry-run false
# Keep newest duplicates instead of oldest
rust-memex dedup -n memories --keep newest
```
**3. Hash backfill (spec P0)** — fills `content_hash` (per-chunk) and
`source_hash` (per-source) for namespaces indexed before v4. Without backfill,
`dedup` reports "Without hash: N (cannot deduplicate)" and is blind to legacy
chunks:
```bash
# Dry-run backfill across all namespaces
rust-memex backfill-hashes
# Backfill one namespace, then commit
rust-memex backfill-hashes -n kb:transcripts --dry-run false
# Machine-readable JSON for scripts / CI
rust-memex backfill-hashes --json
```
**Output with statistics:**
```
Indexing complete:
New chunks: 234
Files indexed: 67
Skipped (duplicate): 33
Deduplication: enabled
```
### LLM-Synthesized Outer Layer (Spec P3)
The default outer layer is a TF-based keyword extract (`--outer-synthesis
keyword`). For transcript-heavy namespaces where keyword splat is noise (CLI
animation gerunds, structural markdown, file-path tokens), the outer layer can
be replaced with a 1-3 sentence summary from a local Ollama model:
```bash
# Wire the outer layer through Ollama (requires --pipeline mode)
rust-memex index -n kb:transcripts /path/to/transcripts/ \
--slice-mode onion \
--pipeline \
--outer-synthesis llm \
--ollama-model qwen2.5:3b \
--ollama-endpoint http://localhost:11434
```
Failure modes (network, non-2xx, malformed JSON, empty completion) silently
fall back to the keyword outer so the pipeline never stalls. Reachable only
through `--pipeline` — passing `--outer-synthesis llm` without `--pipeline` is
rejected up-front by clap.
## Code Structure
```
rust-memex/
├── src/
│ ├── lib.rs # Public API & ServerConfig
│ ├── bin/
│ │ └── rust-memex.rs # CLI binary (serve, index, search, get, expand, etc.)
│ ├── handlers/
│ │ └── mod.rs # MCP request handlers
│ ├── security/
│ │ └── mod.rs # Namespace access control
│ ├── rag/
│ │ └── mod.rs # RAG pipeline + OnionSlice architecture
│ ├── preprocessing/
│ │ └── mod.rs # Noise filtering for conversation exports
│ ├── storage/
│ │ └── mod.rs # LanceDB + Tantivy (schema v4: source_hash + per-chunk content_hash)
│ ├── embeddings/
│ │ └── mod.rs # MLX/FastEmbed bridge
│ └── tui/
│ └── mod.rs # Configuration wizard
└── Cargo.toml
```
## Claude/MCP Integration
Add to `~/.claude.json`:
```json
{
"mcpServers": {
"rust-memex": {
"command": "rust-memex",
"args": ["serve", "--security-enabled"]
}
}
}
```
---
Vibecrafted with AI Agents by Loctree (c)2025 The LibraxisAI Team
Co-Authored-By: [Maciej](void@div0.space) & [Klaudiusz](the1st@whoai.am)