rmcp-memex 0.1.9

RAG/memory MCP server with LanceDB vector storage
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130

# Schema Migration Guide

This document describes how to handle schema changes in rmcp_memex storage layers.

## Schema Versioning

The current schema version is defined in `src/storage/mod.rs`:

```rust
pub const SCHEMA_VERSION: u32 = 1;
```

Increment this version whenever you make breaking changes to:
- LanceDB table structure (columns, embeddings dimension)
- Sled key-value schema
- ChromaDocument fields

## Current Schema (v1)

### LanceDB Table: `mcp_documents`

| Column | Type | Description |
|--------|------|-------------|
| `id` | String | Unique document identifier |
| `namespace` | String | Namespace for isolation |
| `embedding` | FixedSizeList[Float32, 384] | fastembed vector (384 dims) |
| `metadata` | String (JSON) | Serialized metadata object |
| `document` | String | Original text content |

### Sled Keys

| Key Pattern | Value | Description |
|-------------|-------|-------------|
| `{namespace}:{id}` | JSON bytes | Document lookup cache |

## Migration Procedures

### Before Any Migration

1. **Backup your data**:
   ```bash
   cp -r ~/.rmcp_servers/rmcp_memex/lancedb ~/.rmcp_servers/rmcp_memex/lancedb.backup
   cp -r ~/.rmcp_servers/sled ~/.rmcp_servers/sled.backup
   ```

2. **Stop all rmcp_memex instances**

### Migration Strategies

#### Strategy A: Re-index (Recommended for small datasets)

Best for datasets < 100K documents or when embeddings model changes.

```bash
# 1. Backup
cp -r ~/.rmcp_servers/rmcp_memex/lancedb ~/.rmcp_servers/rmcp_memex/lancedb.backup

# 2. Delete old data
rm -rf ~/.rmcp_servers/rmcp_memex/lancedb

# 3. Re-index your documents
# (Use your indexing scripts or MCP tools)
```

#### Strategy B: In-place Migration (Advanced)

For large datasets where re-indexing is expensive.

```rust
// Example migration script (pseudocode)
use rmcp_memex::storage::StorageManager;

async fn migrate_v1_to_v2(storage: &StorageManager) -> Result<()> {
    // 1. Read all documents from old schema
    // 2. Transform to new schema
    // 3. Write to new table
    // 4. Swap tables atomically
    Ok(())
}
```

### Version-Specific Migrations

#### v1 → v2 (Future)

*Not yet defined. This section will be updated when v2 schema is introduced.*

## Adding a Migration

When changing the schema:

1. Increment `SCHEMA_VERSION` in `src/storage/mod.rs`
2. Document the new schema in this file
3. Add migration procedure from previous version
4. Add a test that creates old-schema data and verifies migration
5. Update CHANGELOG.md

## Testing Migrations

```bash
# Run migration tests
cargo test migration

# Manual verification
cargo run -- --db-path /tmp/test_migration_db
```

## Rollback

If migration fails:

```bash
# Restore from backup
rm -rf ~/.rmcp_servers/rmcp_memex/lancedb
cp -r ~/.rmcp_servers/rmcp_memex/lancedb.backup ~/.rmcp_servers/rmcp_memex/lancedb
```

## FAQ

**Q: Do I need to migrate if I only update rmcp_memex?**

A: Only if the SCHEMA_VERSION changes. Check the CHANGELOG for "breaking: schema change" entries.

**Q: Can I run different schema versions side by side?**

A: Not recommended. Use separate `--db-path` for each version during testing.

**Q: How do I check my current schema version?**

A: Currently there's no runtime check. The schema version is implied by the rmcp_memex version you're running. Future versions may store schema version in the database.