memory-wiki 0.2.0

# Memory Wiki MCP Server

A semantic knowledge base with graph relations, powered by local embeddings. Built with Rust for high performance and low memory usage.

[![Crates.io](https://img.shields.io/crates/v/memory-wiki.svg)](https://crates.io/crates/memory-wiki)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

## Features

### Core

- **Semantic Search**: Vector embeddings with fastembed (ONNX-based, runs locally)
- **Graph Relations**: Link entries with typed relations (references, depends_on, etc.)
- **Collections**: Organize entries into collections
- **Persistence**: All data stored locally in JSON files
- **Multi-Root Storage**: Per-workspace isolation with separate data stores

### Search

- **Hybrid Search**: Combines semantic similarity with BM25 keyword matching
- **Time-Based Search**: Natural language queries like "yesterday", "last week"
- **Reranking**: Optional cross-encoder for improved accuracy
- **HyDE**: Hypothetical Document Embeddings for better retrieval
- **Chain-of-Thought Retrieval**: Multi-step reasoning with iterative retrieval

### Advanced Retrieval (NEW!)

- **Multi-Perspective Exploration**: GraphRAG, RAPTOR-style hierarchical retrieval
  - `temporal` - How knowledge evolved over time
  - `relational` - Graph-based multi-hop connections
  - `hierarchical` - Abstract concepts to specific details
  - `entity` - Focus on mentioned entities
  - `contextual` - Group by themes/collections
  - `summary` - CAG-powered hierarchical summaries
  - `hyde` - Hypothetical Document Embeddings
  - `reason` - Chain-of-Thought retrieval
  - `reflect` - Self-reflection & memory consolidation
- **CAG Engine**: Context-Augmented Generation with cached summaries
- **Self-Reflection**: Memory consolidation and gap analysis

### Analysis

- **Entity Extraction**: Automatically detect people, technologies, concepts
- **Auto-Tagging**: Tag entries based on extracted entities
- **Duplicate Detection**: Find similar/duplicate entries
- **Entry Merging**: Consolidate duplicate information

### Ingestion

- **Document Ingestion**: Smart chunking for markdown, code, and text files
- **File Scheduler**: Track and auto-update files on changes
- **Git Metadata**: Automatically capture git branch, commit, and file context
- **Startup Indexing**: Automatically index directories via environment variables
- **Background Indexing**: Async directory scanning with progress tracking
- **LLM-Powered Indexing**: Optional LLM analysis for summaries, entities, relationships

### Web UI (NEW!)

- **Knowledge Graph Visualization**: Interactive D3.js graph with zoom/pan
- **Perspectives Panel**: Explore memories from different angles
- **Search Interface**: Semantic, hybrid, and time-based search
- **Entry Management**: Create, edit, delete entries
- **Real-time Stats**: Dashboard with knowledge base metrics

## Installation

### From crates.io

```bash
cargo install memory-wiki
```

### From source

```bash
git clone https://github.com/Algiras/memory-wiki
cd memory-wiki/memory-wiki
cargo build --release
```

### With LLM support

```bash
cargo build --release --features llm
```

## Configuration

Copy `.env.example` to `.env` and customize:

```bash
# Key settings
MEMORY_WIKI_DATA_DIR=data
MEMORY_WIKI_MODEL=bge-small-en-v1.5
MEMORY_WIKI_INDEX_DIRS=/path/to/notes,/path/to/docs
MEMORY_WIKI_ENABLE_RERANKER=true
```

See `.env.example` for all options.

## Environment Variables

### Core Configuration

| Variable | Description | Default |
|----------|-------------|---------|
| `MEMORY_WIKI_DATA_DIR` | Directory for storing data | `data` |
| `MEMORY_WIKI_PORT` | Port for the web interface | `3001` |
| `MEMORY_WIKI_MODEL` | Embedding model to use | `bge-small-en-v1.5` |
| `RUST_LOG` | Set logging level | `info` |

### Indexing & Processing

| Variable | Description | Default |
|----------|-------------|---------|
| `MEMORY_WIKI_INDEX_DIRS` | Comma-separated directories to index on startup | - |
| `MEMORY_WIKI_INDEX_PATTERNS` | File patterns to index | `*.md,*.txt,*.rs` |
| `MEMORY_WIKI_CHUNK_SIZE` | Size of text chunks for embedding | `1000` |
| `MEMORY_WIKI_CHUNK_OVERLAP` | Overlap between chunks | `200` |
| `MEMORY_WIKI_MAX_FILE_SIZE` | Max file size to process in bytes | `10MB` |

### LLM Configuration (for background indexing)

| Variable | Description | Default |
|----------|-------------|---------|
| `OPENROUTER_API_KEY` | OpenRouter API key (recommended) | - |
| `OPENROUTER_MODEL` | OpenRouter model | `anthropic/claude-3.5-haiku` |
| `OPENAI_API_KEY` | OpenAI API key | - |
| `OPENAI_BASE_URL` | Custom OpenAI-compatible endpoint | - |
| `OPENAI_MODEL` | OpenAI model | `gpt-4o-mini` |
| `ANTHROPIC_API_KEY` | Anthropic API key | - |
| `OLLAMA_HOST` | Ollama server URL | `http://localhost:11434` |
| `OLLAMA_MODEL` | Ollama model | (auto-detected) |

### Feature Flags

| Variable | Description | Default |
|----------|-------------|---------|
| `MEMORY_WIKI_ENABLE_RERANKER` | Enable BGE reranking for better search | `true` |
| `MEMORY_WIKI_ENABLE_SCHEDULER` | Enable background file watching | `true` |
| `MEMORY_WIKI_SIMILARITY_THRESHOLD` | Threshold for duplicate/similarity detection | `0.85` |
| `MEMORY_WIKI_COLLECTION_DEFAULT` | Default collection name | `general` |
| `MEMORY_WIKI_DISABLE_GIT_METADATA` | Disable git metadata detection | `false` |
| `MEMORY_WIKI_SAMPLING_MODEL` | Model hint for MCP sampling (e.g., `claude-3-5-sonnet`) | - |

### Multi-Root Storage

| Variable | Description | Default |
|----------|-------------|---------|
| `MEMORY_WIKI_ROOTS_ENABLED` | Enable per-workspace root isolation | `true` |
| `MEMORY_WIKI_MAX_ROOT_SIZE_MB` | Max storage size per root in MB | `100` |

## Usage

### MCP Mode (for Claude Desktop, Cursor, etc.)

```bash
./target/release/memory-wiki
```

Add to your MCP client config:

```json
{
  "mcpServers": {
    "memory-wiki": {
      "command": "/path/to/memory-wiki",
      "args": []
    }
  }
}
```

### Web UI Mode

```bash
./target/release/memory-wiki --web --port 3001
```

Then open <http://localhost:3001>

## API Overview

Memory Wiki exposes a set of **Simplified Tools** designed for easy interaction with LLMs. For detailed documentation, see [docs/api.md](docs/api.md).

### Core Tools

| Tool | Description |
|------|-------------|
| `memory` | Manage entries (store, get, update, delete, list) |
| `search` | Search knowledge (semantic, hybrid, time-based) |
| `graph` | Manage relations (link, path, related, hubs) |
| `ask` | Smart interactive assistant (remember, find, summarize, connect, review) |
| `stores` | Multi-root storage management (list, switch, global, info) |
| `perspective` | Multi-perspective exploration (temporal, relational, hierarchical, etc.) |

### File & Analysis Tools

| Tool | Description |
|------|-------------|
| `files` | Ingest and watch files/directories |
| `analyze` | Extract entities, find duplicates, auto-tag, merge |
| `collections` | Manage collections (list, create, delete) |
| `stats` | Get knowledge base statistics |
| `export` | Export data to JSON |

### Web API Endpoints

| Endpoint | Description |
|----------|-------------|
| `GET /api/entries` | List all entries |
| `POST /api/entries` | Create new entry |
| `GET /api/search?query=...` | Semantic search |
| `GET /api/graph` | Get knowledge graph |
| `GET /api/perspective/:mode` | Perspective exploration |
| `POST /api/index/start` | Start background indexing |
| `GET /api/index/status` | Get indexing progress |

## New Features

### Multi-Perspective Exploration

Explore your knowledge base from different angles using the `perspective` tool:

```json
{
  "tool": "perspective",
  "arguments": {
    "mode": "relational",
    "query": "machine learning",
    "depth": 2
  }
}
```

Available modes:
- **temporal**: Track knowledge evolution over time
- **relational**: GraphRAG-style multi-hop exploration
- **hierarchical**: RAPTOR-style abstraction levels
- **entity**: Focus on specific entities
- **contextual**: Group by themes
- **summary**: CAG-powered summaries
- **hyde**: Hypothetical Document Embeddings
- **reason**: Chain-of-Thought retrieval
- **reflect**: Self-reflection analysis

### Background Indexing with LLM

Index directories with optional LLM-powered analysis:

```bash
curl -X POST http://localhost:3001/api/index/start \
  -H "Content-Type: application/json" \
  -d '{
    "directory": "/path/to/docs",
    "patterns": ["*.md", "*.rs"],
    "enable_llm": true,
    "build_graph": true
  }'
```

LLM analysis extracts:
- Summaries at multiple abstraction levels
- Entities (people, organizations, concepts)
- Topics and key points
- Relationships between documents
- Auto-generated tags

### Git Metadata Integration

When storing entries from files in git repositories, Memory Wiki automatically captures:

- Branch name
- Commit hash (short and full)
- File path relative to repo root
- Repository name and remote URL
- Author and commit timestamp

Disable with `MEMORY_WIKI_DISABLE_GIT_METADATA=true`.

### Multi-Root Storage

Memory Wiki can isolate storage per workspace root:

- Each MCP client root gets its own data store
- Use `stores` tool to list, switch, or enable global search
- Prevents cross-project data contamination

### Search Reranking

When enabled, search results are reranked using BGE cross-encoder:

- Improves relevance for both semantic and hybrid search
- Response includes `"reranked": true` indicator
- Enable with `MEMORY_WIKI_ENABLE_RERANKER=true`

## Documentation

Full documentation is available in the `docs/` directory:

- [System Architecture](docs/architecture.md)
- [Data Model](docs/data_model.md)
- [API Documentation](docs/api.md)

## Available Embedding Models

| Model | Params | Dimensions | Best For |
|-------|--------|------------|----------|
| bge-small-en-v1.5 | 33M | 384 | Fast, good quality |
| bge-base-en-v1.5 | 109M | 768 | Balanced |
| bge-large-en-v1.5 | 335M | 1024 | Best quality |
| all-minilm-l6-v2 | 22M | 384 | Very fast |
| multilingual-e5-small | 118M | 384 | Multi-language |
| multilingual-e5-base | 278M | 768 | Multi-language, larger |
| nomic-embed-text-v1 | 137M | 768 | Long context |
| paraphrase-albert | 11M | 768 | Smallest |

## Project Structure

```text
src/
├── main.rs               # Entry point
├── config.rs             # Environment configuration
├── handler_simplified.rs # MCP request handler
├── tools/                # Tool definitions
│   ├── mod.rs            # Tool registry
│   └── simplified.rs     # Simplified tool schemas
├── handlers/             # Tool handlers
│   ├── memory.rs         # Entry management
│   ├── search.rs         # Search operations
│   ├── graph.rs          # Graph operations
│   ├── perspective.rs    # Multi-perspective exploration
│   └── ...
├── embeddings.rs         # Embedding model wrapper
├── storage_embedded.rs   # Entry storage
├── graph.rs              # Graph storage
├── multi_store.rs        # Multi-root storage manager
├── git_metadata.rs       # Git metadata extraction
├── scheduler.rs          # File scheduler
├── reranker.rs           # Cross-encoder reranking
├── ingestion.rs          # Document ingestion engine
├── consolidation.rs      # Duplicate detection
├── time_parser.rs        # Natural language time parsing
├── hybrid_search.rs      # BM25 + vector search
├── entity.rs             # Entity extraction
├── cag.rs                # Context-Augmented Generation
├── advanced_retrieval.rs # HyDE, CoT, Self-Reflection
├── background_indexer.rs # Async directory indexing
├── llm_indexer.rs        # LLM-powered analysis
├── web.rs                # Web UI server
└── models.rs             # Data structures
```

## Data Storage

All data is stored in the `data/` directory:

```text
data/
├── entries.json         # Knowledge entries with embeddings
├── graph.json           # Relations between entries
├── scheduler.json       # File tracking state
├── collections/         # Collection metadata
└── roots/               # Per-workspace root stores (when multi-root enabled)
    └── <hash>/
        ├── entries.json
        └── graph.json
```

## License

MIT