---
title: Architecture
description: How basemind is built — from scanning to serving to indexing.
---
import { Aside, FileTree } from '@astrojs/starlight/components';
basemind is a single Rust crate with one binary (`basemind`) serving two roles: **`basemind scan`** indexes a workspace, and **`basemind serve`** runs the MCP stdio server.
## High-Level Pipeline
```
Your project (code + documents + git history)
↓
basemind scan
├─ Tree-sitter parser pool (300+ languages)
├─ Extract L1 (outlines: symbols, signatures, imports)
├─ Extract L2 (calls: callee, byte offset, line/col) [if eager_l2=true]
├─ Extract L3 (structural hash of symbol bodies)
└─ Write to local cache
├─ Content-addressed msgpack blobs (.basemind/blobs/)
└─ Fjall LSM index (.basemind/views/<view>/index.fjall/)
↓
basemind serve
├─ Load index + blobs into memory
├─ Answer MCP tool requests instantly
└─ Watch filesystem for changes (auto-rescan)
```
## Storage: Blobs + Index
basemind persists two complementary forms of your project:
### Content-Addressed Msgpack Blobs
Location: `.basemind/blobs/<hash>.{l1,l2,l3}.msgpack`
Each extraction tier is a separate msgpack blob, keyed by content hash. This enables deduplication across files with identical bodies.
- **L1** (always): outlines with symbols, signatures, imports, doc comments
- **L2** (if eager_l2=true): call sites (callee name, byte offset, line/column)
- **L3**: structural hash of each symbol's body (used by `symbol_history`)
Blobs are immutable and content-addressed, so they can be safely shared across multiple views (scans) and repos.
### Fjall LSM Inverted Index
Location: `.basemind/views/<view>/index.fjall/`
A disk-backed, append-only LSM tree (Fjall) provides lightning-fast lookups:
| Keyspace | Purpose |
|---|---|
| `meta` | Schema version and constants |
| `symbols_by_path` | Per-file symbol lookup |
| `symbols_by_name` | Name-prefix range scans (drives `search_symbols`) |
| `calls_by_path` | Per-file call lookup |
| `calls_by_callee` | Callee-prefix range scans (drives `find_references`) |
| `imports_by_path` | Per-file import lookup |
| `imports_by_module` | Module-prefix range scans (drives `dependents`) |
| `implementations_by_path` | Per-file implementation lookup |
| `implementations_by_trait` | Trait-prefix range scans (drives `find_implementations`) |
| `embeddings` | Reserved for future in-Fjall vector index |
| `memory_by_key` | Agent memory namespaced by (scope, visibility, owner) |
All composite keys are **length-prefixed** (`u16:len ‖ bytes`), ensuring a prefix like `Foo` never spills into `Foobar`.
Schema version is stamped in `meta`. On version mismatch, the next `basemind scan` auto-wipes and rebuilds.
### LanceDB Vector Store
Location: `.basemind/lancedb/`
Powers semantic search (code search, document search, memory search). Vectors are indexed for fast KNN queries; on-disk format is LanceDB's columnar structure.
## Scan Pipeline
`basemind scan` and `basemind watch` (incremental) share the same per-file pipeline:
```
Walker (gitignore-aware) → filter by glob + size cap
↓
rayon par_iter (parallel across all CPU cores)
↓
process_file(rel_path, file_contents):
1. lang::detect() ← tree-sitter language pack
2. Extract L1 (outline) ← tree-sitter queries
3. Extract L2 (calls) ← if eager_l2=true
4. Write L1 to blob store
5. Write L2 to blob store ← if eager_l2=true
6. Index lookup:
a. Read existing entries (if re-scan)
b. Stage all deletes + inserts in one atomic Fjall batch
7. Per-file commit ← Fjall handles cross-thread locking
↓
collect FileResult { rel, l1_hash, l2_hash?, … }
↓
apply_outcomes:
· Write Index meta
· Remove stale files via IndexWriter::remove_file
```
### Key Invariants
**Per-file commit** — Every file commits its Fjall batch before returning. Fjall handles cross-thread locking; the scanner does not.
**Atomic upsert** — `IndexWriter::upsert_file` is read-before-write: it reads existing entries first to derive secondary-index keys for deletion, then stages all deletes + inserts in one batch. This prevents torn state on re-scan.
**Eager L2 cost** — Eager L2 (call-site extraction) is on by default and adds to scan time; the 81k-file TypeScript scan lands around 18s. Set `eager_l2 = false` to skip it for a faster scan, trading away reference search.
**`scan_paths` removal mirror** — When a file disappears between scans, `scan_paths` calls `IndexWriter::remove_file` so secondary indexes don't leak stale entries.
## Extraction Tiers
### L1: Outlines (Always)
Per-file symbol table with signatures, line/column locations, import statements, and doc comments. Generated by tree-sitter queries (hand-written overrides at `src/queries/<language>.scm`, falling back to `tree-sitter-language-pack`'s `tags.scm`).
Used by: `outline`, `search_symbols`, `goto_definition`, call-graph traversal.
### L2: Call Sites (Optional, Default On)
Every call site in a file: callee name, byte offset, line/column. Extracted only if `eager_l2 = true`.
Used by: `find_references`, `find_callers`, `call_graph`.
Disabling it speeds up the scan but makes reference search unavailable.
### L3: Structural Hashes
Cryptographic hash of each symbol's body. Used to detect when a symbol's implementation changed across commits.
Used by: `symbol_history`.
### Document Extraction (xberg)
When `[documents] enabled = true`, `scanner_docs.rs` runs a separate pass:
- Reads PDFs, Office files, HTML, images, email
- Extracts text (with OCR for images)
- Chunks and embeds via LanceDB
- Indexed by `search_documents`
## Language Support
basemind uses the **tree-sitter language pack** (TSLP) to detect and parse 300+ languages. Language detection is dynamic:
```
File extension → tree_sitter_language_pack::detect_language
↓
LangId = &'static str (the pack name, e.g. "rust", "python", "typescript")
↓
Parser from pool (lazy init, reused across files)
↓
Query (hand-written override OR TSLP tags.scm fallback)
↓
Extract symbols + calls
```
All languages with a `tags.scm` query in TSLP (~100) work out of the box. For richer extraction, add a `.scm` override at `src/queries/<pack-name>.scm`.
## Git Layer
`basemind git` commands are powered by `gix` (a pure-Rust Git implementation) with a **git-history index** for microsecond latency:
### Git-History Index
Location: `.basemind/git-cache/`
Two-tier caching:
- **In-process LRU** (1024 entries per category by default; tune via `basemind serve --git-cache-mem`)
- **Disk store** (SHA-keyed):
- `commit_files/<sha>.msgpack` — file list for a commit (immutable, never stales)
- `log/<head_sha>__<scope>.msgpack` — commit log (keyed by HEAD, rolls off naturally)
- `blame/<sha>__<path_hash>.msgpack` — blame data (immutable, never stales)
History queries (`commits_touching`, `recent_changes`, `blame_*`) are posting-list lookups over this index, answering in **tens of microseconds** flat.
The index is a pure accelerator: it's used only when fresh (`last_indexed_head == HEAD`) and otherwise walks history directly, so it can never serve stale results. It rebuilds automatically on history rewrite (filter-repo, rebase, force-push).
## Schema Versioning
Two persistent schemas, both tracking `RELEASE_MINOR` from `src/version.rs`:
- **`INDEX_SCHEMA_VER`** in `src/index/mod.rs` — Fjall keyspace format
- **`SCHEMA_VER`** in `src/extract/mod.rs` — msgpack blob format
### Versioning Rules
**Patch release** (`0.1.0` → `0.1.1`): `RELEASE_MINOR` unchanged. Blobs + index must stay compatible — no format changes.
**Minor release** (`0.1.x` → `0.2.0`): `RELEASE_MINOR` increments. On next `basemind scan`, `.basemind/` is auto-wiped and rebuilt from source. Users are notified in release notes.
**Wipe-on-mismatch** is the explicit migration story. No manual steps — `basemind scan` handles it.
## Directory Layout
```
.basemind/
├── basemind.toml # Configuration
├── agent-id # Persistent agent identity
├── blobs/
│ ├── abc123….l1.msgpack # Outline: symbols, signatures, imports
│ ├── abc123….l2.msgpack # Calls (if eager_l2=true)
│ └── abc123….l3.msgpack # Structural hashes
├── views/
│ └── <view>/
│ └── index.fjall/ # Fjall LSM index
├── git-cache/
│ ├── commit_files/ # Cached file lists per commit
│ ├── log/ # Cached logs
│ └── blame/ # Cached blame data
└── lancedb/ # Vector store (documents, memory, code search)
```
## MCP Surface
`basemind serve` exposes a stdio MCP server. All paths are byte-precise repo-relative (`RelPath`).
Tool conventions:
- **Paths**: always `RelPath` (repo-relative), never arbitrary strings
- **Lists**: capped at 1000 items (default 100); responses include `any_truncated` flag and cursor for pagination
- **Index scans**: internally use `scan_cap = limit * 8` to bound work on common names
- **Descriptions**: state matching semantics (substring vs. prefix, scope-aware vs. name-only)
- **Stability**: new fields are additive with `#[serde(default)]`
Tool bodies live in `src/mcp/helpers*.rs` (sliced by area: `helpers_documents.rs`, `helpers_calls.rs`, `helpers_graph.rs`, etc.); `src/mcp/tools*.rs` contain only thin `#[tool]` shims.
## Agent Comms & Shared Memory
A separate daemon process (singleton per user) owns a second Fjall store for agent communication:
- **Rooms**: per-repo, per-workspace, or global scope; agents auto-join based on scope chain
- **Messages**: front-matter stored in `messages_by_room`, bodies in `message_body` (fetched on demand for efficiency)
- **Memory**: namespaced by `(scope, visibility, owner)` — group memory is shared, individual memory is private to one agent
- **Vectors**: LanceDB stores embeddings for memory + document search
Why a separate daemon? Each `basemind serve` takes an exclusive flock on `.basemind/`, so comms can't live there. The daemon is user-global and enforces singleton via socket-bind-as-lock.
<Aside>
The architecture trades off simplicity for performance: content-addressed blobs enable deduplication, the inverted index gives instant lookups, and the git-history index makes history queries microsecond-fast. At the same time, incremental scanning and a split memory substrate support multi-agent coordination without blocking.
</Aside>