# Cerebro Architecture
## The Overall Data Pipeline
```mermaid
graph TD
A[Raw Document / URL / Event] --> B(Chunker)
C[OpenAI / LLM App] -->|MCP / Rust API| API(MemoryEngine)
B -->|Chunks| API
API --> D{Router & Compute Engine}
D <-->|Traits: Embedder| E1[Local Rust ML: Candle/ORT]
D <-->|Traits: Embedder| E2[Remote APIs: OpenAI/Gemini]
D -->|Semantic Vectors| F1[(PgVectorStore)]
D -->|Working State| F2[(MemoryVectorStore)]
```
## Module Structure
Cerebro is organized as a single crate with clean module boundaries:
### `models`
Core data structures: `Document`, `Chunk`, `Node`, `Metadata`. All fully serializable via Serde.
### `traits`
The universal trait system that all backends implement:
* `Chunker` — splits Documents into Chunks
* `Embedder` — converts text into vector embeddings
* `VectorStore` — persists and searches Nodes
* `KVStore` — fast key-value state for Working Memory
* `CerebroError` — unified error hierarchy
### `chunker`
* `RecursiveCharacterChunker` — character-boundary-safe text splitter.
* `HtmlSemanticChunker` — layout-aware semantic splitter.
### `compute`
Embedding providers:
- `MockEmbedder` — deterministic dummy embeddings for offline testing.
- `OpenAIEmbedder` — production OpenAI API integration.
- `LocalEmbedder` — native CPU inference using HuggingFace's `candle`.
- `AnthropicVoyageEmbedder` — Claude-aligned vectors through Voyage AI.
### `storage`
Vector store backends:
- `MemoryVectorStore` — fast in-memory concurrent storage.
- `PgVectorStore` — PostgreSQL + pgvector (Hybrid search support).
- `QdrantVectorStore` — high-volume distributed Qdrant driver.
- `MemoryKVStore` — fast key-value state for working memory.
### `engine`
The core orchestration layer:
* `MemoryEngine` — coordinates the primary ingest/query flows.
* `ConsolidationWorker` — background Tokio task that prunes memory and optimizes semantic density.
* `GraphMemoryLayer` — bridge for Neo4j/Cypher entity persistence.
### `ingest`
Specialized document extractors:
- `PdfIngestor` — extracts text from raw PDF bytes.
### `ffi`
Cross-language interface bridge:
- `python` — PyO3 bindings.
- `wasm` — Wasm-bindgen targets.
---