# databases/ — Unified Database Layer
Backend-agnostic database abstraction for the Brainwires storage system.
## Bring Your Own Database
The two core traits — `StorageBackend` and `VectorDatabase` — are always available with no feature flags required. Implement one or both on your own struct to plug any database into the framework.
### StorageBackend (CRUD + vector search for domain stores)
```rust
use anyhow::Result;
use async_trait::async_trait;
use brainwires_storage::databases::{
StorageBackend, FieldDef, Filter, Record, ScoredRecord,
};
pub struct MyDatabase {
// your connection pool, client, etc.
}
#[async_trait]
impl StorageBackend for MyDatabase {
async fn ensure_table(&self, table_name: &str, schema: &[FieldDef]) -> Result<()> {
// Create the table if it doesn't exist. Must be idempotent.
todo!()
}
async fn insert(&self, table_name: &str, records: Vec<Record>) -> Result<()> {
// Insert rows. Each Record is Vec<(column_name, FieldValue)>.
todo!()
}
async fn query(
&self,
table_name: &str,
filter: Option<&Filter>,
limit: Option<usize>,
) -> Result<Vec<Record>> {
// Return matching rows. None filter = return all (up to limit).
todo!()
}
async fn delete(&self, table_name: &str, filter: &Filter) -> Result<()> {
// Delete matching rows.
todo!()
}
// count() has a default impl that calls query().len() — override for efficiency.
async fn vector_search(
&self,
table_name: &str,
vector_column: &str,
vector: Vec<f32>,
limit: usize,
filter: Option<&Filter>,
) -> Result<Vec<ScoredRecord>> {
// Return rows ranked by cosine/L2 similarity to `vector`.
// If your DB has no native vector search, compute client-side.
todo!()
}
}
```
### VectorDatabase (RAG embedding storage + hybrid search)
```rust
use anyhow::Result;
use async_trait::async_trait;
use brainwires_storage::databases::{
VectorDatabase, ChunkMetadata, DatabaseStats, SearchResult,
};
#[async_trait]
impl VectorDatabase for MyDatabase {
async fn initialize(&self, dimension: usize) -> Result<()> {
// Create collections/tables for the given embedding dimension (e.g. 384).
todo!()
}
async fn store_embeddings(
&self,
embeddings: Vec<Vec<f32>>,
metadata: Vec<ChunkMetadata>,
contents: Vec<String>,
root_path: &str,
) -> Result<usize> {
// Store embedding vectors with their source metadata.
// root_path scopes embeddings to a project directory.
// Return the number of embeddings stored.
todo!()
}
async fn search(
&self,
query_vector: Vec<f32>,
query_text: &str,
limit: usize,
min_score: f32,
project: Option<String>,
root_path: Option<String>,
hybrid: bool,
) -> Result<Vec<SearchResult>> {
// Vector similarity search. If hybrid=true, combine with BM25 keyword
// scoring (use bm25_helpers module or your DB's native full-text search).
todo!()
}
async fn search_filtered(
&self,
query_vector: Vec<f32>,
query_text: &str,
limit: usize,
min_score: f32,
project: Option<String>,
root_path: Option<String>,
hybrid: bool,
file_extensions: Vec<String>,
languages: Vec<String>,
path_patterns: Vec<String>,
) -> Result<Vec<SearchResult>> {
// Same as search() but with additional metadata filters.
todo!()
}
async fn delete_by_file(&self, file_path: &str) -> Result<usize> {
// Delete all embeddings for a given source file. Return count deleted.
todo!()
}
async fn clear(&self) -> Result<()> {
// Drop all embeddings.
todo!()
}
async fn get_statistics(&self) -> Result<DatabaseStats> {
// Return collection/table stats (counts, sizes, etc.)
todo!()
}
async fn flush(&self) -> Result<()> {
// Persist any buffered writes. No-op if your backend auto-flushes.
Ok(())
}
async fn count_by_root_path(&self, root_path: &str) -> Result<usize> {
// Count embeddings scoped to a project root.
todo!()
}
async fn get_indexed_files(&self, root_path: &str) -> Result<Vec<String>> {
// Return unique file paths indexed under this root.
todo!()
}
// search_with_embeddings() has a default impl that delegates to search()
// and returns empty embedding vectors. Override if your DB can return
// stored vectors (used by the spectral diversity reranker).
}
```
### Wiring it up
Wrap your implementation in `Arc` and pass it wherever the framework expects a backend:
```rust
use std::sync::Arc;
let db = Arc::new(MyDatabase::new(/* ... */).await?);
// Domain stores (messages, conversations, tasks, etc.)
let messages = MessageStore::new(db.clone(), embeddings.clone());
let conversations = ConversationStore::new(db.clone());
// RAG subsystem
let rag = RagClient::with_vector_db(db.clone());
// Cognition layer
let brain = BrainClient::with_backend(db.clone());
```
If your struct implements both traits, everything shares one connection — no duplication.
## Type Reference
All types are in `types.rs` and re-exported from `databases::*`:
| `FieldDef` | Column definition (name, type, nullable) |
| `FieldType` | Column types: `Utf8`, `Int32`, `Int64`, `UInt32`, `UInt64`, `Float32`, `Float64`, `Boolean`, `Vector(dim)` |
| `FieldValue` | Typed nullable column value (with accessor methods like `.as_str()`, `.as_i64()`, `.as_vector()`) |
| `Record` | A row: `Vec<(String, FieldValue)>` |
| `ScoredRecord` | A row + similarity score (from vector search) |
| `Filter` | Query filter: `Eq`, `Ne`, `Lt`, `Gt`, `In`, `And`, `Or`, `NotNull`, `IsNull`, `Raw(String)` |
| `ChunkMetadata` | Source metadata for RAG embeddings (file path, line range, language) |
| `SearchResult` | RAG search hit (content, metadata, score) |
| `DatabaseStats` | Collection statistics (counts, sizes) |
`Filter::Raw(String)` is an escape hatch for backend-specific filter expressions that don't map to the structured variants.
## Built-in Backends
| LanceDB | `LanceDatabase` | YES | YES | `lance-backend` (default) |
| PostgreSQL + pgvector | `PostgresDatabase` | YES | YES | `postgres-backend` |
| MySQL / MariaDB | `MySqlDatabase` | YES | NO | `mysql-backend` |
| SurrealDB | `SurrealDatabase` | YES | YES | `surrealdb-backend` |
| Qdrant | `QdrantDatabase` | NO | YES | `qdrant-backend` |
| Pinecone | `PineconeDatabase` | NO | YES | `pinecone-backend` |
| Milvus | `MilvusDatabase` | NO | YES | `milvus-backend` |
| Weaviate | `WeaviateDatabase` | NO | YES | `weaviate-backend` |
| NornicDB | `NornicDatabase` | NO | YES | `nornicdb-backend` |
## Module Layout
```
databases/
traits.rs -- StorageBackend + VectorDatabase trait definitions
types.rs -- Record, FieldDef, FieldValue, Filter, ScoredRecord
capabilities.rs -- BackendCapabilities runtime discovery
bm25_helpers.rs -- shared BM25 scoring for client-side keyword search
sql/ -- shared SQL dialect layer (SqlDialect trait)
postgres.rs -- PostgreSQL dialect
mysql.rs -- MySQL dialect
surrealdb.rs -- SurrealDB dialect
lance/ -- LanceDB backend (default)
postgres/ -- PostgreSQL + pgvector backend
mysql/ -- MySQL / MariaDB backend
surrealdb/ -- SurrealDB backend
qdrant/ -- Qdrant backend
pinecone/ -- Pinecone backend
milvus/ -- Milvus backend
weaviate/ -- Weaviate backend
nornicdb/ -- NornicDB backend (multi-transport: REST/Bolt/gRPC)
```