claw-vector

claw-vector is the semantic memory engine behind ClawDB. It combines SQLite-backed metadata and FTS5 keyword search, memory-mapped vector persistence, adaptive Flat and HNSW indexing, hybrid retrieval, metadata filtering, reranking, and a Python embedding microservice exposed over gRPC and HTTP.

What It Provides

• Persistent vector storage with SQLite metadata and mmap-backed raw vectors.
• Automatic index selection: Flat for small collections, HNSW once collections grow.
• ANN, filtered, and hybrid vector plus keyword retrieval.
• Search responses that include both ranked hits and execution metrics.
• A Python embedding service built on sentence-transformers with Prometheus metrics and health probes.
• Rust library and gRPC server entrypoints for application integration.

Crate At A Glance

• Crate name: claw-vector
• Current crate version: 0.1.1
• Rust edition: 2021
• MSRV target: 1.75
• Primary entrypoint: VectorEngine
• Binary entrypoint: claw-vector-server

Install from crates.io:

[dependencies]
claw-vector = "0.1.1"

Or add from CLI:

cargo add claw-vector

Architecture

						   +----------------------------------+
						   | Python embedding service         |
Text ingest and queries -->| sentence-transformers / ONNX     |
						   | gRPC :50051  HTTP :8080          |
						   +----------------+-----------------+
											|
											v
+------------------------+     +------------------------------+
| VectorEngine           |---->| CollectionManager            |
| Rust API and gRPC      |     | lifecycle, persistence,      |
| ANN and hybrid search  |     | index selection, restore     |
+-----------+------------+     +-----------+------------------+
			|                                  |
			v                                  v
 +--------------------------+      +--------------------------+
 | ANN and rerank pipeline  |      | SQLite metadata store    |
 | Flat or HNSW             |      | collections, records,    |
 | filters, hybrid fusion   |      | UUID mapping, FTS5       |
 +-------------+------------+      +-------------+------------+
			   |                                   |
			   v                                   v
	  +------------------+               +---------------------+
	  | mmap vector file |               | persisted indexes   |
	  | raw f32 payloads |               | Flat and HNSW state |
	  +------------------+               +---------------------+

Storage Model

Collection definitions, text payloads, metadata, UUID mappings, and keyword-search state live in SQLite. The database is opened in WAL mode and schema migrations are applied automatically on startup.

Raw vectors are stored separately in fixed-slot memory-mapped files. That keeps metadata queries cheap while letting the engine hydrate vectors only when a request actually asks for them or when reranking needs them.

Each collection starts on a Flat index and automatically switches to HNSW around 1,000 vectors. The collection manager persists the chosen index type so reopen and restore follow the same search path that was active before shutdown.

Search Model

The Rust API returns a SearchResponse:

pub struct SearchResponse {
	pub results: Vec<SearchResult>,
	pub metrics: SearchMetrics,
}

SearchMetrics includes query dimensionality, candidate counts, post-filter counts, and end-to-end latency in microseconds.

Metadata filters support:

• eq
• gt
• lt
• contains
• in
• exists
• and
• or
• not

Hybrid retrieval combines ANN candidates with SQLite FTS5 keyword hits. The alpha field on HybridQuery controls the blend between vector similarity and keyword relevance, where 1.0 is vector-only and 0.0 is keyword-only.

Available rerankers:

• diversity for MMR-style result diversification
• recency for boosting newer records
• composite for chaining reranking passes

Public Rust API Overview

Core types you will typically interact with:

• VectorEngine for collection lifecycle and query execution.
• VectorConfig for runtime configuration and tuning.
• SearchQuery and HybridQuery for ANN and vector+keyword retrieval.
• MetadataFilter for structured filtering on JSON metadata.
• BatchSearchQuery and BatchUpsertResult for bulk operations.

Typical request flow in applications:

1. Build VectorConfig and initialize VectorEngine.
2. Create or restore a collection.
3. Upsert records by text (service embeddings) or raw vectors.
4. Query using ANN or hybrid search.
5. Inspect SearchMetrics for latency and candidate diagnostics.
6. Close the engine to flush indexes and state.

Quick Start

1. Start the embedding service

With Docker Compose:

docker compose up --build claw-vector-svc

Or run it locally:

cd python
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,inference]"
claw-vector-svc

The service exposes gRPC on 127.0.0.1:50051 and HTTP on 127.0.0.1:8080 by default.

2. Use the Rust engine

use claw_vector::{
	DistanceMetric, MetadataFilter, SearchQuery, VectorConfig, VectorEngine,
};
use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
	let config = VectorConfig::builder()
		.db_path("data/claw_vector.db")
		.index_dir("data/claw_vector_indices")
		.embedding_service_url("http://127.0.0.1:50051")
		.build()?;

	let engine = VectorEngine::new(config).await?;

	engine
		.create_collection("docs", 384, DistanceMetric::Cosine)
		.await?;

	engine
		.upsert(
			"docs",
			"claw-vector persists vector metadata in SQLite",
			json!({"tenant": "acme", "topic": "storage", "priority": 3}),
		)
		.await?;

	let response = engine
		.search(SearchQuery {
			collection: "docs".into(),
			vector: vec![0.12; 384],
			top_k: 5,
			filter: Some(MetadataFilter::Eq {
				key: "tenant".into(),
				value: json!("acme"),
			}),
			include_vectors: false,
			include_metadata: true,
			ef_search: Some(64),
			reranker: None,
		})
		.await?;

	println!("results={}", response.results.len());
	println!("latency_us={}", response.metrics.latency_us);

	engine.close().await?;
	Ok(())
}

3. Optional: run the Rust gRPC server

cargo run --bin claw-vector-server

The server listens on 0.0.0.0:50051 by default. Override that with CLAW_GRPC_ADDR.

Embedding Service

The Python service loads the embedding model during FastAPI lifespan startup, warms it up, then starts the async gRPC server used by the Rust engine.

HTTP endpoints:

• GET /health
• GET /ready
• POST /embed
• POST /batch-embed
• GET /model-info
• GET /metrics

Authentication:

• HTTP requests accept X-Claw-Api-Key.
• gRPC requests accept x-claw-api-key (Rust server) or authorization: Bearer <key> (Python embedding service).
• When keys are configured, unauthorized requests return 401 (HTTP) or Unauthenticated (gRPC).

Example request:

curl http://127.0.0.1:8080/embed \
  -H 'content-type: application/json' \
  -d '{"texts":["semantic memory for clawdb"],"normalize":true}'

Example response shape:

{
  "vectors": [
	{
	  "values": [0.01, -0.03, 0.42],
	  "dimensions": 384
	}
  ],
  "model_name": "sentence-transformers/all-MiniLM-L6-v2",
  "latency_ms": 12
}

Configuration Reference

Rust engine

VectorConfig::builder() controls all runtime settings. The path and embedding endpoint settings can also be supplied through environment variables.

Python embedding service

Testing And Validation

Rust:

cargo test --lib --tests
cargo check --benches

Python:

cd python
pytest tests/

Additional checks commonly used in CI:

cargo clippy --all-targets --all-features -- -D warnings
cargo fmt --all -- --check
cargo bench --bench vector_bench --no-run
ruff check .

The engine test suite covers collection lifecycle, persistence across reopen, hybrid search, metadata filters, embedding cache behavior, and Flat to HNSW migration.

Benchmarks And Performance Targets

Criterion benchmarks live in benches/vector_bench.rs and cover:

• single-vector upsert
• batch upsert of 100 records
• Flat search at 500 vectors
• HNSW search at 1,000 and 100,000 vectors
• filtered ANN search at 10,000 vectors
• embedding cache hit latency
• hybrid search at 1,000 text-backed records

Operational targets for the current implementation:

• HNSW search p99 under 10 ms at 100k vectors
• cache-hit embedding lookups under 1 us
• embedding throughput above 2,000 texts per minute for the default MiniLM model on suitable hardware

Use Criterion directly when you want full benchmark runs:

cargo bench --bench vector_bench

Development Notes

• Rust protobuf generation is handled in build.rs with vendored protoc, so local protobuf installation is not required.
• SQLite migrations are embedded and applied automatically by the Rust store layer.
• docker-compose.yml provides a ready-to-run embedding service and an optional Rust dev container profile.

Operational Guidance

• Use workspace isolation (workspace_id) in multi-tenant environments to separate data, indexes, and auth/rate-limit scopes.
• Keep vector dimensions fixed per collection; dimension drift should be handled by creating a new collection and reindexing.
• For large ingestion jobs, use batch upserts and tune ef_construction/m_connections before initial load.
• For low-latency query paths, keep include_vectors off unless callers truly need full vector payloads.
• Back up both SQLite files and index/mmap directories to preserve full recall behavior after restore.

Release Notes

Version 0.1.1 updates dependency constraints to the latest compatible direct versions, refreshes lockfile resolutions, and expands crate documentation for integration, operations, and CI validation.