Crate velesdb_core

Expand description

§velesdb-core

The embedded tri-engine of VelesDB: vector, graph and columnar metadata in one Rust database.

§Objective

Semantic retrieval usually means running a vector store, a graph database and a relational store side by side, then stitching their results together in application code — three deployments, three consistency stories, three query languages.

velesdb-core is the embedded engine that collapses those three into one process and one language. Vectors (HNSW + SIMD), typed graph edges and typed columnar metadata live in the same collection and are queried together with VelesQL. No server, no network hop, no external dependency: it is a Rust library that reads and writes a directory on your disk.

If you do not need retrieval over your own data, you do not need this crate.

§Use cases

A desktop or CLI application that must search its own documents offline, with no service to install and no data leaving the machine.
A RAG pipeline that filters candidates on structured metadata (tenant, date, status) in the same query as the vector search, instead of post-filtering results and losing recall.
A recommendation feature where “similar to this item” must be combined with “and connected to the user by at most 2 hops” — vector plus graph traversal in one statement.
An AI agent that needs durable memory (facts, events, learned procedures) with TTL and snapshots, embedded in the agent process itself.
An embedded/edge deployment where a 32x-compressed index must fit in RAM on constrained hardware.

§Prerequisites

Requirement	Minimum version	Note
Rust	1.90	Workspace MSRV, pinned in `rust-toolchain.toml`
Cargo	shipped with Rust	No other build tool required
Disk	writable directory	The `persistence` feature (on by default) memory-maps files there
Embeddings	any source	This crate does not compute embeddings — you supply the vectors
GPU	optional	Only for the `gpu` feature; falls back to SIMD when absent

§Installation

cargo add velesdb-core

For WASM or any target without a filesystem, disable the default feature:

cargo add velesdb-core --no-default-features

§First success in 60 seconds

Create a project, add the two dependencies, paste this into src/main.rs, run it.

cargo new veles-hello && cd veles-hello
cargo add velesdb-core serde_json

use serde_json::json;
use velesdb_core::{Database, DistanceMetric, Point};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // 1. Open (or create) a local database directory.
    let db = Database::open("./veles-quickstart")?;

    // 2. One collection = one vector dimension + one distance metric (both immutable).
    db.create_collection("documents", 4, DistanceMetric::Cosine)?;
    let documents = db
        .get_vector_collection("documents")
        .ok_or("collection not found")?;

    // 3. Insert points: id, vector, optional JSON payload.
    documents.upsert(vec![
        Point::new(1, vec![1.0, 0.0, 0.0, 0.0], Some(json!({"title": "rust"}))),
        Point::new(2, vec![0.0, 1.0, 0.0, 0.0], Some(json!({"title": "python"}))),
        Point::new(3, vec![0.9, 0.1, 0.0, 0.0], Some(json!({"title": "cargo"}))),
    ])?;

    // 4. flush() is the explicit durability barrier.
    documents.flush()?;

    // 5. Search: top-2 nearest neighbours of [1, 0, 0, 0].
    for hit in documents.search(&[1.0, 0.0, 0.0, 0.0], 2)? {
        let title = hit
            .point
            .payload
            .as_ref()
            .and_then(|payload| payload.get("title"))
            .and_then(serde_json::Value::as_str)
            .unwrap_or("<none>");
        println!("id={} title={title} score={:.4}", hit.point.id, hit.score);
    }

    Ok(())
}

$ cargo run
id=1 title=rust score=1.0000
id=3 title=cargo score=0.9939

Success looks exactly like that: two lines, id=1 first with a cosine score of 1.0000 (the query vector is identical to point 1), id=3 second. Point 2 is orthogonal to the query and is correctly excluded from the top 2.

Anything else is a failure — in particular, a second cargo run prints Error: CollectionExists("documents") because the collection is already persisted in ./veles-quickstart. Delete that directory to start over, or skip create_collection when the collection already exists.

§Configuration

Compile-time features (Cargo.toml):

Feature	Default	Effect
`persistence`	on	mmap storage, WAL, rayon parallelism, tokio. Turn off for WASM.
`gpu`	off	wgpu compute pipeline for batch distance kernels; falls back to SIMD
`openapi`	off	`utoipa::ToSchema` derives on the `api_types` DTOs
`update-check`	off	HTTP client for automatic version checking
`internal-bench`	off	Exposes internal hooks used by some benches
`bench-sift1m`	off	SIFT1M benchmark. Links `ureq`/TLS as a regular dependency — never enable in a shipping build
`loom`	off	Loom concurrency testing (nightly only)
`test-fault-injection`	off	RAII guards forcing internal failures in tests. Never enable in production

Runtime settings (HNSW parameters, limits, storage, logging) are read from velesdb.toml — see the configuration guide.

§Examples

examples/ — crash_driver (crash-recovery test driver), profile_batch_insert (flamegraph target for HNSW batch insert), simd_precision_check (SIMD vs scalar validation). These are engine tooling, not tutorials.
examples/rust/ — multimodel_search.rs, a runnable vector + graph + metadata query.
examples/mini_recommender/ and examples/ecommerce_recommendation/ — complete standalone applications.

§API / commands

Generated reference: docs.rs/velesdb-core. Import map (where each type lives): Core API map.

Task guides, all moved out of this README so it stays readable:

Guide	What it covers
Collections, metrics, storage	Collection model, the 5 distance metrics, embedding dimensions, payload format, quantized storage modes, bulk ingestion, durability
`VelesQL` reference	Vector/text/hybrid queries, metadata filters, `WITH` options, operator table, `JOIN` limit, `EXPLAIN`
Sparse vectors and fusion	Named sparse indexes, DAAT `MaxScore`, RRF and Relative Score fusion
Streaming inserts	`StreamIngester`, backpressure, delta buffer (insert-and-search)
Query plan cache	Two-tier LRU cache, write-generation invalidation, `EXPLAIN` cache fields, metrics
Agent Memory SDK (Rust)	Semantic, episodic and procedural memory, TTL, eviction, snapshots
Core performance	Every published number, its measurement context, and how to reproduce it
Graph patterns · Multi-model queries	Graph modelling and cross-engine statements
Search modes · Tuning guide · Quantization	Recall/latency trade-offs
Write concurrency · Concurrency and locking	The write model and file locking

§Performance

Two headline numbers, both measured rather than estimated. Every figure, its hardware and its reproduction command live in Core performance.

Claim	Measured	Context
Native HNSW search with AVX-512/AVX2/NEON SIMD	450µs p50 end-to-end	10K points, 384D, WAL on, recall ≥ 96%
`ColumnStore` filtering vs. scanning JSON payloads	up to 130x faster	integer equality, 100K rows

Reproduce with cargo bench -p velesdb-core --bench hnsw_benchmark and cargo bench -p velesdb-core --bench column_filter_benchmark.

Numbers move with hardware and dataset. Treat them as the shape of the engine’s cost, not a guarantee for your workload — measure on yours.

§Known limits

No embedding generation. You bring the vectors; the crate never calls a model or the network to produce them.
No clustering, sharding or replication. velesdb-core is a single-process embedded engine. One process at a time may open a database directory: a second one fails with DatabaseLocked.
One writer per collection. Concurrent readers are fine; concurrent writers to the same collection serialize — see Write concurrency.
Metric and dimension are immutable. Changing either means creating a new collection and reindexing.
JOIN ... USING (...) supports one column only. Multi-column USING parses but does not execute; use JOIN ... ON left = right instead.
No agent-memory service layer here. The explainable MemoryService, why() and the deterministic context compiler (compile_context) live one level up in velesdb-memory, which depends on this crate — never the reverse.
WASM builds are read/compute only. --no-default-features removes mmap storage, WAL, rayon and tokio along with the persistence feature.

§Compatibility

velesdb-core is a library, not an agent or MCP surface, so this table lists the platforms and toolchains the project builds and tests on.

Environment	Status	Note
Rust 1.90 (pinned)	Supported	`rust-toolchain.toml`; CI uses the same version
Linux `x86_64`	Supported	CI: `cargo check --workspace --all-targets --all-features`
Linux aarch64	Supported	CI: dedicated ARM64 benchmark runner (`ubuntu-24.04-arm`)
Windows `x86_64` (MSVC)	Supported	CI: `--all-features` check on `windows-latest`
macOS aarch64 / `x86_64`	Supported	Release pipeline builds both Darwin targets
`wasm32-unknown-unknown`	Supported, restricted	CI checks `--no-default-features` only; no filesystem persistence
Rust nightly	Build-checked	Only for the `loom` concurrency feature

§Troubleshooting

Symptom	Cause	Fix
`Error: CollectionExists("documents")`	`create_collection` re-run against an existing on-disk collection	Delete the database directory, or call `get_vector_collection` first and only create when it returns `None`
`[VELES-004] Vector dimension mismatch: expected 4, got 3`	The vector (on insert or query) does not match the dimension fixed at creation	Use your embedding model’s exact output size; the dimension cannot be changed after creation
`[VELES-031] Database is already opened by another process: <path>`	A second process tried to open the same directory	Close the first process, or point the second one at another directory — one writer process per database
`get_vector_collection` returns `None` for a name you created	The name belongs to a graph or metadata-only collection	Use `get_graph_collection` / `get_metadata_collection`, or `get_any_collection` for the type-erased handle
Data missing after a crash or `kill -9`	`upsert` updates in-memory/WAL state; destructors are best-effort	Call `flush()` as your explicit commit boundary

§License

VelesDB Core License 1.0 — see LICENSE.

Last updated: 2026-07-25 · Applies to: velesdb-core 5.0.0 · Report a docs error

§Crate-level notes

Local-first tri-engine database for AI agents, written in Rust: vector (HNSW + SIMD) + graph (typed edges, traversal, MATCH) + columnar (typed metadata filters), queried together through VelesQL — parse with velesql::Parser and run with Database::execute_query, mixing NEAR $v, WHERE predicates, and MATCH graph patterns in one statement (see the repository guide docs/guides/MULTIMODEL_QUERIES.md for the full multi-model tour). Designed for semantic search, agent memory, recommendation, and RAG.

§Features

Blazing Fast: HNSW index with explicit SIMD (4x faster)
5 Distance Metrics: Cosine, Euclidean, Dot Product, Hamming, Jaccard
Hybrid Search: Vector + BM25 full-text with RRF fusion
Graph Engine: typed edges on any collection, BFS/DFS, MATCH patterns
VelesQL: one SQL-like language across all three engines
Quantization: SQ8 (4x) and Binary (32x) memory compression
Persistent Storage: Memory-mapped files for efficient disk access

§Quick Start

use velesdb_core::{Database, DistanceMetric, Point, StorageMode};
use serde_json::json;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create a new database
    let db = Database::open("./data")?;

    // Create a collection (all 5 metrics available)
    db.create_collection("documents", 768, DistanceMetric::Cosine)?;
    // Or with quantization: DistanceMetric::Hamming + StorageMode::Binary

    let collection = db.get_vector_collection("documents").ok_or("Collection not found")?;

    // Insert vectors (upsert takes ownership)
    collection.upsert(vec![
        Point::new(1, vec![0.1; 768], Some(json!({"title": "Hello World"}))),
    ])?;

    // Search for similar vectors
    let query_vector = vec![0.1; 768];
    let results = collection.search(&query_vector, 10)?;

    // Hybrid search (vector + text)
    let hybrid = collection.hybrid_search(&query_vector, "hello", 5, Some(0.7))?;
}

Re-exports§

pub use index::HnswIndex;
pub use index::HnswParams;
pub use index::SearchQuality;
pub use index::VectorIndex;
pub use collection::streaming::BackpressureError;
pub use collection::streaming::StreamIngester;
pub use collection::streaming::StreamingConfig;
pub use collection::AnyCollection;
pub use collection::CollectionDiagnostics;
pub use collection::CollectionType;
pub use collection::EdgeType;
pub use collection::GraphCollection;
pub use collection::GraphEdge;
pub use collection::GraphNode;
pub use collection::GraphSchema;
pub use collection::IndexHealth;
pub use collection::IndexInfo;
pub use collection::MetadataCollection;
pub use collection::NodeType;
pub use collection::OrderByIndexState;
pub use collection::OrderByIndexSuggestion;
pub use collection::ScrollBatch;
pub use collection::TraversalConfig;
pub use collection::TraversalPath;
pub use collection::TraversalResult;
pub use collection::ValueType;
pub use collection::VectorCollection;
pub use collection::EXPIRES_AT_KEY;
pub use contiguous_ops::pad_to_simd_width;
pub use distance::DistanceMetric;
pub use distance::CONDITION_TYPE_NAMES;
pub use distance::DISTANCE_METRIC_NAMES;
pub use error::Error;
pub use error::Result;
pub use filter::Condition;
pub use filter::Filter;
pub use lock_rank::assert_lock_order;
pub use lock_rank::LockRank;
pub use point::ComponentScores;
pub use point::Point;
pub use point::SearchResult;
pub use quantization::cosine_similarity_quantized;
pub use quantization::cosine_similarity_quantized_simd;
pub use quantization::dot_product_quantized;
pub use quantization::dot_product_quantized_simd;
pub use quantization::euclidean_squared_quantized;
pub use quantization::euclidean_squared_quantized_simd;
pub use quantization::BinaryQuantizedVector;
pub use quantization::QuantizationCodec;
pub use quantization::QuantizedVector;
pub use quantization::StorageMode;
pub use quantization::STORAGE_MODE_NAMES;
pub use scored_result::ScoredResult;
pub use validation::validate_collection_name;
pub use validation::validate_dimension;
pub use validation::validate_dimension_match;
pub use validation::MAX_COLLECTION_NAME_LENGTH;
pub use validation::MAX_DIMENSION;
pub use validation::MIN_DIMENSION;
pub use wire::stable_hash::hash_edge_id;
pub use wire::stable_hash::hash_id;
pub use wire::stable_hash::hash_id_bytes;
pub use column_store::BatchUpdate;
pub use column_store::BatchUpdateResult;
pub use column_store::BatchUpsertResult;
pub use column_store::ColumnStore;
pub use column_store::ColumnStoreError;
pub use column_store::ColumnType;
pub use column_store::ColumnValue;
pub use column_store::ExpireResult;
pub use column_store::StringId;
pub use column_store::StringTable;
pub use column_store::TypedColumn;
pub use column_store::UpsertResult;
pub use config::ConfigError;
pub use config::HnswConfig;
pub use config::LimitsConfig;
pub use config::QuantizationConfig;
pub use config::QuantizationType;
pub use config::SearchConfig;
pub use config::SearchMode;
pub use config::VelesConfig;
pub use config::LoggingConfig;
pub use config::ServerConfig;
pub use config::StorageConfig;
pub use fusion::FusionError;
pub use fusion::FusionStrategy;
pub use fusion::DEFAULT_WEIGHTED_AVG_WEIGHT;
pub use fusion::DEFAULT_WEIGHTED_HIT_WEIGHT;
pub use fusion::DEFAULT_WEIGHTED_MAX_WEIGHT;
pub use guardrails::QueryLimits;
pub use metrics::average_metrics;
pub use metrics::compute_latency_percentiles;
pub use metrics::hit_rate;
pub use metrics::mean_average_precision;
pub use metrics::mrr;
pub use metrics::ndcg_at_k;
pub use metrics::precision_at_k;
pub use metrics::recall_at_k;
pub use metrics::LatencyStats;
pub use metrics::DurationHistogram;
pub use metrics::GuardRailsMetrics;
pub use metrics::OperationalMetrics;
pub use metrics::QueryStats;
pub use metrics::TraversalMetrics;
pub use observer::DatabaseObserver;
pub use observer::AccessDecision;
pub use observer::AccessScope;
pub use observer::QueryAccessContext;
pub use observer::QueryOperationKind;
pub use storage::DurabilityMode;

Modules§

agent: Agent Memory Patterns SDK (EPIC-010)
alloc_guard: RAII guards for safe manual memory management.
api_types: Canonical request/response DTOs shared across API layers.
cache: Caching layer for VelesDB (SOTA 2026).
collection: Collection management for VelesDB.
column_store: Column-oriented storage for high-performance metadata filtering.
compression: Column compression for VelesDB (SOTA 2026).
config: VelesDB Configuration Module
config_quantization: Quantization configuration types (PQ-06).
conformance: Cross-implementation conformance harness (frozen golden reference vectors). Cross-implementation conformance harness (Requirement 7).
contiguous_ops: Reorder, batch distance, and lifecycle operations for ContiguousVectors.
distance: Distance metrics for vector similarity calculations.
error: Error types for VelesDB.
filter: Metadata filtering for vector search.
fusion: Multi-query fusion strategies for VelesDB.
gpu: GPU-accelerated vector operations using wgpu (WebGPU).
guardrails: Production-grade query guard-rails (EPIC-048).
half_precision: Half-precision floating point support for memory-efficient vector storage.
index: Index implementations for efficient vector search.
lock_rank: Compiled lock-rank invariant for the global lock-acquisition order.
metrics: Search quality metrics, operational monitoring, and query diagnostics.
observer: DatabaseObserver — extension hook for velesdb-premium.
perf_optimizations: Performance optimizations module for ultra-fast vector operations.
point: Point data structure representing a vector with metadata.
quantization: Scalar Quantization (SQ8) and Binary Quantization for memory-efficient vector storage.
scored_result: Unified scored result type for vector search and graph traversal.
simd_dispatch: Zero-overhead SIMD function dispatch.
simd_native: Native SIMD intrinsics for maximum performance.
sparse_index: Sparse vector types, inverted index, and search – always compiled (no persistence dependency). Sparse vector types, inverted index, and search.
storage: Storage backends for persistent vector storage.
sync: Synchronization primitives with loom support for concurrency testing.
validation: Unified validation helpers.
vector_ref: Zero-copy vector reference abstraction.
velesql: VelesQL - SQL-like query language for VelesDB.
wire: Binary wire formats (VRB1 raw-bulk) — pure, persistence-free, wasm-safe. Binary wire formats shared across the server, CLI, and SDKs.

Macros§

simd_4acc_dot_loop: 4-accumulator unrolled SIMD loop for dot product (ILP optimization).
simd_4acc_l2_loop: 4-accumulator unrolled SIMD loop for squared L2 distance.
simd_8acc_dot_loop: 8-accumulator unrolled SIMD loop for dot product (ILP optimization).
simd_8acc_l2_loop: 8-accumulator unrolled SIMD loop for squared L2 distance.
sum_remainder_unrolled_8: Macro for unrolled remainder sum computation (1-7 elements). Generates optimal code for remainders 1-7 with 4->2->1 unrolling.
sum_squared_remainder_unrolled_8: Macro for unrolled squared L2 remainder (1-7 elements).

Structs§

Database: Database instance managing collections and storage.

Enums§

GatedRead: A non-VelesQL read routed through the control-plane gate.