OxiRS Vec - Vector Search Engine

Status: v0.3.2 - Released 2026-07-12

✨ Production Release: Production-ready with API stability guarantees and comprehensive testing (1,754 tests passing).

High-performance vector search infrastructure for semantic similarity search in RDF knowledge graphs.

Features

Vector Indexing

• HNSW Index - Hierarchical Navigable Small World graphs for fast approximate nearest neighbor search
• Flat Index - Exact search for smaller datasets
• IVF Index - Inverted file index for large-scale datasets
• Dynamic Updates - Real-time index updates without full rebuilds
• Real-Time Embedding Pipeline - Streaming embedding updates with configurable consistency, per-ID version history, and live health/alerting monitoring (see Real-Time Embedding Pipeline below)

Search Capabilities

• Similarity Search - Find semantically similar entities
• Filtered Search - Combine vector similarity with RDF constraints
• Batch Operations - Efficient bulk indexing and search
• Multiple Distance Metrics - Cosine, Euclidean, Manhattan, Dot product

Integration

• SPARQL Extension - Vector search functions in SPARQL queries
• GraphQL Support - Vector similarity in GraphQL queries
• Embedding Models - Integration with various embedding providers
• Storage Backends - Persistent vector indices

Installation

Add to your Cargo.toml:

# Experimental feature
[dependencies]
oxirs-vec = "0.3.2"

Quick Start

Basic Vector Search

use oxirs_vec::{VectorStore, IndexType, DistanceMetric};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create vector store with HNSW index
    let mut store = VectorStore::builder()
        .index_type(IndexType::HNSW)
        .dimension(768)  // Embedding dimension
        .distance_metric(DistanceMetric::Cosine)
        .build()?;

    // Add vectors
    store.add_vector("entity1", &embedding1)?;
    store.add_vector("entity2", &embedding2)?;

    // Build index
    store.build_index()?;

    // Search for similar vectors
    let results = store.search(&query_vector, 10, 0.8)?;
    for result in results {
        println!("ID: {}, Score: {}", result.id, result.score);
    }

    Ok(())
}

SPARQL Integration

use oxirs_vec::sparql::VectorFunctions;

let sparql = r#"
    PREFIX vec: <http://oxirs.org/vec/>

    SELECT ?entity ?score WHERE {
        ?entity a foaf:Person .

        # Vector similarity search
        ?entity vec:similarTo "machine learning researcher" .
        ?entity vec:similarity ?score .

        FILTER (?score > 0.8)
    }
    ORDER BY DESC(?score)
    LIMIT 10
"#;

Architecture

Index Types

HNSW (Hierarchical Navigable Small World)

• Use Case: General purpose, balanced performance
• Search Time: O(log N)
• Build Time: O(N log N)
• Memory: Moderate

Flat Index

• Use Case: Small datasets, exact search required
• Search Time: O(N)
• Build Time: O(N)
• Memory: Low

IVF (Inverted File)

• Use Case: Large datasets, acceptable approximate results
• Search Time: O(√N)
• Build Time: O(N)
• Memory: Moderate

Distance Metrics

pub enum DistanceMetric {
    Cosine,      // For normalized embeddings
    Euclidean,   // For absolute distances
    Manhattan,   // For high-dimensional spaces
    DotProduct,  // For similarity scores
}

Advanced Features

Filtered Search

Combine vector similarity with RDF constraints:

use oxirs_vec::FilteredSearch;

let filters = FilteredSearch::builder()
    .add_constraint("rdf:type", "foaf:Person")
    .add_constraint("foaf:age", |age: i32| age > 18)
    .build();

let results = store.filtered_search(&query_vector, filters, 10)?;

Batch Operations

Efficient bulk indexing:

let batch = vec![
    ("entity1", embedding1),
    ("entity2", embedding2),
    ("entity3", embedding3),
];

store.add_batch(batch)?;
store.build_index()?;

Incremental Updates

// Add without full rebuild
store.add_incremental("new_entity", &embedding)?;

// Periodic optimization
store.optimize_index()?;

Performance

Benchmarks (on sample datasets)

Benchmarked on M1 Mac with 768-dimensional vectors

Configuration

let config = VectorStoreConfig {
    index_type: IndexType::HNSW,
    dimension: 768,
    distance_metric: DistanceMetric::Cosine,

    // HNSW-specific parameters
    hnsw_m: 16,              // Number of connections per node
    hnsw_ef_construction: 200, // Construction time accuracy
    hnsw_ef_search: 100,      // Search time accuracy

    // Storage options
    persist_path: Some("./vector_index".into()),
    cache_size: 1000,
};

Integration Examples

With oxirs-embed

use oxirs_embed::EmbeddingModel;
use oxirs_vec::VectorStore;

// Generate embeddings
let model = EmbeddingModel::load("sentence-transformers/all-mpnet-base-v2")?;
let embedding = model.encode("Machine learning research")?;

// Index and search
let mut store = VectorStore::new(IndexType::HNSW, 768)?;
store.add_vector("doc1", &embedding)?;

With oxirs-core (RDF)

use oxirs_core::Dataset;
use oxirs_vec::RdfVectorIndex;

let dataset = Dataset::from_file("knowledge_graph.ttl")?;
let mut index = RdfVectorIndex::new(&dataset)?;

// Index entities by their descriptions
for entity in dataset.subjects() {
    if let Some(description) = dataset.get_description(&entity) {
        let embedding = model.encode(&description)?;
        index.add_entity(&entity, &embedding)?;
    }
}

Real-Time Embedding Pipeline

real_time_embedding_pipeline is a streaming embedding-update system for high-throughput, low-latency workloads. It is organized into config, traits, types, pipeline, streaming, coordination, monitoring, versioning, and consistency submodules, all compiled in and wired live into RealTimeEmbeddingPipeline:

• consistency - InconsistencyRepairEngine detects and repairs embedding drift with severity-based repair strategies
• versioning - VersionManager keeps a per-ID embedding version history with configurable retention
• monitoring - PipelinePerformanceMonitor/MetricsCollector/AlertManager for metrics collection, health checks, and severity-throttled alerting
• coordination - UpdateCoordinator synchronizes concurrent update batches

use oxirs_vec::real_time_embedding_pipeline::{
    RealTimeEmbeddingPipeline, PipelineConfig, ConsistencyLevel
};

let config = PipelineConfig {
    max_batch_size: 1000,
    consistency_level: ConsistencyLevel::Session,
    ..Default::default()
};

let runtime = tokio::runtime::Runtime::new()?;
runtime.block_on(async {
    let mut pipeline = RealTimeEmbeddingPipeline::new(config)?;
    pipeline.start().await?;
    pipeline.stop().await?;
    Ok::<(), Box<dyn std::error::Error>>(())
})?;
# Ok::<(), Box<dyn std::error::Error>>(())

Status

Production Release (v0.3.2)

• ✅ HNSW/IVF/Flat indices with persisted dataset support
• ✅ SPARQL/GraphQL integration enhanced with federation-aware vector filters
• ✅ CLI pipelines for batch embedding import/export and monitoring
• ✅ SciRS2 metrics for query latency, recall, and index health
• ✅ Real-time embedding pipeline with consistency repair, versioning, and monitoring submodules wired live (previously declared but commented out as unimplemented)
• ✅ GPU acceleration abstractions (Pure Rust, no-op by default); real NVIDIA CUDA acceleration is provided by the separate oxirs-vec-adapter-cuda crate (publish = false), which quarantines the cuda-runtime-sys C FFI off this crate's Pure-Rust dependency surface per the COOLJAPAN Pure Rust Policy v2 — the former cuda/gpu-full features were removed
• ✅ Distributed indexing - Raft-based replication and cross-datacenter sync (distributed module)

Contributing

This is an experimental module. Feedback and contributions are welcome!

License

Apache-2.0

See Also

• oxirs-embed - Embedding generation
• oxirs-arq - SPARQL query engine
• oxirs-core - RDF data model