Crate ipfrs_semantic 

IPFRS Semantic - Vector Search and Semantic Routing

This crate provides high-performance semantic search and routing capabilities for IPFRS, enabling content discovery based on vector embeddings and semantic similarity.

Features

• HNSW-based Vector Search - Fast approximate nearest neighbor search
• Semantic Routing - Content discovery based on embeddings
• Hybrid Search - Combine vector search with metadata filtering
• Vector Quantization - Memory-efficient index compression (PQ, OPQ, Scalar)
• Learned Indices - ML-based indexing with Recursive Model Index (RMI)
• Logic Integration - TensorLogic reasoning with embeddings
• DiskANN - Disk-based indexing for 100M+ vectors
• SIMD Optimization - ARM NEON and x86 SSE/AVX acceleration
• Caching - Hot embedding cache with LRU eviction
• Batch Query Processing - Parallel batch queries for high throughput
• Query Re-ranking - Multi-criteria result re-ranking with weighted scoring
• Query Analytics - Performance tracking and query pattern analysis
• Multi-Modal Search - Unified search across text, image, audio, video, and code
• Differential Privacy - Privacy-preserving embeddings with configurable privacy budgets
• Dynamic Updates - Online embedding updates and version migration
• Vector Quality Analysis - Data validation, anomaly detection, and quality metrics (NEW!)
• Index Diagnostics - Health monitoring, performance profiling, and issue detection (NEW!)
• Index Optimization - Automatic parameter tuning and resource management (NEW!)
• Auto-Scaling Advisor - Intelligent scaling recommendations for production deployments (NEW!)

Quick Start

Basic Semantic Search

use ipfrs_semantic::{SemanticRouter, RouterConfig};
use ipfrs_core::Cid;

// Create a semantic router with default configuration
let router = SemanticRouter::with_defaults()?;

// Index content with embeddings (typically from a model like BERT, CLIP, etc.)
let cid1: Cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi".parse()?;
let embedding1 = vec![0.1, 0.2, 0.3]; // 768-dim embedding in real use

// Add to index
router.add(&cid1, &vec![0.5; 768])?;

// Query for similar content
let query_embedding = vec![0.5; 768];
let results = router.query(&query_embedding, 10).await?;

for result in results {
    println!("CID: {}, Score: {}", result.cid, result.score);
}

Batch Query for High Throughput

use ipfrs_semantic::{SemanticRouter, RouterConfig};
use ipfrs_core::Cid;

// Create a semantic router
let router = SemanticRouter::with_defaults()?;

// Index multiple items
let items = vec![
    ("bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi".parse::<Cid>()?, vec![0.1; 768]),
    ("bafybeihpjhkeuiq3k6nqa3fkgeigeri7iebtrsuyuey5y6vy36n345xmbi".parse::<Cid>()?, vec![0.2; 768]),
    ("bafybeif2pall7dybz7vecqka3zo24irdwabwdi4wc55jznaq75q7eaavvu".parse::<Cid>()?, vec![0.3; 768]),
];

router.add_batch(&items)?;

// Batch query - process multiple queries in parallel
let query_embeddings = vec![
    vec![0.15; 768],
    vec![0.25; 768],
    vec![0.35; 768],
];

// More efficient than querying one by one
let batch_results = router.query_batch(&query_embeddings, 10).await?;

for (i, results) in batch_results.iter().enumerate() {
    println!("Query {} found {} results", i, results.len());
}

// Get batch statistics
let stats = router.batch_stats(&batch_results);
println!("Total queries: {}", stats.total_queries);
println!("Avg results per query: {:.2}", stats.avg_results_per_query);
println!("Avg score: {:.4}", stats.avg_score);

Hybrid Search with Metadata Filtering

use ipfrs_semantic::{HybridIndex, HybridConfig, HybridQuery, Metadata, MetadataValue, MetadataFilter};
use ipfrs_core::Cid;

// Create hybrid index
let config = HybridConfig::default();
let index = HybridIndex::new(config)?;

// Index content with metadata
let cid: Cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi".parse()?;
let embedding = vec![0.5; 768];

let mut metadata = Metadata::new();
metadata.set("type", MetadataValue::String("image".to_string()));
metadata.set("size", MetadataValue::Integer(1024));

index.insert(&cid, &embedding, Some(metadata))?;

// Query with filters using builder pattern
let filter = MetadataFilter::eq("type", MetadataValue::String("image".to_string()));
let query = HybridQuery::knn(vec![0.5; 768], 10)
    .with_filter(filter);

let response = index.search(query).await?;
println!("Found {} results", response.results.len());

Vector Quantization for Memory Efficiency

use ipfrs_semantic::{ProductQuantizer, ScalarQuantizer};

// Create Product Quantizer (8-32x compression)
let dimension = 768;
let num_subspaces = 8;
let bits_per_subspace = 8;

let mut pq = ProductQuantizer::new(dimension, num_subspaces, bits_per_subspace)?;

// Train on representative data (1000 training samples, max 10 iterations)
let training_data: Vec<Vec<f32>> = vec![vec![0.5; 768]; 1000];
pq.train(&training_data, 10)?;

// Quantize embeddings
let embedding = vec![0.5; 768];
let quantized = pq.quantize(&embedding)?;

println!("Original size: {} bytes", dimension * 4);
println!("Quantized size: {} bytes", quantized.codes.len());

DiskANN for Large-Scale Indexing

use ipfrs_semantic::{DiskANNIndex, DiskANNConfig};
use ipfrs_core::Cid;

// Create DiskANN index for 100M+ vectors
let config = DiskANNConfig {
    dimension: 768,
    max_degree: 32,
    ..Default::default()
};

let mut index = DiskANNIndex::new(config);
index.create("/tmp/diskann_index")?;

// Insert vectors (stored on disk, not in RAM)
let cid: Cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi".parse()?;
let embedding = vec![0.5; 768];
index.insert(&cid, &embedding)?;

// Search with constant memory usage
let results = index.search(&embedding, 10)?;
println!("Found {} results from disk", results.len());

Learned Index Structures

use ipfrs_semantic::{LearnedIndex, RMIConfig, ModelType};
use ipfrs_core::Cid;

// Create a learned index with Recursive Model Index (RMI)
let config = RMIConfig {
    num_models: 10,              // Number of second-stage models
    model_type: ModelType::Linear, // Linear, Polynomial, or NeuralNetwork
    training_iterations: 100,
    learning_rate: 0.01,
    error_threshold: 0.05,
};

let mut index = LearnedIndex::new(config);

// Add embeddings - the index learns data distribution
for i in 0..1000 {
    let cid = Cid::default();
    let embedding = vec![i as f32 / 1000.0; 768];
    index.add(cid, embedding)?;
}

// The index automatically rebuilds and trains models
let query = vec![0.5; 768];
let results = index.search(&query, 10)?;

// Check statistics
let stats = index.stats();
println!("Indexed {} points using {} models",
         stats.data_points, stats.num_models);

TensorLogic Integration

use ipfrs_semantic::{LogicSolver, SolverConfig};
use ipfrs_tensorlogic::{Predicate, Term, Constant};
use ipfrs_core::Cid;

// Create a logic solver with semantic similarity
let config = SolverConfig {
    max_depth: 100,
    similarity_threshold: 0.8,
    top_k_similar: 10,
    embedding_dim: 384,
    detect_cycles: true,
};

let mut solver = LogicSolver::new(config)?;

// Add facts to the knowledge base
let cid1: Cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi".parse()?;
let fact1 = Predicate::new("likes".to_string(), vec![
    Term::Const(Constant::String("alice".to_string())),
    Term::Const(Constant::String("rust".to_string())),
]);
solver.add_fact(fact1, cid1)?;

let cid2: Cid = "bafybeihpjhkeuiq3k6nqa3fkgeigeri7iebtrsuyuey5y6vy36n345xmbi".parse()?;
let fact2 = Predicate::new("likes".to_string(), vec![
    Term::Const(Constant::String("bob".to_string())),
    Term::Const(Constant::String("python".to_string())),
]);
solver.add_fact(fact2, cid2)?;

// Add a rule for similarity-based matching
// Rule: similar(X, Y) :- likes(X, Lang), likes(Y, Lang)
let head = Predicate::new("similar".to_string(), vec![
    Term::Var("X".to_string()),
    Term::Var("Y".to_string())
]);
let body = vec![
    Predicate::new("likes".to_string(), vec![
        Term::Var("X".to_string()),
        Term::Var("Lang".to_string())
    ]),
    Predicate::new("likes".to_string(), vec![
        Term::Var("Y".to_string()),
        Term::Var("Lang".to_string())
    ]),
];
solver.add_rule(head, body)?;

// Query using semantic similarity
let query = Predicate::new("likes".to_string(), vec![
    Term::Var("Who".to_string()),
    Term::Const(Constant::String("rust".to_string())),
]);

let results = solver.query(&query)?;
for substitution in results {
    println!("Found substitution: {:?}", substitution);
}

// Get solver statistics
let stats = solver.stats();
println!("Total facts: {}", stats.num_facts);
println!("Total rules: {}", stats.num_rules);
println!("Indexed predicates: {}", stats.num_indexed_predicates);

Custom Embedding Model Integration

use ipfrs_semantic::{SemanticRouter, RouterConfig, DistanceMetric};
use ipfrs_core::Cid;

// Configure router for your embedding model
let config = RouterConfig {
    dimension: 768,  // BERT-base dimension
    metric: DistanceMetric::Cosine,
    max_connections: 16,
    ef_construction: 200,
    ef_search: 50,
    cache_size: 1000,
};

let router = SemanticRouter::new(config)?;

// Function to generate embeddings from your model
// This is a placeholder - replace with your actual model
fn generate_embedding(text: &str) -> Vec<f32> {
    // Example: Use sentence-transformers, Hugging Face transformers, etc.
    // let model = SentenceTransformer::new("all-MiniLM-L6-v2");
    // model.encode(text)

    // For this example, just return a dummy embedding
    vec![0.5; 768]
}

// Index documents with embeddings
let documents = vec![
    ("Rust programming language", "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"),
    ("Python machine learning", "bafybeihpjhkeuiq3k6nqa3fkgeigeri7iebtrsuyuey5y6vy36n345xmbi"),
    ("Distributed systems", "bafybeif2pall7dybz7vecqka3zo24irdwabwdi4wc55jznaq75q7eaavvu"),
];

for (text, cid_str) in documents {
    let cid: Cid = cid_str.parse()?;
    let embedding = generate_embedding(text);
    router.add(&cid, &embedding)?;
}

// Query with natural language
let query_text = "rust systems programming";
let query_embedding = generate_embedding(query_text);
let results = router.query(&query_embedding, 5).await?;

println!("Top results for '{}':", query_text);
for result in results {
    println!("  CID: {}, Score: {:.3}", result.cid, result.score);
}

Distributed Semantic Search

For large-scale deployments across multiple nodes:

use ipfrs_semantic::{SemanticDHTNode, SemanticDHTConfig, VectorIndex, DistanceMetric};
use ipfrs_network::libp2p::PeerId;
use ipfrs_core::Cid;

// Configure distributed semantic DHT
let config = SemanticDHTConfig {
    embedding_dim: 768,
    replication_factor: 3,     // Replicate to 3 peers
    routing_table_size: 20,    // Top 20 nearest peers
    distance_metric: DistanceMetric::Cosine,
    max_hops: 5,               // Maximum query propagation hops
    query_timeout_ms: 5000,    // 5 second timeout
};

// Create local vector index
let local_index = VectorIndex::new(768, DistanceMetric::Cosine, 16, 200)?;

// Create DHT node
let local_peer_id = PeerId::random();
let dht_node = SemanticDHTNode::new(config, local_peer_id, local_index);

// Add peer to routing table
use ipfrs_semantic::SemanticPeer;
let peer_id = PeerId::random();
let peer_embedding = vec![0.5; 768];  // Aggregate embedding of peer's data
let peer = SemanticPeer::new(peer_id, peer_embedding);
dht_node.routing_table().add_peer(peer)?;

// Insert data (automatically replicated to nearest peers)
let cid: Cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi".parse()?;
let embedding = vec![0.7; 768];
dht_node.insert(&cid, &embedding).await?;

// Distributed k-NN search across the network
let query_embedding = vec![0.6; 768];
let results = dht_node.search_distributed(&query_embedding, 10).await?;

println!("Found {} results from distributed search", results.len());
for result in results {
    println!("  CID: {}, Score: {:.3}", result.cid, result.score);
}

// Update peer clusters for locality optimization
dht_node.routing_table().update_clusters(3)?;

// Get DHT statistics
let stats = dht_node.get_stats();
println!("DHT Stats:");
println!("  Peers: {}", stats.num_peers);
println!("  Clusters: {}", stats.num_clusters);
println!("  Local entries: {}", stats.num_local_entries);
println!("  Queries processed: {}", stats.queries_processed);
println!("  Avg latency: {:.2}ms", stats.avg_query_latency_ms);

Federated Multi-Index Search

Query multiple indices simultaneously with heterogeneous distance metrics:

use ipfrs_semantic::{
    FederatedQueryExecutor, FederatedConfig, AggregationStrategy,
    LocalIndexAdapter, VectorIndex, DistanceMetric
};
use ipfrs_core::Cid;
use parking_lot::RwLock;
use std::sync::Arc;

// Configure federated queries
let mut config = FederatedConfig::default();
config.aggregation_strategy = AggregationStrategy::RankFusion; // Best for heterogeneous metrics
config.privacy_preserving = true;  // Enable differential privacy
config.privacy_noise_level = 0.01; // Small noise for privacy

let executor = FederatedQueryExecutor::new(config);

// Create multiple indices with different metrics
let index1 = VectorIndex::new(768, DistanceMetric::Cosine, 16, 200)?;
let index2 = VectorIndex::new(768, DistanceMetric::L2, 16, 200)?;
let index3 = VectorIndex::new(768, DistanceMetric::DotProduct, 16, 200)?;

// Populate indices with data
// (In practice, these might be from different organizations or data sources)
let cid1: Cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi".parse()?;
let embedding1 = vec![0.5; 768];
Arc::new(RwLock::new(index1)).write().insert(&cid1, &embedding1)?;

// Register indices for federated queries
let adapter1 = LocalIndexAdapter::new(
    Arc::new(RwLock::new(VectorIndex::new(768, DistanceMetric::Cosine, 16, 200)?)),
    "org1_index".to_string()
);
let adapter2 = LocalIndexAdapter::new(
    Arc::new(RwLock::new(VectorIndex::new(768, DistanceMetric::L2, 16, 200)?)),
    "org2_index".to_string()
);

executor.register_index(Arc::new(adapter1))?;
executor.register_index(Arc::new(adapter2))?;

// Query all registered indices simultaneously
let query_embedding = vec![0.6; 768];
let results = executor.query(&query_embedding, 10).await?;

println!("Federated search found {} results", results.len());
for result in results {
    println!(
        "  CID: {}, Score: {:.3}, Source: {}, Metric: {:?}",
        result.cid, result.score, result.source_index_id, result.source_metric
    );
}

// Query specific indices only
let specific_results = executor.query_indices(
    &query_embedding,
    10,
    &["org1_index".to_string()]
).await?;

// Get federated query statistics
let stats = executor.stats();
println!("Federated Query Stats:");
println!("  Total queries: {}", stats.total_queries);
println!("  Indices queried: {}", stats.total_indices_queried);
println!("  Avg latency: {:.2}ms", stats.avg_latency_ms);

Multi-Modal Semantic Search

Search across different data types (text, images, audio, etc.) in a unified embedding space:

use ipfrs_semantic::{MultiModalIndex, MultiModalConfig, MultiModalEmbedding, Modality};
use ipfrs_core::Cid;

// Create multi-modal index
let mut config = MultiModalConfig::default();
config.project_to_unified = true;  // Enable unified embedding space
config.unified_dim = 512;

let mut index = MultiModalIndex::new(config);

// Register different modalities with their native dimensions
index.register_modality(Modality::Text, 768)?;  // BERT embeddings
index.register_modality(Modality::Image, 512)?;  // ResNet embeddings
index.register_modality(Modality::Audio, 768)?;  // Wav2Vec embeddings

// Add text content
let text_cid: Cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi".parse()?;
let text_embedding = MultiModalEmbedding::new(
    vec![0.1; 768],  // Text embedding from BERT
    Modality::Text
);
index.add(text_cid, text_embedding)?;

// Add image content
let image_cid: Cid = "bafybeigvgzoolh3cxsculpsjkz3hxfpg37pszqx3j7i5fwzgjmrmtv5wmi".parse()?;
let image_embedding = MultiModalEmbedding::new(
    vec![0.2; 512],  // Image embedding from ResNet
    Modality::Image
);
index.add(image_cid, image_embedding)?;

// Search within a specific modality
let text_query = MultiModalEmbedding::new(vec![0.15; 768], Modality::Text);
let text_results = index.search_modality(&text_query, 5, None)?;

// Cross-modal search: find similar content across all modalities
let cross_modal_results = index.search_cross_modal(&text_query, 10, None)?;
for (cid, score, modality) in cross_modal_results {
    println!("Found {:?} content: {} (score: {:.3})", modality, cid, score);
}

// Get statistics
let stats = index.stats();
for (modality, stat) in stats {
    println!("{:?}: {} embeddings, {} dims", modality, stat.num_embeddings, stat.dimension);
}

Privacy-Preserving Search with Differential Privacy

Protect embedding privacy while maintaining search utility:

use ipfrs_semantic::{PrivacyMechanism, PrivacyBudget, PrivateEmbedding, TradeoffAnalyzer};

// Create a privacy mechanism (epsilon-differential privacy)
let epsilon = 1.0;  // Privacy budget
let sensitivity = 1.0;  // L2 sensitivity of embeddings
let mechanism = PrivacyMechanism::laplacian(epsilon, sensitivity)?;

// Create a private embedding
let original_embedding = vec![0.5; 768];
let private_emb = PrivateEmbedding::new(original_embedding, mechanism);

// Use the noisy embedding for public release
let public_embedding = private_emb.public_embedding();
let (epsilon, delta) = private_emb.privacy_params();
println!("Privacy: ε={}, δ={}", epsilon, delta);
println!("Expected utility loss: {:.3}", private_emb.utility_loss());

// Track privacy budget across multiple queries
let budget = PrivacyBudget::new(10.0, 0.001)?;  // Total budget

// Consume budget for each query
budget.consume(0.5, 0.0001)?;
budget.consume(0.5, 0.0001)?;

println!("Remaining budget: {:.2}", budget.remaining());

// Analyze privacy-utility trade-offs
let analyzer = TradeoffAnalyzer::new(sensitivity);
let tradeoffs = analyzer.analyze(768);
for point in tradeoffs {
    println!("ε={:.1}: utility loss={:.2}", point.epsilon, point.utility_loss);
}

// Find best epsilon for target utility
if let Some(best_epsilon) = analyzer.find_epsilon_for_utility(768, 15.0) {
    println!("Best ε for utility loss <15.0: {:.2}", best_epsilon);
}

Dynamic Embedding Updates and Version Migration

Manage evolving embeddings with version control and online updates:

use ipfrs_semantic::{DynamicIndex, ModelVersion, OnlineUpdater, EmbeddingTransform};
use ipfrs_core::Cid;

// Create a dynamic index with version tracking
let v1 = ModelVersion::new(1, 0, 0);
let index = DynamicIndex::new(v1.clone(), 768)?;

// Add embeddings to version 1.0.0
let cid: Cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi".parse()?;
let embedding_v1 = vec![0.5; 768];
index.insert(&cid, &embedding_v1, None)?;

// Add a new model version with transformation
let v2 = ModelVersion::new(1, 1, 0);
let transform = EmbeddingTransform::identity(v1.clone());
index.add_version(v2.clone(), Some(transform))?;

// Set the new version as active
index.set_active_version(v2.clone())?;

// Add new embeddings to v2
let cid2: Cid = "bafybeigvgzoolh3cxsculpsjkz3hxfpg37pszqx3j7i5fwzgjmrmtv5wmi".parse()?;
let embedding_v2 = vec![0.6; 768];
index.insert(&cid2, &embedding_v2, Some(v2))?;

// Online fine-tuning with momentum
let updater = OnlineUpdater::new(0.01, 0.9);  // learning_rate, momentum

// Apply gradient updates
let gradient = vec![0.001; 768];
let updated_embedding = updater.update(&cid, &embedding_v2, &gradient);

// Track versions
let stats = index.version_stats();
for (version, stat) in stats {
    println!("Version {}: {} embeddings (active: {})",
        version, stat.num_embeddings, stat.is_active);
}

// Online updater statistics
let updater_stats = updater.stats();
println!("Online updater: lr={}, momentum={}, tracking {} embeddings",
    updater_stats.learning_rate, updater_stats.momentum, updater_stats.num_tracked);

Performance

SIMD Acceleration

The crate includes SIMD-optimized distance computations:

use ipfrs_semantic::{l2_distance, cosine_distance, dot_product};

let vec1 = vec![1.0, 2.0, 3.0, 4.0];
let vec2 = vec![0.5, 1.5, 2.5, 3.5];

// Uses ARM NEON or x86 SSE/AVX when available
let l2_dist = l2_distance(&vec1, &vec2);
let cos_dist = cosine_distance(&vec1, &vec2);
let dot_prod = dot_product(&vec1, &vec2);

Performance Targets

• Query latency: < 1ms for 1M vectors (cached)
• Query latency: < 5ms for 1M vectors (uncached)
• Index build time: < 10min for 1M vectors
• Memory usage: < 2GB for 1M × 768-dim vectors
• Recall@10: > 95% for k-NN search

Architecture

Core Components

• VectorIndex - HNSW-based vector search index
• SemanticRouter - High-level routing with caching
• HybridIndex - Hybrid search with metadata filtering
• DiskANNIndex - Disk-based indexing for massive scale

Optimization Layers

• Quantization - ProductQuantizer, OptimizedProductQuantizer, ScalarQuantizer
• Caching - HotEmbeddingCache, AlignedVector
• SIMD - l2_distance, cosine_distance, dot_product

Logic Integration

• LogicSolver - TensorLogic reasoning with embeddings
• QueryExecutor - SPARQL-like query language
• ProvenanceTracker - Audit trails and provenance

Use Cases

Semantic Content Discovery

Find similar content based on embeddings from models like:

• Text: BERT, RoBERTa, Sentence Transformers
• Images: CLIP, ResNet, ViT
• Multi-modal: CLIP, ALIGN

Recommendation Systems

Build recommendation engines that find similar:

• Documents based on text embeddings
• Images based on visual features
• Users based on behavior embeddings

Distributed AI Model Routing

Route AI inference requests to:

• Find similar cached results
• Locate relevant model weights
• Discover related training data

Configuration

Index Tuning

use ipfrs_semantic::{VectorIndex, DistanceMetric, ParameterTuner, UseCase};

// Get recommended parameters for your use case
let rec = ParameterTuner::recommend(
    100_000,              // number of vectors
    768,                  // dimension
    UseCase::HighRecall   // optimize for recall
);

// Create index with recommended parameters
let index = VectorIndex::new(
    768,
    DistanceMetric::Cosine,
    rec.m,
    rec.ef_construction
)?;

println!("M: {}, efConstruction: {}", rec.m, rec.ef_construction);
println!("Estimated recall@10: {:.2}%", rec.estimated_recall * 100.0);

Query Language

The crate provides a SPARQL-like query language for complex knowledge base queries:

use ipfrs_semantic::{Query, QueryPattern, QueryExecutor, FilterExpr, TermPattern};
use ipfrs_tensorlogic::{KnowledgeBase, Predicate, Term, Constant};

// Create knowledge base
let mut kb = KnowledgeBase::new();

// Add some facts
let fact1 = Predicate::new("person".to_string(), vec![
    Term::Const(Constant::String("alice".to_string())),
]);
kb.add_fact(fact1);

let fact2 = Predicate::new("age".to_string(), vec![
    Term::Const(Constant::String("alice".to_string())),
    Term::Const(Constant::Int(30)),
]);
kb.add_fact(fact2);

// Create query executor
let executor = QueryExecutor::new(kb);

// Build a query using the builder pattern
let query = Query::new()
    .select("name")
    .select("age_val")
    .where_pattern(QueryPattern::Pattern {
        name: Some("person".to_string()),
        args: vec![TermPattern::Variable("name".to_string())],
    })
    .where_pattern(QueryPattern::Pattern {
        name: Some("age".to_string()),
        args: vec![
            TermPattern::Variable("name".to_string()),
            TermPattern::Variable("age_val".to_string()),
        ],
    })
    .limit(10);

// Execute the query
let result = executor.execute(query)?;

println!("Found {} results", result.bindings.len());
for binding in result.bindings {
    println!("  Name: {:?}, Age: {:?}", binding.get("name"), binding.get("age_val"));
}

// Query statistics
println!("Patterns evaluated: {}", result.stats.patterns_evaluated);
println!("Execution time: {} ms", result.stats.execution_time_ms);

Query Features

• SELECT clause: Specify variables to return
• WHERE patterns: Pattern matching with wildcards and variables
• FILTER expressions: Filter results with boolean logic
• LIMIT/OFFSET: Pagination support
• Query optimization: Automatic join order optimization and filter pushdown

Boolean Queries

use ipfrs_semantic::{BooleanQuery, Query, FilterExpr};

// AND query: match both conditions
let and_query = BooleanQuery::And(vec![
    Query::new().select("x"),
    Query::new().select("y"),
]);

// OR query: match either condition
let or_query = BooleanQuery::Or(vec![
    Query::new().select("x"),
    Query::new().select("y"),
]);

// NOT query: negate a query
let not_query = BooleanQuery::Not(Box::new(
    Query::new().select("x")
));

Error Handling

All operations return Result<T, ipfrs_core::Error>:

use ipfrs_semantic::SemanticRouter;
use ipfrs_core::Error;

match SemanticRouter::with_defaults() {
    Ok(router) => println!("Router created successfully"),
    Err(Error::InvalidInput(msg)) => eprintln!("Invalid input: {}", msg),
    Err(e) => eprintln!("Error: {}", e),
}

Advanced Features

Vector Quality Analysis

Validate embeddings and detect anomalies:

use ipfrs_semantic::{analyze_quality, detect_anomaly, compute_batch_stats};

// Analyze a single vector
let embedding = vec![0.1, 0.2, 0.3, 0.4, 0.5];
let quality = analyze_quality(&embedding);

println!("Quality score: {:.2}", quality.quality_score);
println!("Is valid: {}", quality.is_valid);
println!("Is normalized: {}", quality.is_normalized);
println!("Sparsity: {:.1}%", quality.sparsity * 100.0);

// Detect anomalies
let report = detect_anomaly(
    &embedding,
    0.3,   // expected mean
    0.15,  // expected std dev
    1.0,   // expected L2 norm
    0.1,   // mean tolerance
    0.1,   // std dev tolerance
    0.2,   // norm tolerance
);

if report.is_anomaly {
    println!("Anomaly detected: {}", report.description);
    println!("Confidence: {:.1}%", report.confidence * 100.0);
}

// Analyze batch of vectors
let vectors = vec![
    vec![0.1, 0.2, 0.3],
    vec![0.4, 0.5, 0.6],
    vec![0.7, 0.8, 0.9],
];
let batch_stats = compute_batch_stats(&vectors);

println!("Average quality: {:.2}", batch_stats.avg_quality);
println!("Valid vectors: {}/{}", batch_stats.valid_count, batch_stats.count);

Index Diagnostics and Health Monitoring

Monitor index health and performance:

use ipfrs_semantic::{VectorIndex, diagnose_index, HealthMonitor, SearchProfiler};
use std::time::Duration;

let mut index = VectorIndex::with_defaults(128)?;

// Run diagnostics
let report = diagnose_index(&index);

println!("Health status: {:?}", report.status);
println!("Index size: {} vectors", report.size);
println!("Memory usage: ~{:.2} MB", report.memory_usage as f64 / 1e6);

for issue in &report.issues {
    println!("Issue ({:?}): {}", issue.severity, issue.description);
    if let Some(fix) = &issue.suggested_fix {
        println!("  Suggested fix: {}", fix);
    }
}

for rec in &report.recommendations {
    println!("Recommendation: {}", rec);
}

// Set up periodic health monitoring
let mut monitor = HealthMonitor::new(Duration::from_secs(60));

if monitor.should_check() {
    let report = monitor.check(&index);
    println!("Health check: {:?}", report.status);
}

// Profile search performance
let mut profiler = SearchProfiler::new();

// Simulate queries
profiler.record_query(Duration::from_millis(5));
profiler.record_query(Duration::from_millis(3));
profiler.record_query(Duration::from_millis(4));

let stats = profiler.stats();
println!("Total queries: {}", stats.total_queries);
println!("Average latency: {:?}", stats.avg_latency);
println!("QPS: {:.2}", stats.qps);

Index Optimization

Automatically tune index parameters:

use ipfrs_semantic::{analyze_optimization, OptimizationGoal, QueryOptimizer, MemoryOptimizer};
use std::time::Duration;

// Analyze and get optimization recommendations
let result = analyze_optimization(
    50_000,  // index size
    768,     // dimension
    16,      // current M
    200,     // current ef_construction
    OptimizationGoal::Balanced,
);

println!("Current quality score: {:.2}", result.current_score);
println!("Recommended M: {}", result.recommended_m);
println!("Recommended ef_construction: {}", result.recommended_ef_construction);
println!("Recommended ef_search: {}", result.recommended_ef_search);
println!("Estimated improvement: {:.1}%", result.estimated_improvement * 100.0);

for reason in &result.reasoning {
    println!("  - {}", reason);
}

// Adaptive query optimization
let mut query_optimizer = QueryOptimizer::new(
    50,                            // initial ef_search
    Duration::from_millis(10),     // target latency
);

// The optimizer adjusts ef_search based on observed latency
for _ in 0..20 {
    query_optimizer.record_query(Duration::from_millis(15));
}

println!("Optimized ef_search: {}", query_optimizer.get_ef_search());

// Memory budget optimization
let mut memory_optimizer = MemoryOptimizer::new(1024 * 1024 * 1024); // 1GB

let (m, ef_c, max_vectors) = memory_optimizer.recommend_config(768);
println!("For 1GB budget:");
println!("  Recommended M: {}", m);
println!("  Recommended ef_construction: {}", ef_c);
println!("  Max vectors: {}", max_vectors);

Re-exports

pub use hnsw::BuildHealthStats;

pub use hnsw::DistanceMetric;

pub use hnsw::IncrementalBuildStats;

pub use hnsw::ParameterRecommendation;

pub use hnsw::ParameterTuner;

pub use hnsw::RebuildStats;

pub use hnsw::SearchResult;

pub use hnsw::UseCase;

pub use hnsw::VectorIndex;

pub use router::BatchStats;

pub use router::CacheStats;

pub use router::QueryFilter;

pub use router::RouterConfig;

pub use router::RouterStats;

pub use router::SemanticRouter;

pub use hybrid::FilterStrategy;

pub use hybrid::HybridConfig;

pub use hybrid::HybridIndex;

pub use hybrid::HybridQuery;

pub use hybrid::HybridResponse;

pub use hybrid::HybridResult;

pub use hybrid::PruningStats;

pub use metadata::Metadata;

pub use metadata::MetadataFilter;

pub use metadata::MetadataStore;

pub use metadata::MetadataValue;

pub use metadata::TemporalOptions;

pub use quantization::OptimizedProductQuantizer;

pub use quantization::PQCode;

pub use quantization::ProductQuantizer;

pub use quantization::QuantizationBenchmark;

pub use quantization::QuantizationBenchmarker;

pub use quantization::QuantizationComparison;

pub use quantization::QuantizedVector;

pub use quantization::ScalarQuantizer;

pub use stats::IndexHealth;

pub use stats::IndexStats;

pub use stats::MemoryUsage;

pub use stats::PerfTimer;

pub use stats::StatsSnapshot;

pub use diskann::SearchResult as DiskANNSearchResult;

pub use diskann::CompactionStats;

pub use diskann::DiskANNConfig;

pub use diskann::DiskANNIndex;

pub use diskann::DiskANNStats;

pub use solver::LogicSolver;

pub use solver::PredicateEmbedder;

pub use solver::ProofSearch;

pub use solver::ProofTreeNode;

pub use solver::SolverConfig;

pub use solver::SolverStats;

pub use kb_query::BooleanQuery;

pub use kb_query::FilterExpr;

pub use kb_query::Query;

pub use kb_query::QueryExecutor;

pub use kb_query::QueryPattern;

pub use kb_query::QueryResult;

pub use kb_query::QueryStats;

pub use kb_query::TermPattern;

pub use kb_query::TermType;

pub use provenance::AuditLogEntry;

pub use provenance::AuditOperation;

pub use provenance::EmbeddingMetadata;

pub use provenance::EmbeddingSource;

pub use provenance::EmbeddingVersion;

pub use provenance::FeatureAttribution;

pub use provenance::ProvenanceStats;

pub use provenance::ProvenanceTracker;

pub use provenance::SearchExplanation;

pub use provenance::VersionHistory;

pub use simd::cosine_distance;

pub use simd::dot_product;

pub use simd::l2_distance;

pub use cache::AdaptiveCacheStrategy;

pub use cache::AlignedVector;

pub use cache::CacheInvalidator;

pub use cache::HotCacheStats;

pub use cache::HotEmbeddingCache;

pub use cache::InvalidationPolicy;

pub use multimodal::Modality;

pub use multimodal::ModalityAlignment;

pub use multimodal::ModalityStats;

pub use multimodal::MultiModalConfig;

pub use multimodal::MultiModalEmbedding;

pub use multimodal::MultiModalIndex;

pub use privacy::NoiseDistribution;

pub use privacy::PrivacyBudget;

pub use privacy::PrivacyBudgetStats;

pub use privacy::PrivacyMechanism;

pub use privacy::PrivateEmbedding;

pub use privacy::QueryRecord;

pub use privacy::TradeoffAnalyzer;

pub use privacy::TradeoffPoint;

pub use dynamic::DynamicIndex;

pub use dynamic::EmbeddingTransform;

pub use dynamic::ModelVersion;

pub use dynamic::OnlineUpdater;

pub use dynamic::OnlineUpdaterStats;

pub use dynamic::VersionStats;

pub use dht::DHTQuery;

pub use dht::DHTQueryResponse;

pub use dht::ReplicationStrategy;

pub use dht::SemanticDHTConfig;

pub use dht::SemanticDHTStats;

pub use dht::SemanticPeer;

pub use dht::SemanticRoutingTable;

pub use dht_node::SemanticDHTNode;

pub use dht_node::SyncStats;

pub use federated::AggregationStrategy;

pub use federated::FederatedConfig;

pub use federated::FederatedQueryExecutor;

pub use federated::FederatedQueryStats;

pub use federated::FederatedSearchResult;

pub use federated::LocalIndexAdapter;

pub use federated::QueryableIndex;

pub use reranking::ReRanker;

pub use reranking::ReRankingConfig;

pub use reranking::ReRankingStrategy;

pub use reranking::ScoreComponent;

pub use reranking::ScoredResult;

pub use analytics::AnalyticsSummary;

pub use analytics::AnalyticsTracker;

pub use analytics::DetectedPattern;

pub use analytics::QueryMetrics;

pub use analytics::QueryTimer;

pub use auto_scaling::ActionType;

pub use auto_scaling::AdvisorConfig;

pub use auto_scaling::AutoScalingAdvisor;

pub use auto_scaling::ScalingAction;

pub use auto_scaling::ScalingRecommendations;

pub use auto_scaling::TrendReport;

pub use auto_scaling::WorkloadMetrics;

pub use learned::LearnedIndex;

pub use learned::LearnedIndexStats;

pub use learned::ModelType;

pub use learned::RMIConfig;

pub use adapters::BackendConfig;

pub use adapters::BackendMigration;

pub use adapters::BackendRegistry;

pub use adapters::BackendSearchResult;

pub use adapters::BackendStats;

pub use adapters::IpfrsBackend;

pub use adapters::MigrationStats;

pub use adapters::VectorBackend;

pub use vector_quality::analyze_quality;

pub use vector_quality::compute_batch_stats;

pub use vector_quality::compute_diversity;

pub use vector_quality::compute_stats;

pub use vector_quality::cosine_similarity;

pub use vector_quality::detect_anomaly;

pub use vector_quality::find_outliers;

pub use vector_quality::AnomalyReport;

pub use vector_quality::AnomalyType;

pub use vector_quality::VectorQuality;

pub use vector_quality::VectorStats;

pub use diagnostics::diagnose_index;

pub use diagnostics::DiagnosticIssue;

pub use diagnostics::DiagnosticReport;

pub use diagnostics::HealthMonitor;

pub use diagnostics::HealthStatus;

pub use diagnostics::IssueCategory;

pub use diagnostics::IssueSeverity;

pub use diagnostics::PerformanceMetrics;

pub use diagnostics::ProfilerStats;

pub use diagnostics::SearchProfiler;

pub use optimization::analyze_optimization;

pub use optimization::MemoryOptimizer;

pub use optimization::OptimizationGoal;

pub use optimization::OptimizationResult;

pub use optimization::QueryOptimizer;

pub use utils::average_embedding;

pub use utils::create_hybrid_index_from_map;

pub use utils::health_check;

pub use utils::index_with_quality_check;

pub use utils::normalize_vector;

pub use utils::normalize_vectors;

pub use utils::validate_embeddings;

pub use utils::BatchEmbeddingStats;

pub use utils::BatchIndexResult;

pub use utils::HealthCheckResult;

pub use prod_tests::EnduranceTest;

pub use prod_tests::EnduranceTestConfig;

pub use prod_tests::EnduranceTestResults;

pub use prod_tests::StressTest;

pub use prod_tests::StressTestConfig;

pub use prod_tests::StressTestResults;

pub use regression::MetricSummary;

pub use regression::RegressionConfig;

pub use regression::RegressionDetector;

pub use regression::RegressionIssue;

pub use regression::RegressionReport;

pub use benchmark_comparison::BenchmarkResult;

pub use benchmark_comparison::BenchmarkSuite;

pub use benchmark_comparison::ComparisonReport;

pub use benchmark_comparison::IndexConfig;

pub use benchmark_comparison::ParameterSweep;

pub use migration::BatchMigration;

pub use migration::ConfigMigration;

pub use migration::DimensionMigration;

pub use migration::IndexMigration;

pub use migration::MetricMigration;

pub use migration::MigrationConfig;

pub use migration::MigrationProgress;

Modules

adapters

Vector database adapters for external integration.

analytics

Query analytics and performance tracking

auto_scaling

Auto-scaling advisor for production deployments

benchmark_comparison

Benchmark comparison utilities for evaluating different configurations

cache

Advanced caching for vector embeddings

dht

Distributed Semantic DHT

dht_node

Distributed Semantic DHT Node

diagnostics

Index diagnostics and health monitoring

diskann

DiskANN: Disk-based Approximate Nearest Neighbor Search

dynamic

Dynamic embedding updates for evolving embedding spaces

federated

Federated Query Support for Multi-Index Search

hnsw

HNSW vector index for semantic search

hybrid

Hybrid search combining vector similarity with metadata filtering

kb_query

Knowledge Base Query Language

learned

Learned index structures using ML models for data indexing.

metadata

Metadata storage and filtering for hybrid search

migration

Index migration utilities

multimodal

Multi-modal embedding support for unified semantic search across text, image, and audio.

optimization

Index optimization utilities

privacy

Differential privacy for embeddings

prod_tests

Production readiness testing utilities

provenance

Provenance Tracking for Embeddings

quantization

Vector quantization for memory-efficient storage

regression

Performance regression detection and tracking

reranking

Query result re-ranking

router

Semantic router for content discovery

simd

SIMD-optimized distance computation

solver

Logic solver for reasoning queries with semantic integration

stats

Index statistics and monitoring

utils

Utility functions and helpers for common semantic search workflows

vector_quality

Vector quality analysis and validation utilities