# OxiRS Vector Search - Python Bindings
High-performance vector search engine with SPARQL integration, now available for Python!
## Overview
OxiRS Vector Search provides state-of-the-art vector similarity search with native SPARQL integration, built in Rust for maximum performance and exposed to Python through PyO3 bindings. This enables seamless integration with the Python ML ecosystem while maintaining near-native performance.
## Features
### 🚀 **High Performance**
- **Sub-millisecond search**: <1ms average query time on 1M vectors
- **Memory efficient**: <4GB RAM for 10M 384-dim vectors
- **Rust-powered**: Native performance with Python convenience
- **GPU acceleration**: CUDA support for massive datasets
### 🧠 **Advanced AI Integration**
- **Multiple embedding strategies**: TF-IDF, sentence transformers, OpenAI, custom models
- **Cross-modal search**: Text, image, audio, and video embeddings
- **Neural architecture search**: Automatic embedding optimization
- **Real-time updates**: Live index maintenance and streaming ingestion
### 🔍 **Sophisticated Search**
- **Advanced indices**: HNSW, IVF, LSH, ball tree, and more
- **Rich similarity metrics**: Cosine, Euclidean, Pearson, Jaccard, and statistical measures
- **Hierarchical similarity**: Ontology-aware semantic similarity
- **Graph-scoped search**: Context-aware search within knowledge graphs
### 🌐 **SPARQL Integration**
- **Native SPARQL functions**: `vec:similar()`, `vec:search()`, `vec:searchIn()`
- **Federated search**: Distributed vector search across endpoints
- **Custom functions**: User-defined similarity metrics and operations
- **RDF-aware**: Native integration with knowledge graphs and ontologies
### 📊 **Analytics & Monitoring**
- **Performance analytics**: Real-time monitoring and optimization recommendations
- **Quality assessment**: Vector distribution analysis and quality scoring
- **Anomaly detection**: Automatic detection of unusual patterns
- **Comprehensive benchmarking**: Industry-standard performance evaluation
## Installation
### From PyPI (Recommended)
```bash
pip install oxirs-vec[ml] # Includes ML ecosystem integration
```
### From Source
```bash
# Install Rust and maturin
# Clone and build
git clone https://github.com/cool-japan/oxirs.git
cd oxirs/engine/oxirs-vec
maturin develop --features python,simd,parallel
```
## Quick Start
```python
import oxirs_vec
import numpy as np
# Create vector store with HNSW index
store = oxirs_vec.VectorStore(
embedding_strategy="sentence_transformer",
index_type="hnsw",
max_connections=16,
ef_construction=200
)
# Index documents
store.index_resource("doc1", "Machine learning and artificial intelligence")
store.index_resource("doc2", "Deep neural networks for computer vision")
store.index_resource("doc3", "Natural language processing with transformers")
# Search for similar content
results = store.similarity_search(
query="AI and deep learning",
limit=5,
threshold=0.3,
metric="cosine"
)
for result in results:
print(f"{result['id']}: {result['score']:.3f}")
```
## Advanced Usage
### NumPy Integration
```python
import numpy as np
# Direct vector operations
vectors = np.random.rand(1000, 256).astype(np.float32)
vector_ids = [f"vec_{i}" for i in range(1000)]
metadata = [{"category": i % 10} for i in range(1000)]
# Batch indexing
store.index_batch(vector_ids, vectors, metadata)
# Vector search
query_vector = np.random.rand(256).astype(np.float32)
results = store.vector_search(query_vector, limit=10)
```
### Analytics and Optimization
```python
# Vector analytics
analytics = oxirs_vec.VectorAnalytics()
analysis = analytics.analyze_vectors(vectors, labels)
print(f"Sparsity: {analysis['sparsity']:.3f}")
print(f"Intrinsic dimension: {analysis['intrinsic_dimension']}")
# Get optimization recommendations
recommendations = analytics.get_recommendations()
for rec in recommendations:
print(f"{rec['type']}: {rec['description']}")
print(f"Expected improvement: {rec['expected_improvement']:.1f}%")
```
### SPARQL Integration
```python
# Create SPARQL vector search interface
sparql_search = oxirs_vec.SparqlVectorSearch(store)
# Register custom functions
sparql_search.register_function(
"vec:semanticSimilarity",
arity=3,
description="Compute semantic similarity between resources"
)
# Execute SPARQL with vector extensions
query = """
SELECT ?resource ?similarity WHERE {
?resource vec:similar("machine learning", 5, 0.3) .
BIND(vec:similarity(?resource, "http://example.org/ai") AS ?similarity)
}
ORDER BY DESC(?similarity)
"""
results = sparql_search.execute_query(query)
```
### Performance Monitoring
```python
# Get detailed statistics
stats = store.get_stats()
print(f"Total vectors: {stats['total_vectors']:,}")
print(f"Memory usage: {stats['memory_usage_bytes'] / 1024**2:.1f} MB")
print(f"Average search time: {stats['avg_search_time_ms']:.2f} ms")
# Performance optimization
store.optimize() # Rebuild index for optimal performance
```
## Integration Examples
### Scikit-learn Integration
```python
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.datasets import fetch_20newsgroups
# Load dataset
newsgroups = fetch_20newsgroups(subset='train', categories=['sci.med', 'sci.space'])
# Create TF-IDF vectors
vectorizer = TfidfVectorizer(max_features=1000)
tfidf_vectors = vectorizer.fit_transform(newsgroups.data)
# Convert to dense format for OxiRS
dense_vectors = tfidf_vectors.toarray().astype(np.float32)
# Index in OxiRS
store = oxirs_vec.VectorStore(embedding_strategy="tf_idf", index_type="hnsw")
doc_ids = [f"doc_{i}" for i in range(len(dense_vectors))]
store.index_batch(doc_ids, dense_vectors)
```
### Pandas Integration
```python
import pandas as pd
# Create DataFrame with results
results = store.similarity_search("space exploration", limit=20)
df = pd.DataFrame(results)
# Analyze results
print(df['score'].describe())
print(df.groupby('metadata.category')['score'].mean())
# Export results
df.to_csv('search_results.csv', index=False)
```
### Jupyter Notebook Visualization
```python
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.decomposition import PCA
# Get vectors for visualization
vector_data = []
for doc_id in store.get_vector_ids()[:100]:
vector = store.get_vector(doc_id)
if vector is not None:
vector_data.append(vector)
vectors_array = np.array(vector_data)
# Dimensionality reduction
pca = PCA(n_components=2)
vectors_2d = pca.fit_transform(vectors_array)
# Create visualization
plt.figure(figsize=(10, 8))
plt.scatter(vectors_2d[:, 0], vectors_2d[:, 1], alpha=0.6)
plt.title('Vector Space Visualization (PCA)')
plt.xlabel('Principal Component 1')
plt.ylabel('Principal Component 2')
plt.show()
```
## Performance Benchmarks
| Index 1M vectors (256D) | 45s | 1.2GB | HNSW index |
| Search 1M vectors | <1ms | - | 10-NN search |
| Batch insert 10K vectors | 2.1s | - | Real-time updates |
| Index optimization | 12s | - | Background process |
### Comparison with Alternatives
| **OxiRS** | **0.8** | **1.2** | **SPARQL, Multi-modal, Analytics** |
| FAISS | 1.2 | 1.8 | GPU acceleration |
| Annoy | 2.1 | 1.1 | Memory mapping |
| Weaviate | 5.4 | 2.3 | GraphQL API |
## Configuration Options
### Vector Store Configuration
```python
store = oxirs_vec.VectorStore(
# Embedding strategy
embedding_strategy="sentence_transformer", # or "tf_idf", "openai", "custom"
# Index configuration
index_type="hnsw", # or "memory", "ivf", "lsh"
max_connections=16, # HNSW parameter
ef_construction=200, # HNSW build quality
# Performance tuning
cache_size=1000000, # Vector cache size
batch_size=1000, # Batch processing size
parallel_threads=8, # Parallel processing
# GPU acceleration
use_gpu=True,
gpu_device_ids=[0, 1], # Multi-GPU support
)
```
### Analytics Configuration
```python
analytics = oxirs_vec.VectorAnalytics()
analytics.configure(
sample_size=10000, # Analysis sample size
confidence_level=0.95, # Statistical confidence
anomaly_sensitivity=0.8, # Anomaly detection threshold
recommendation_priority="performance" # or "memory", "quality"
)
```
## Error Handling
```python
try:
results = store.similarity_search("query", limit=10)
except oxirs_vec.VectorSearchError as e:
print(f"Search error: {e}")
except oxirs_vec.EmbeddingError as e:
print(f"Embedding error: {e}")
except oxirs_vec.IndexError as e:
print(f"Index error: {e}")
```
## Best Practices
### 1. **Index Selection**
- Use **HNSW** for high-quality approximate search
- Use **Memory** for exact search on small datasets
- Use **IVF** for very large datasets with compression
- Use **LSH** for fast approximate search
### 2. **Performance Optimization**
- **Normalize vectors** before indexing for consistent similarity scores
- **Batch operations** when possible for better throughput
- **Optimize index** periodically for best search performance
- **Use GPU acceleration** for large datasets (>1M vectors)
### 3. **Memory Management**
- Monitor memory usage with `store.get_stats()`
- Use vector compression for memory-constrained environments
- Consider memory-mapped indices for very large datasets
- Implement cache warming for predictable query patterns
### 4. **Quality Assurance**
- Use analytics to monitor vector quality and distribution
- Implement A/B testing for embedding strategies
- Monitor search relevance and user feedback
- Regular benchmarking against ground truth datasets
## API Reference
### Classes
- **`VectorStore`**: Main vector storage and search interface
- **`VectorAnalytics`**: Vector quality analysis and optimization
- **`SparqlVectorSearch`**: SPARQL integration for semantic queries
### Functions
- **`compute_similarity(vec1, vec2, metric)`**: Compute similarity between vectors
- **`normalize_vector(vector)`**: Normalize vector to unit length
- **`batch_normalize(vectors)`**: Normalize multiple vectors efficiently
### Exceptions
- **`VectorSearchError`**: General vector search errors
- **`EmbeddingError`**: Embedding generation errors
- **`IndexError`**: Index operation errors
## Contributing
We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details.
### Development Setup
```bash
# Install development dependencies
pip install oxirs-vec[dev]
# Run tests
pytest tests/
# Format code
black .
isort .
# Type checking
mypy .
```
## License
This project is dual-licensed under MIT and Apache 2.0 licenses. See [LICENSE](LICENSE) for details.
## Support
- **Documentation**: [https://docs.oxirs.dev](https://docs.oxirs.dev)
- **Issues**: [GitHub Issues](https://github.com/cool-japan/oxirs/issues)
- **Community**: [Discord](https://discord.gg/oxirs)
- **Email**: support@oxirs.dev
---
**Built with ❤️ using Rust and PyO3 for maximum performance and Python compatibility.**