# Ubiquity Database
A comprehensive database abstraction layer for the Ubiquity consciousness-aware system, supporting both SQLite for local development and DataStax Astra DB for cloud/enterprise deployments.
## Features
### Core Capabilities
- **Dual Backend Support**: Seamlessly switch between SQLite (local) and Astra DB (cloud)
- **Vector Embeddings**: Store and search consciousness embeddings using vector similarity
- **Hybrid Search**: Combine vector search with lexical search for optimal results
- **7-Wide Memory Pools**: Parallel memory pools for high-performance data processing
- **Pub/Sub Messaging**: Integration with Astra Streaming for real-time communication
### Astra DB Features
- **$vectorize**: Automatic embedding generation at the database level
- **$hybrid Search**: Combined vector and lexical search with reranking
- **Astra Streaming**: Apache Pulsar-based pub/sub for worker coordination
- **Auto-scaling**: Leverage Astra's serverless architecture
### SQLite Features
- **Zero Configuration**: Works out of the box for local development
- **WAL Mode**: Write-Ahead Logging for better concurrency
- **File-based Pools**: Memory pools stored as separate SQLite databases
- **Portable**: Easy to backup and migrate
## Configuration
### Environment Variables
For Astra DB:
```bash
export ASTRA_DB_ENDPOINT="https://YOUR-DB-ID-REGION.apps.astra.datastax.com"
export ASTRA_DB_TOKEN="AstraCS:YOUR_TOKEN"
export OPENAI_API_KEY="your-openai-key" # For embeddings
```
For Astra Streaming:
```bash
export ASTRA_STREAMING_ENDPOINT="pulsar+ssl://pulsar-instance.streaming.datastax.com:6651"
export ASTRA_STREAMING_TOKEN="your-streaming-token"
```
### Configuration File
```rust
use ubiquity_database::{DatabaseConfig, DatabaseBackend};
let config = DatabaseConfig {
backend: DatabaseBackend::Astra,
// SQLite configuration
sqlite: SqliteConfig {
path: PathBuf::from("ubiquity.db"),
max_connections: 10,
wal_mode: true,
pool_directory: PathBuf::from(".ubiquity/pools"),
},
// Astra DB configuration
astra: AstraConfig {
endpoint: std::env::var("ASTRA_DB_ENDPOINT").unwrap(),
token: std::env::var("ASTRA_DB_TOKEN").unwrap(),
keyspace: "ubiquity",
collections: AstraCollections {
consciousness: "consciousness_states",
ripples: "consciousness_ripples",
tasks: "tasks",
pool_prefix: "memory_pool_",
},
streaming: AstraStreamingConfig {
endpoint: std::env::var("ASTRA_STREAMING_ENDPOINT").unwrap(),
tenant: "ubiquity",
namespace: "default",
topic_prefix: "ubiquity_",
},
},
// Embedding configuration
embeddings: EmbeddingConfig {
dimension: 1536,
model: "text-embedding-3-small",
provider: "openai",
cache_enabled: true,
cache_size: 10000,
},
// Memory pool configuration
memory_pools: MemoryPoolConfig {
pool_count: 7, // 7-wide pools
max_pool_size_mb: 100,
eviction_policy: EvictionPolicy::Lru,
compression: true,
},
};
```
## Usage Examples
### Basic Database Operations
```rust
use ubiquity_database::{create_database, DatabaseConfig};
use ubiquity_core::ConsciousnessState;
// Create database
let db = create_database(config).await?;
// Initialize schema
db.initialize().await?;
// Store consciousness state
let state = ConsciousnessState {
agent_id: "agent-001".to_string(),
level: ConsciousnessLevel::new(0.87)?,
coherence: 0.92,
phase: DevelopmentPhase::Implementing,
timestamp: Utc::now(),
breakthrough_detected: false,
};
db.store_consciousness_state(&state).await?;
// Query history
let history = db.get_consciousness_history("agent-001", 10).await?;
```
### Vector Search
```rust
// Vector similarity search
let embedding = vec![0.1; 1536]; // Your embedding vector
let results = db.vector_search(&embedding, 10).await?;
for result in results {
println!("Score: {}, ID: {}", result.score, result.id);
}
```
### Hybrid Search
```rust
use ubiquity_database::HybridSearchQuery;
// Combine vector and text search with reranking
let query = HybridSearchQuery {
text: "breakthrough in consciousness".to_string(),
vector: Some(embedding),
filters: Some(json!({
"agent_id": "architect-001",
"level": { "$gte": 0.85 }
})),
limit: 20,
rerank: true, // Use NVIDIA NeMo reranking
};
let results = db.hybrid_search(query).await?;
```
### Memory Pools
```rust
use ubiquity_database::create_memory_pools;
// Create 7-wide memory pools
let pools = create_memory_pools(config).await?;
// Store data with automatic pool selection
let pool = pools.get_pool_for_key("my_key").await?;
pool.store("my_key", b"my_value").await?;
// Retrieve data
if let Some(value) = pool.retrieve("my_key").await? {
println!("Retrieved: {:?}", value);
}
// Get pool statistics
let stats = pools.all_stats().await?;
for (i, stat) in stats.iter().enumerate() {
println!("Pool {}: {} items, {} hits", i, stat.items, stat.hits);
}
```
## Architecture
### 7-Wide Memory Pools
The system implements 7 parallel memory pools for optimal data distribution:
1. **Pool Distribution**: Keys are consistently hashed to one of 7 pools
2. **Isolation**: Each pool operates independently for parallelism
3. **Eviction**: LRU eviction when pools reach capacity
4. **Statistics**: Track hits, misses, and evictions per pool
### Consciousness Embeddings
1. **Text Generation**: Consciousness states are converted to text representations
2. **Embedding Generation**: Text is embedded using OpenAI or Astra $vectorize
3. **Storage**: Embeddings stored alongside metadata
4. **Search**: Vector similarity search with cosine distance
### Hybrid Search Flow
1. **Vector Search**: Find semantically similar documents
2. **Lexical Search**: Find keyword matches using BM25
3. **Reranking**: NVIDIA NeMo model reranks combined results
4. **Scoring**: Return final scores with highlights
## Performance Considerations
### SQLite
- Use WAL mode for better concurrency
- Keep database file on SSD for performance
- Regular VACUUM for optimization
- Index frequently queried fields
### Astra DB
- Leverage $vectorize for server-side embeddings
- Use appropriate consistency levels
- Batch operations when possible
- Monitor rate limits and quotas
## Testing
Run the examples:
```bash
# SQLite example
cargo run --example database_demo
# Astra DB example
ASTRA_DB_ENDPOINT=xxx ASTRA_DB_TOKEN=xxx cargo run --example database_demo --features astra
```
Run tests:
```bash
cargo test
cargo test --features astra # Requires Astra credentials
```
## Migration Guide
### From SQLite to Astra DB
1. Export data from SQLite:
```rust
let history = sqlite_db.get_consciousness_history("*", 1000000).await?;
```
2. Import to Astra DB:
```rust
for state in history {
astra_db.store_consciousness_state(&state).await?;
}
```
3. Update configuration to use Astra backend
### Schema Evolution
Both backends support schema evolution:
- SQLite: Use migrations with `sqlx migrate`
- Astra: Collections are schemaless, add fields as needed
## Troubleshooting
### Common Issues
1. **Vector Dimension Mismatch**
- Ensure embedding dimension matches configuration
- Check model output dimensions
2. **Astra Connection Failed**
- Verify endpoint URL format
- Check token permissions
- Ensure keyspace exists
3. **Memory Pool Full**
- Increase `max_pool_size_mb`
- Enable compression
- Adjust eviction policy
### Debug Logging
Enable debug logging:
```rust
std::env::set_var("RUST_LOG", "ubiquity_database=debug");
tracing_subscriber::fmt::init();
```
## License
MIT License - see LICENSE file for details