# Aurora DB
> A lightweight, real-time embedded database designed for modern applications
[](https://crates.io/crates/aurora-db)
[](https://docs.rs/aurora-db)
[](https://opensource.org/licenses/MIT)
## Why Aurora Exists
Most embedded databases force you to choose: either **simple key-value storage** with manual indexing and caching, or **heavy SQL databases** with complex setup. I wanted something different.
**Aurora was born from a simple need**: a database that **just works** for building real-time applications. No external services, no complex configuration, no choosing between twenty different storage engines. Just install it, open it, and start building.
### The Philosophy
**Real-time by default.** Data changes should propagate instantly. Your UI shouldn't poll—it should react. Background jobs should process reliably without external queues. This isn't optional; it's how modern applications work.
**Lightweight without compromise.** Embedded doesn't mean primitive. Aurora handles indexing, caching, pub/sub, reactive queries, and background workers—all built-in. You get production-grade features without the operational overhead.
**Performance through intelligence.** The hybrid hot/cold architecture wasn't an accident. Frequently accessed data lives in memory (200K ops/sec), everything else persists to disk (10K ops/sec). Schema-aware selective indexing means I only index what you actually query. Smart defaults, intelligent caching, zero configuration to get started.
### The Design
Every feature in Aurora solves a real problem:
- **PubSub** → Because polling databases is wasteful
- **Reactive Queries** → Because UIs should update automatically
- **Durable Workers** → Because background jobs shouldn't need Redis
- **Computed Fields** → Because derived data shouldn't duplicate storage
- **Write Buffering** → Because high-throughput writes shouldn't kill performance
- **Schema-Aware Indexing** → Because indexing every field is insane
This wasn't scope creep—it was intentional. Building real-time apps requires all these pieces, and they should work together seamlessly.
### What's Next
The current bottleneck is in-memory index management. I'm currently looking into how **DiceDB**, **Redis**, and **RocksDB** handle high-performance in-memory structures, exploring:
- Lock-free index structures inspired by DiceDB's concurrent patterns
- Memory-efficient data structures from Redis's designs
- RocksDB's LSM-tree approach for better write amplification
- Smarter cache eviction using access pattern analysis
---
## Table of Contents
### Getting Started
- [Schema Management](./src/docs/schema.md) - Define collections and field types
- [CRUD Operations](./src/docs/crud.md) - Create, read, update, and delete documents
- [Querying](./src/docs/querying.md) - Filter, sort, and retrieve data
### Advanced Features
- [PubSub System](./src/docs/pubsub.md) - Real-time change notifications
- [Reactive Queries](./src/docs/reactive.md) - Auto-updating query results
- [Durable Workers](./src/docs/workers.md) - Background job processing
- [Computed Fields](./src/docs/computed-fields.md) - Derived values from existing data
### Optimization
- [Performance Guide](./src/docs/performance.md) - Tuning, caching, and best practices
## Quick Start
### Installation
```toml
[dependencies]
aurora-db = "0.3.0"
```
### Basic Usage
```rust
use aurora_db::{Aurora, FieldType, Value};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
// Open database
let db = Aurora::open("myapp.db")?;
// Create a collection
db.new_collection("users", vec![
("name", FieldType::String, false),
("email", FieldType::String, true), // unique
("age", FieldType::Int, false),
])?;
// Insert a document
let user_id = db.insert_into("users", vec![
("name", Value::String("Alice".to_string())),
("email", Value::String("alice@example.com".to_string())),
("age", Value::Int(30)),
]).await?;
// Query documents
let users = db.query("users")
.filter(|f| f.gt("age", 25))
.collect()
.await?;
println!("Found {} users over 25", users.len());
Ok(())
}
```
## Feature Overview
### Collections & Schema
Define structured collections with type-safe fields and unique constraints.
```rust
db.new_collection("products", vec![
("sku", FieldType::String, true), // unique
("name", FieldType::String, false),
("price", FieldType::Float, false),
("in_stock", FieldType::Bool, false),
])?;
```
[Learn more →](./src/docs/schema.md)
### Powerful Querying
Filter, sort, paginate, and search your data with an intuitive API.
```rust
let products = db.query("products")
.filter(|f| f.eq("in_stock", true) && f.lt("price", 100.0))
.order_by("price", true)
.limit(10)
.collect()
.await?;
```
[Learn more →](./src/docs/querying.md)
### Real-time Updates
Subscribe to data changes with the PubSub system.
```rust
let mut subscription = db.subscribe("orders", None).await?;
while let Ok(event) = subscription.recv().await {
println!("Order changed: {:?}", event);
}
```
[Learn more →](./src/docs/pubsub.md)
### ⚡ Reactive Queries
Automatically update query results when data changes.
```rust
let active_users = db.reactive_query("users")
.filter(|f| f.eq("online", true))
.build()
.await?;
// Results automatically update when users go online/offline
let current_users = active_users.get_results().await?;
```
[Learn more →](./src/docs/reactive.md)
### Background Jobs
Process tasks asynchronously with automatic retries.
```rust
// Define a job handler
struct EmailHandler;
#[async_trait]
impl JobHandler for EmailHandler {
async fn handle(&self, job: &Job) -> JobResult {
send_email(&job.payload).await?;
Ok(())
}
}
// Start worker
let mut executor = WorkerExecutor::new(db.clone(), 4);
executor.register_handler("send_email", Box::new(EmailHandler));
executor.start().await?;
// Enqueue job
db.enqueue_job("send_email", payload, None, Priority::High).await?;
```
[Learn more →](./src/docs/workers.md)
### Computed Fields
Derive values automatically from document data.
```rust
let mut registry = ComputedFieldsRegistry::new();
// Full name from first + last
registry.register(
"users",
"full_name",
Expression::Concat {
fields: vec!["first_name".to_string(), "last_name".to_string()],
separator: " ".to_string(),
},
);
db.set_computed_fields(registry);
```
[Learn more →](./src/docs/computed-fields.md)
### High Performance
Optimized architecture with hot/cold storage and intelligent caching.
| Hot Cache Read | ~200K ops/sec |
| Indexed Insert | ~17K ops/sec |
| Indexed Query | ~50K ops/sec |
[Learn more →](./src/docs/performance.md)
## Architecture
Aurora uses a hybrid storage architecture:
```
┌─────────────────────────────────────────┐
│ Application Layer │
├─────────────────────────────────────────┤
│ PubSub │ Reactive │ Workers │ Computed │
├─────────────────────────────────────────┤
│ Query Engine │
├─────────────────────────────────────────┤
│ Hot Cache (In-Memory) │ Indices │
├──────────────────────────┴───────────────┤
│ Cold Storage (Sled - On Disk) │
└─────────────────────────────────────────┘
```
### Storage Layers
- **Hot Cache**: In-memory cache for frequently accessed data (~200K ops/sec)
- **Cold Storage**: Persistent disk storage using Sled (~10K ops/sec)
- **Indices**: In-memory indices for fast lookups (O(1) complexity)
### Advanced Features
- **PubSub**: Real-time change notifications
- **Reactive Queries**: Auto-updating query results
- **Durable Workers**: Background job processing with retries
- **Computed Fields**: Virtual fields calculated at query time
## Configuration
Customize Aurora's behavior with `AuroraConfig`:
```rust
use aurora_db::{Aurora, AuroraConfig, EvictionPolicy};
let config = AuroraConfig {
// Cache settings
hot_cache_size_mb: 256,
cold_cache_capacity_mb: 512,
eviction_policy: EvictionPolicy::LRU,
// Write buffering
enable_write_buffering: true,
write_buffer_size: 1000,
write_buffer_flush_interval_ms: 100,
// Index settings
max_index_entries_per_field: 100000,
// Cleanup intervals
hot_cache_cleanup_interval_secs: 300,
cold_flush_interval_ms: 1000,
..Default::default()
};
let db = Aurora::open_with_config("mydb.db", config)?;
```
[Learn more →](./src/docs/performance.md#configuration-tuning)
## Use Cases
### Real-time Applications
- Chat applications with live message updates
- Collaborative editing with reactive queries
- Live dashboards with auto-updating metrics
- Gaming leaderboards
### Background Processing
- Email delivery with retry logic
- Image processing pipelines
- Data exports and reports
- Webhook delivery
### Data-Intensive Apps
- E-commerce with inventory tracking
- Analytics with computed metrics
- Content management systems
- IoT data collection
## Best Practices
### 1. Index Strategy
```rust
// DO: Index frequently queried fields
db.create_index("users", "email").await?;
db.create_index("orders", "customer_id").await?;
// DON'T: Index every field
// Only index what you query
```
### 2. Query Optimization
```rust
// DO: Use projections and limits
let users = db.query("users")
.select(vec!["id", "name"]) // Only needed fields
.limit(100) // Bounded result set
.collect()
.await?;
// DON'T: Unbounded queries
let users = db.query("users").collect().await?;
```
### 3. Write Performance
```rust
// DO: Use batch operations
db.batch_insert("logs", log_entries).await?;
// DON'T: Individual inserts in loops
for entry in log_entries {
db.insert_into("logs", entry).await?;
}
```
### 4. Real-time Updates
```rust
// DO: Use reactive queries for UI state
let online_users = ReactiveState::new(
db.clone(),
"users",
|f| f.eq("online", true)
).await?;
// DON'T: Poll the database
loop {
let users = db.query("users")
.filter(|f| f.eq("online", true))
.collect()
.await?;
tokio::time::sleep(Duration::from_secs(1)).await;
}
```
## Examples
Check out the `examples/` directory for complete examples:
- `reactive_demo.rs` - Reactive queries in action
- `contains_query_demo.rs` - Advanced querying
- `indexing_performance.rs` - Performance testing
- `index_demo.rs` - Index usage patterns
Run an example:
```bash
cargo run --example reactive_demo
```
## API Reference
For detailed API documentation:
```bash
cargo doc --open
```
## Community & Support
- **GitHub**: [github.com/bethel-nz/aurora](https://github.com/bethel-nz/aurora)
- **Issues**: [github.com/bethel-nz/aurora/issues](https://github.com/bethel-nz/aurora/issues)
- **Crates.io**: [crates.io/crates/aurora-db](https://crates.io/crates/aurora-db)
## Version History
### v0.3.0 (Current)
- Schema-aware selective indexing (3-5x faster inserts)
- Schema caching for reduced overhead
- Optimized benchmark suite
- Fixed compiler warnings
- Comprehensive documentation
### v0.2.1
- PubSub system for real-time updates
- Reactive queries
- Durable workers with retry logic
- Computed fields
- Write buffer optimization
- Multiple cache eviction policies
### v0.1.x
- Initial release
- Basic CRUD operations
- Collections and schema
- Query system
- Transactions
## License
MIT License - see LICENSE file for details
---
**Ready to build?** Start with the [Schema Management](./src/docs/schema.md) guide!