# LuckDB: A Lightweight JSON Document Database in Rust
[](https://crates.io/crates/luckdb)
[](https://gitlab.com/andrew_ryan/luckdb)
[](https://gitlab.com/andrew_ryan/luckdb/-/raw/master/LICENSE)
LuckDB is a lightweight, in-memory JSON document database written in Rust, inspired by MongoDB. It provides a simple yet powerful API for storing, querying, and manipulating JSON documents with support for indexing, aggregation, and persistence.
## Table of Contents
- [Features](#features)
- [Installation](#installation)
- [Quick Start](#quick-start)
- [API Reference](#api-reference)
- [Client](#client)
- [Database](#database)
- [Collection](#collection)
- [Documents](#documents)
- [Queries](#queries)
- [Updates](#updates)
- [Indexes](#indexes)
- [Aggregation](#aggregation)
- [Examples](#examples)
- [Basic CRUD Operations](#basic-crud-operations)
- [Advanced Querying](#advanced-querying)
- [Indexing](#indexing)
- [Aggregation Pipeline](#aggregation-pipeline)
- [Persistence](#persistence)
- [Client-Server Mode](#client-server-mode)
- [License](#license)
## Features
- **Document Storage**: Store JSON documents with automatic ID generation
- **Rich Querying**: Support for complex queries with multiple operators
- **Indexing**: Create indexes for improved query performance
- **Aggregation Pipeline**: Transform and analyze data with aggregation stages
- **Update Operations**: Flexible document updates with various operators
- **Persistence**: Save and load data from disk
- **Client-Server Mode**: Networked access to the database
- **Bulk Operations**: Efficient bulk write operations
- **Geospatial Queries**: Basic support for geospatial queries
## Installation
Add LuckDB to your `Cargo.toml`:
```toml
[dependencies]
luckdb = "0.1.0"
```
## Quick Start
```rust
use luckdb::{Client, DocId, Query, UpdateDocument};
use serde_json::json;
fn main() -> luckdb::Result<()> {
// Create a new client
let mut client = Client::new();
// Get a database
let db = client.db("mydb");
// Get a collection
let collection = db.collection("users");
// Insert a document
let doc = json!({
"name": "Alice",
"age": 30,
"city": "New York",
"interests": ["reading", "hiking"]
});
let id = collection.insert(doc)?;
println!("Inserted document with ID: {}", id);
// Query the collection
let query = Query::new().eq("name", "Alice".into());
let results = collection.find(query, None)?;
for (id, doc) in results {
println!("Found document {}: {}", id, doc);
}
Ok(())
}
```
## API Reference
### Client
The `Client` is the entry point to LuckDB. It manages multiple databases.
```rust
let mut client = Client::new();
// With storage path for persistence
let mut client = Client::with_storage_path("mongodb://localhost", "./data");
```
#### Methods
- `db(&mut self, name: &str) -> &mut Database`: Get or create a database
- `list_database_names(&self) -> Vec<String>`: List all databases
- `drop_database(&mut self, name: &str) -> Result<()>`: Drop a database
- `save(&self) -> Result<()>`: Save all data to disk
- `load(&mut self) -> Result<()>`: Load data from disk
### Database
A `Database` contains multiple collections.
```rust
let db = client.db("mydb");
```
#### Methods
- `collection(&mut self, name: &str) -> &mut Collection`: Get or create a collection
- `list_collection_names(&self) -> Vec<String>`: List all collections
- `create_collection(&mut self, name: &str, options: Option<CreateCollectionOptions>) -> Result<()>`: Create a collection with options
- `drop_collection(&mut self, name: &str) -> Result<()>`: Drop a collection
- `stats(&self) -> Result<DatabaseStats>`: Get database statistics
- `run_command(&mut self, command: &Document) -> Result<Document>`: Run a database command
### Collection
A `Collection` stores JSON documents.
```rust
let collection = db.collection("users");
```
#### Methods
- `insert(&mut self, doc: Document) -> Result<DocId>`: Insert a document
- `insert_many(&mut self, docs: Vec<Document>) -> Result<Vec<DocId>>`: Insert multiple documents
- `find(&self, query: Query, options: Option<FindOptions>) -> Result<Vec<(DocId, Document)>>`: Find documents matching a query
- `find_one(&self, query: Query, options: Option<FindOptions>) -> Result<(DocId, Document)>`: Find a single document
- `update_one(&mut self, query: Query, update: UpdateDocument, upsert: bool) -> Result<usize>`: Update the first matching document
- `update_many(&mut self, query: Query, update: UpdateDocument) -> Result<usize>`: Update all matching documents
- `replace_one(&mut self, query: Query, replacement: Document, upsert: bool) -> Result<usize>`: Replace the first matching document
- `delete_one(&mut self, query: Query) -> Result<usize>`: Delete the first matching document
- `delete_many(&mut self, query: Query) -> Result<usize>`: Delete all matching documents
- `count_documents(&self, query: Query) -> Result<usize>`: Count documents matching a query
- `create_index(&mut self, index: Index) -> Result<()>`: Create an index
- `drop_index(&mut self, name: &str) -> Result<()>`: Drop an index
- `list_indexes(&self) -> Result<Vec<Index>>`: List all indexes
- `aggregate(&self, pipeline: Vec<AggregationStage>) -> Result<Vec<Document>>`: Run an aggregation pipeline
- `distinct(&self, field: &str, query: Option<Query>) -> Result<Vec<Value>>`: Get distinct values for a field
- `bulk_write(&mut self, operations: Vec<BulkWriteOperation>, options: Option<BulkWriteOptions>) -> Result<BulkWriteResult>`: Execute bulk write operations
### Documents
Documents are JSON values represented by `serde_json::Value`.
```rust
use serde_json::json;
let doc = json!({
"name": "Alice",
"age": 30,
"city": "New York",
"interests": ["reading", "hiking"]
});
```
Each document automatically gets an `_id` field of type `DocId` when inserted.
### Queries
Queries are built using the `Query` struct and its methods.
```rust
use luckdb::{Query, Value};
use serde_json::json;
// Simple equality query
let query = Query::new().eq("name", "Alice".into());
// Complex query with multiple conditions
let query = Query::new()
.eq("city", "New York".into())
.gt("age", json!(25))
.in_("interests", vec!["reading".into(), "hiking".into()]);
// Using logical operators
let query1 = Query::new().eq("city", "New York".into());
let query2 = Query::new().eq("city", "San Francisco".into());
let query = Query::new().or(vec![query1, query2]);
```
#### Query Operators
- `eq(key, value)`: Field equals value
- `ne(key, value)`: Field not equal to value
- `gt(key, value)`: Field greater than value
- `gte(key, value)`: Field greater than or equal to value
- `lt(key, value)`: Field less than value
- `lte(key, value)`: Field less than or equal to value
- `in_(key, values)`: Field in array of values
- `nin(key, values)`: Field not in array of values
- `exists(key, exists)`: Field exists (or not)
- `regex(key, pattern)`: Field matches regex pattern
- `and(queries)`: Logical AND of queries
- `or(queries)`: Logical OR of queries
- `nor(queries)`: Logical NOR of queries
- `not(query)`: Logical NOT of query
- `all(key, values)`: Array contains all values
- `elem_match(key, query)`: Array element matches query
- `size(key, size)`: Array has specified size
- `near(key, point, max_distance)`: Geospatial near query
- `within(key, shape)`: Geospatial within query
- `intersects(key, shape)`: Geospatial intersects query
### Updates
Update operations are built using the `UpdateDocument` struct.
```rust
use luckdb::UpdateDocument;
use serde_json::json;
let update = UpdateDocument::new()
.set("status", "active".into())
.inc("login_count", json!(1))
.push("tags", "premium".into());
```
#### Update Operators
- `set(key, value)`: Set field to value
- `unset(key)`: Remove field
- `inc(key, value)`: Increment field by value
- `mul(key, value)`: Multiply field by value
- `rename(old_key, new_key)`: Rename field
- `set_on_insert(key, value)`: Set field on insert
- `min(key, value)`: Set field to minimum of current and value
- `max(key, value)`: Set field to maximum of current and value
- `current_date(key, type_spec)`: Set field to current date
- `push(key, value)`: Push value to array field
- `push_all(key, values)`: Push all values to array field
- `add_to_set(key, value)`: Add value to array if not present
- `pop(key, pos)`: Remove first or last element of array
- `pull(key, condition)`: Remove elements matching condition
- `pull_all(key, values)`: Remove all specified values from array
- `bit(key, operation)`: Bitwise operation
### Indexes
Indexes improve query performance.
```rust
use luckdb::{Index, IndexType};
// Create a simple index
let index = Index::new("name_index".to_string(), vec![("name".to_string(), IndexType::Ascending)]);
collection.create_index(index)?;
// Create a compound index
let index = Index::new("compound_index".to_string(), vec![
("city".to_string(), IndexType::Ascending),
("age".to_string(), IndexType::Descending)
]);
collection.create_index(index)?;
// Create a unique index
let index = Index::new("email_index".to_string(), vec![("email".to_string(), IndexType::Ascending)])
.unique(true);
collection.create_index(index)?;
```
### Aggregation
Aggregation pipelines transform and analyze data.
```rust
use luckdb::{AggregationStage, GroupOperation, GroupId, SortOrder};
use serde_json::json;
let pipeline = vec![
// Match documents
AggregationStage::Match(Query::new().eq("status", "active".into())),
// Group by city and count
AggregationStage::Group(GroupSpecification {
id: GroupId::Field("city".to_string()),
operations: {
let mut ops = std::collections::HashMap::new();
ops.insert("count".to_string(), GroupOperation::Sum(json!(1)));
ops.insert("avg_age".to_string(), GroupOperation::Avg("$age".into()));
ops
},
}),
// Sort by count
AggregationStage::Sort(vec![("count".to_string(), SortOrder::Descending)]),
// Limit results
AggregationStage::Limit(10),
];
let results = collection.aggregate(pipeline)?;
```
## Examples
### Basic CRUD Operations
```rust
use luckdb::{Client, Query, UpdateDocument};
use serde_json::json;
fn main() -> luckdb::Result<()> {
let mut client = Client::new();
let db = client.db("test");
let collection = db.collection("users");
// Create
let doc = json!({
"name": "Alice",
"age": 30,
"city": "New York",
"interests": ["reading", "hiking"]
});
let id = collection.insert(doc)?;
println!("Created document with ID: {}", id);
// Read
let query = Query::new().eq("name", "Alice".into());
let results = collection.find(query, None)?;
for (id, doc) in results {
println!("Found document {}: {}", id, doc);
}
// Update
let update = UpdateDocument::new()
.set("age", json!(31))
.push("interests", "travel".into());
let count = collection.update_one(Query::new().eq("name", "Alice".into()), update, false)?;
println!("Updated {} documents", count);
// Delete
let count = collection.delete_one(Query::new().eq("name", "Alice".into()))?;
println!("Deleted {} documents", count);
Ok(())
}
```
### Advanced Querying
```rust
use luckdb::{Query, Value};
use serde_json::json;
fn main() -> luckdb::Result<()> {
let mut client = Client::new();
let db = client.db("test");
let collection = db.collection("users");
// Insert some test data
collection.insert(json!({"name": "Alice", "age": 30, "city": "New York", "active": true}))?;
collection.insert(json!({"name": "Bob", "age": 25, "city": "San Francisco", "active": false}))?;
collection.insert(json!({"name": "Charlie", "age": 35, "city": "New York", "active": true}))?;
collection.insert(json!({"name": "David", "age": 40, "city": "Chicago", "active": true}))?;
// Find active users in New York
let query = Query::new()
.eq("city", "New York".into())
.eq("active", Value::Bool(true));
let results = collection.find(query, None)?;
println!("Active users in New York:");
for (id, doc) in results {
println!(" {}: {}", id, doc);
}
// Find users older than 30 or inactive users
let query1 = Query::new().gt("age", json!(30));
let query2 = Query::new().eq("active", Value::Bool(false));
let query = Query::new().or(vec![query1, query2]);
let results = collection.find(query, None)?;
println!("Users older than 30 or inactive:");
for (id, doc) in results {
println!(" {}: {}", id, doc);
}
// Find users with specific interests
collection.insert(json!({
"name": "Eve",
"age": 28,
"city": "Boston",
"interests": ["reading", "coding", "hiking"]
}))?;
let query = Query::new().all("interests", vec!["reading".into(), "hiking".into()]);
let results = collection.find(query, None)?;
println!("Users with both reading and hiking interests:");
for (id, doc) in results {
println!(" {}: {}", id, doc);
}
Ok(())
}
```
### Indexing
```rust
use luckdb::{Client, Index, IndexType, Query};
use serde_json::json;
fn main() -> luckdb::Result<()> {
let mut client = Client::new();
let db = client.db("test");
let collection = db.collection("users");
// Insert test data
for i in 0..1000 {
collection.insert(json!({
"name": format!("User {}", i),
"age": i % 50 + 20,
"city": ["New York", "San Francisco", "Chicago", "Boston"][i % 4],
"active": i % 3 != 0
}))?;
}
// Create indexes
let name_index = Index::new("name_index".to_string(), vec![("name".to_string(), IndexType::Ascending)]);
collection.create_index(name_index)?;
let city_age_index = Index::new("city_age_index".to_string(), vec![
("city".to_string(), IndexType::Ascending),
("age".to_string(), IndexType::Descending)
]);
collection.create_index(city_age_index)?;
// Query using indexes
let query = Query::new().eq("city", "New York".into()).gt("age", json!(30));
let results = collection.find(query, None)?;
println!("Found {} users in New York older than 30", results.len());
// List indexes
let indexes = collection.list_indexes()?;
println!("Indexes:");
for index in indexes {
println!(" {}: {:?}", index.name, index.key);
}
Ok(())
}
```
### Aggregation Pipeline
```rust
use luckdb::{Client, AggregationStage, GroupOperation, GroupId, SortOrder, Query};
use serde_json::json;
fn main() -> luckdb::Result<()> {
let mut client = Client::new();
let db = client.db("test");
let collection = db.collection("sales");
// Insert test data
collection.insert(json!({
"product": "Laptop",
"category": "Electronics",
"price": 1200,
"quantity": 1,
"date": "2023-01-15",
"customer": "Alice"
}))?;
collection.insert(json!({
"product": "Phone",
"category": "Electronics",
"price": 800,
"quantity": 2,
"date": "2023-01-16",
"customer": "Bob"
}))?;
collection.insert(json!({
"product": "Desk Chair",
"category": "Furniture",
"price": 200,
"quantity": 1,
"date": "2023-01-17",
"customer": "Alice"
}))?;
collection.insert(json!({
"product": "Monitor",
"category": "Electronics",
"price": 300,
"quantity": 1,
"date": "2023-01-18",
"customer": "Charlie"
}))?;
// Calculate total sales by category
let pipeline = vec![
AggregationStage::Group(GroupSpecification {
id: GroupId::Field("category".to_string()),
operations: {
let mut ops = std::collections::HashMap::new();
ops.insert("total_sales".to_string(),
GroupOperation::Sum(json!({ "$multiply": ["$price", "$quantity"] })));
ops.insert("count".to_string(), GroupOperation::Sum(json!(1)));
ops
},
}),
AggregationStage::Sort(vec![("total_sales".to_string(), SortOrder::Descending)])
];
let results = collection.aggregate(pipeline)?;
println!("Sales by category:");
for doc in results {
println!(" {}", doc);
}
// Find top customers
let pipeline = vec![
AggregationStage::Group(GroupSpecification {
id: GroupId::Field("customer".to_string()),
operations: {
let mut ops = std::collections::HashMap::new();
ops.insert("total_spent".to_string(),
GroupOperation::Sum(json!({ "$multiply": ["$price", "$quantity"] })));
ops.insert("purchase_count".to_string(), GroupOperation::Sum(json!(1)));
ops
},
}),
AggregationStage::Sort(vec![("total_spent".to_string(), SortOrder::Descending)]),
AggregationStage::Limit(3)
];
let results = collection.aggregate(pipeline)?;
println!("Top customers:");
for doc in results {
println!(" {}", doc);
}
Ok(())
}
```
### Persistence
```rust
use luckdb::Client;
use serde_json::json;
fn main() -> luckdb::Result<()> {
// Create a client with storage path
let mut client = Client::with_storage_path("mongodb://localhost", "./data");
// Load existing data if any
client.load()?;
// Get or create database and collection
let db = client.db("myapp");
let collection = db.collection("users");
// Insert a document
let doc = json!({
"name": "Alice",
"email": "alice@example.com",
"created_at": "2023-01-01T00:00:00Z"
});
let id = collection.insert(doc)?;
println!("Inserted document with ID: {}", id);
// Save data to disk
client.save()?;
println!("Data saved to disk");
// Later, load the data again
let mut client2 = Client::with_storage_path("mongodb://localhost", "./data");
client2.load()?;
let db2 = client2.db("myapp");
let collection2 = db2.collection("users");
// Query the loaded data
let results = collection2.find(luckdb::Query::new(), None)?;
println!("Loaded {} documents", results.len());
for (id, doc) in results {
println!(" {}: {}", id, doc);
}
Ok(())
}
```
### Client-Server Mode
```rust
use luckdb::{Client, Server};
use std::net::SocketAddr;
use std::path::PathBuf;
fn main() -> luckdb::Result<()> {
// Start the server in a separate thread
let server_thread = std::thread::spawn(|| {
let addr: SocketAddr = "127.0.0.1:27017".parse().unwrap();
let storage_path = Some(PathBuf::from("./data"));
let mut server = Server::new(addr, storage_path);
server.start().unwrap();
});
// Give the server time to start
std::thread::sleep(std::time::Duration::from_millis(100));
// Connect to the server
let remote_client = luckdb::RemoteClient::new("127.0.0.1:27017".parse().unwrap());
let mut connection = remote_client.connect()?;
// Send commands
let response = connection.send_command("INSERT mydb users {\"name\":\"Alice\",\"age\":30}")?;
println!("Server response: {}", response);
let response = connection.send_command("FIND mydb users {\"name\":\"Alice\"}")?;
println!("Server response: {}", response);
let response = connection.send_command("SAVE")?;
println!("Server response: {}", response);
// Close the connection
connection.close()?;
// Stop the server
let response = connection.send_command("EXIT")?;
println!("Server response: {}", response);
server_thread.join().unwrap();
Ok(())
}
```
Note: This is just a basic example to give you an idea of how to use LuckDB. In a real-world application, you may want to add more error handling and security features.
```rust
#![allow(warnings)]
use luckdb::{Client, Server};
use std::net::SocketAddr;
use std::path::PathBuf;
fn main() -> luckdb::Result<()> {
// Start the server in a separate thread
let server_thread = std::thread::spawn(|| {
// Create server with authentication
let addr: SocketAddr = "127.0.0.1:27017".parse().unwrap();
let storage_path = Some(PathBuf::from("./data"));
let mut server = Server::new(addr, storage_path)
.with_auth("admin".to_string(), "password123".to_string());
server.start().unwrap();
});
// Give the server time to start
std::thread::sleep(std::time::Duration::from_millis(100));
// Connect to the server
let remote_client = luckdb::RemoteClient::new("127.0.0.1:27017".parse().unwrap());
let mut connection = remote_client.connect()?;
let response = connection.send_command("AUTH admin password123")?;
println!("Authentication response: {}", response);
// Send commands
let response = connection.send_command("INSERT mydb users {\"name\":\"Alice\",\"age\":30}")?;
println!("Server response: {}", response);
let response = connection.send_command("FIND mydb users {\"name\":\"Alice\"}")?;
println!("Server response: {}", response);
let response = connection.send_command("SAVE")?;
println!("Server response: {}", response);
// Close the connection
// connection.close()?;
// Stop the server
let response = connection.send_command("EXIT")?;
println!("Server response: {}", response);
server_thread.join().unwrap();
Ok(())
}
```
## License
LuckDB is licensed under the MIT License. See [LICENSE](LICENSE) for details.