# remdb - Embedded In-Memory Database
[中文版](./README.md)
remdb is a lightweight embedded in-memory database designed for resource-constrained embedded systems, supporting no_std environments with predictable memory usage and high performance.
## Key Features
- **In-Memory Table Storage**: Efficient in-memory table implementation supporting insert, delete, query, and traversal operations
- **Indexing Mechanisms**:
- Hash-based primary key index providing O(1) query performance
- Multiple secondary index types: Hash, SortedArray, BTree (default), TTree
- Support for range queries with SortedArray, BTree and TTree indices
- **Transaction Support**: Complete ACID transaction support, including atomicity, consistency, isolation, and durability
- **Memory Management**: Supports static and dynamic memory allocation with fixed-size block memory pool
- **Platform Abstraction Layer**: Supports both POSIX and baremetal environments
- **Compile-time Configuration**: Table and database configuration via macros for performance optimization
- **Low Power Mode**: Optimized memory usage with reduced transaction log write frequency
- **Incremental Snapshot**: Only saves records with changed version numbers, reducing snapshot size and save time
- **SQL Query Support**: Supports standard SQL SELECT statements to query in-memory database data, including aggregate functions, mathematical functions, time conversion functions, JOIN operations, and LIKE pattern matching operator
- **SQL DDL Support**: Supports CREATE TABLE and DROP TABLE statements, allowing dynamic creation and deletion of table structures
- **SQL Database Management Support**: Supports CREATE DATABASE and DROP DATABASE statements for database creation and management
- **UTF8 Character Support**: Full UTF8 character encoding support, including string storage, character functions, LIKE operator, and sorting
- **JSON Support**: Native JSON data type with support for JSON path queries, JSON modification functions, JSON indexing, and integration with other features
- **SAMPLE BY Syntax**: Supports time series data sampling at specified intervals, providing concise time window aggregation syntax
- **FILL Syntax**: Supports filling missing time windows in time series data to ensure time series continuity, with multiple filling methods
- **Database Monitoring**: Real-time monitoring of database metrics, including memory usage, query performance, and transaction status
- **UDP-based Reliable Data Pub/Sub**: Supports unicast, broadcast, and multicast modes with NACK-based retransmission
- **High Availability Support**:
- Master-slave replication mechanism supporting one-master-one-slave or one-master-multi-slave topology
- Automatic failure detection and failover based on heartbeat mechanism
- Support for both synchronous and asynchronous replication consistency modes:
- Synchronous mode: Master node waits for confirmation from at least one slave before returning, ensuring data consistency
- Asynchronous mode: Master node returns immediately, replicating to slaves asynchronously for higher performance
- Automatic failover with service interruption window less than 2 seconds
- Slave node acknowledgment mechanism: Slaves send acknowledgment to master after receiving WAL logs
- Replication status checking: Regularly checks replication status including slave count and latency
- Support for full and incremental synchronization: Slaves can request full sync or incremental sync from specific log index
- **Startup Sync Protocol**: Complete startup synchronization mechanism ensuring data consistency between slave and master nodes
- Protocol flow: SYNC_REQUEST → SYNC_DATA_BEGIN → SYNC_DATA_CHUNK* → SYNC_DATA_END → SYNC_ACK
- Full sync: Sends complete database snapshot with support for chunked transfer of large data
- Incremental sync: Sends WAL logs after specified log index to reduce network transfer
- Data integrity: Supports CRC32 checksum verification
- **Vector Database Support**:
- Native vector data type: `VECTOR(dimension)`
- Support for multiple distance metrics: L2 (Euclidean), IP (Inner Product), COSINE (Cosine Similarity)
- Multiple vector index types: HNSW, HNSW_SQ (with scalar quantization), HNSW_BQ (with binary quantization), IVF, IVF_FLAT (with flat quantization), IVF_PQ (with product quantization)
- Vector similarity search: Support for L2 distance `<->`, inner product `<#>`, cosine similarity `<=>` operators
- Hybrid search: Support for combining vector search with scalar filtering
- **Time Series Database Support**:
- Dedicated time series table implementation optimized for time series data storage and querying
- Support for multiple compression algorithms
- Support for time series data partitioning
- Support for time series data lifecycle management
- Support for time series data indexing
- **C Language Interface**: Provides C language API for C/C++ applications
- **RBAC Permission Management**: Role-based access control (RBAC) supporting user, role, and permission management for fine-grained data access control
- **AI Model Inference**: Integrated ONNX runtime for AI model inference, supporting built-in models and custom model loading
- **WAL Log Compression**: Supports LZ4 and Zstd WAL log compression algorithms to reduce log storage space
- **System Tables**: Provides system table management for database metadata query and system information monitoring
## Technical Characteristics
- **Zero External Dependencies**: No external library dependencies, supports no_std environments
- **Predictable Memory Usage**: Static memory allocation suitable for resource-constrained embedded systems
- **Compile-time Optimization**: Compile-time configuration via macros reduces runtime overhead
- **Multi-platform Support**: Supports both POSIX and baremetal environments
- **Type Safety**: Leverages Rust's type system to ensure data safety
- **Efficient Synchronization**: Implements spinlock synchronization mechanism suitable for multi-threaded environments
## Quick Start
### Installation
Add remdb to your Cargo.toml file:
```toml
[dependencies]
remdb = { path = "./remdb", default-features = false }
# Optional features
# features = ["std", "posix", "pubsub", "ha"]
# Note: ha depends on pubsub feature, enabling ha will automatically enable pubsub
```
### Feature Description
| std | - | Enable standard library support |
| posix | - | Enable POSIX platform support |
| baremetal | log | Enable baremetal platform support (no standard library dependencies) |
| pubsub | std | Enable UDP-based reliable data publish/subscribe functionality |
| ha | pubsub | Enable high availability support (master-slave replication mechanism) |
| log | - | Enable logging functionality |
| debug | - | Enable debug level logging (only effective in debug builds) |
| c-api | - | Enable C language API interface |
| wal-compression-lz4 | lz4 | Enable LZ4 WAL log compression |
| wal-compression-zstd | zstd | Enable Zstd WAL log compression |
| model-runtime | ort, serde, bincode, tokio, ndarray | Enable ONNX runtime model inference |
| model-download | reqwest, sha2, std, futures, ureq | Enable model download functionality |
### Logging Configuration
remdb provides flexible logging configuration options supporting log output in different environments:
#### Standard Library Environment Logging Configuration
```rust
use remdb::log::{init_logger, init_logger_with_file};
// Basic initialization (output to console, debug level)
init_logger();
// Initialization with file output
// debug_mode: true - Output DEBUG and above level logs
// debug_mode: false - Output INFO and above level logs only
init_logger_with_file("/var/log/remdb.log", true).unwrap();
```
#### no_std Environment Logging Configuration
In no_std environment, log level is automatically controlled by build mode:
- **Debug build**: Output DEBUG and above level logs
- **Release build**: Output WARN and above level logs only
```rust
use remdb::log::init_logger;
// Initialize no_std logging
init_logger();
// Use logging macros
use remdb::log::{debug, info, warn, error};
debug!("Debug message"); // Only output in debug builds
info!("Info message");
warn!("Warning message");
error!("Error message");
```
#### Log Level Description
| TRACE | Most detailed trace information | ✓ | ✗ |
| DEBUG | Debug information | ✓ | ✗ |
| INFO | Normal information | ✓ | ✓ |
| WARN | Warning information | ✓ | ✓ |
| ERROR | Error information | ✓ | ✓ |
## Three Ways to Use remdb with Rust
remdb provides three main ways to use it with Rust to meet different scenario requirements:
### 1. Direct Table Data Structure Definition
Use the `remdb::table!` macro to directly define table structures, which is the most basic usage suitable for simple scenarios:
```rust
#![no_std]
#![feature(alloc_error_handler)]
extern crate alloc;
use core::alloc::Layout;
use remdb::*;
// Define memory buffer
static mut DB_MEMORY: [u8; 65536] = [0u8; 65536];
// Directly define table structure
remdb::table!(
users,
100, // Maximum record count
primary_key: id,
secondary_index: name,
fields: {
id: i32,
name: str(32), // 32-byte fixed-length string
age: i8,
active: bool,
created_at: u64
}
);
// Define database configuration
remdb::database!(
tables: [users]
);
// Memory allocation error handler
#[alloc_error_handler]
fn alloc_error_handler(layout: Layout) -> ! {
panic!("Allocation error: {:?}", layout);
}
fn main() {
unsafe {
// Initialize memory allocator
memory::allocator::init_global_allocator(
DB_MEMORY.as_mut_ptr(),
DB_MEMORY.len()
);
// Initialize platform abstraction layer
platform::init_platform(platform::posix::get_posix_platform());
// Initialize global database
let db = init_global_db(
database!(tables: [users]),
&mut [None; 1],
&mut [None; 1],
&mut [None; 1]
).unwrap();
// Use database...
}
}
```
### 2. MemTable Definition with Macros
Use the `#[derive(MemdbTable)]` macro to define tables, supporting inline DDL and external DDL files for more flexible table definition:
#### Inline DDL Mode
```rust
use remdb_macros::MemdbTable;
// Define table with indexes using inline DDL
#[derive(MemdbTable)]
#[memdb_schema(ddl = "CREATE TABLE user (id INTEGER PRIMARY KEY, name TEXT NOT NULL, age INTEGER, active BOOLEAN);
CREATE INDEX idx_user_name ON user USING btree (name);
CREATE INDEX idx_user_age ON user USING hash (age);")]
struct UserTable;
fn main() {
// Test generated User struct
let user = User {
id: 1,
name: "Alice".to_string(),
age: Some(30),
active: Some(true),
};
println!("Generated User struct: {:?}", user);
println!("User name: {}", user.name);
println!("User age: {:?}", user.age);
}
```
#### File Mode
```rust
use remdb_macros::MemdbTable;
// Define tables with indexes using external DDL file
#[derive(MemdbTable)]
#[memdb_schema(file = "./schema.ddl")]
struct MyDatabase;
// schema.ddl content:
// CREATE TABLE user (
// id INTEGER PRIMARY KEY,
// name TEXT NOT NULL,
// email TEXT UNIQUE NOT NULL
// );
//
// CREATE INDEX idx_user_name ON user USING btree (name);
// CREATE INDEX idx_user_email ON user (email); -- Default to BTree
```
### 3. Dynamic DDL Creation with DdlExecutor
Use the `DdlExecutor` trait to dynamically create tables and indexes at runtime, suitable for scenarios requiring flexible configuration:
```rust
use remdb::{RemDb, DdlExecutor, types::{DataType, IndexType}};
use remdb::config::{DbConfig, MemoryAllocator};
use core::ptr::NonNull;
// Simple memory allocator implementation
struct SimpleAllocator {
base_ptr: NonNull<u8>,
size: usize,
used: usize,
}
impl SimpleAllocator {
pub const fn new(base_ptr: NonNull<u8>, size: usize) -> Self {
Self {
base_ptr,
size,
used: 0,
}
}
}
impl MemoryAllocator for SimpleAllocator {
fn allocate(&self, size: usize) -> Option<NonNull<u8>> {
let new_used = self.used + size;
if new_used <= self.size {
let ptr = NonNull::new((self.base_ptr.as_ptr() as usize + self.used) as *mut u8)?;
Some(ptr)
} else {
None
}
}
fn deallocate(&self, _ptr: NonNull<u8>, _size: usize) {
// Simplified implementation, no actual memory deallocation
}
}
fn main() {
// Allocate memory for database
let mut buffer = [0u8; 1024 * 1024]; // 1MB
let base_ptr = NonNull::new(buffer.as_mut_ptr()).unwrap();
// Create memory allocator
let allocator = SimpleAllocator::new(base_ptr, buffer.len());
// Create database configuration
let config = DbConfig {
tables: vec![],
total_memory: buffer.len(),
low_power_mode_supported: false,
low_power_max_records: None,
memory_allocator: &allocator,
#[cfg(feature = "pubsub")]
pubsub_config: None,
#[cfg(feature = "ha")]
ha_role: remdb::config::HARole::Auto,
#[cfg(feature = "ha")]
replication_mode: remdb::config::ReplicationMode::Asynchronous,
#[cfg(feature = "ha")]
ha_config: None,
#[cfg(feature = "ha")]
replication_sync_timeout: 5000,
};
// Initialize table and index arrays
let mut tables = [None; 8];
let mut primary_indices = [None; 8];
let mut secondary_indices = [None; 8];
// Create database instance
let mut db = RemDb::new(
&config,
&mut tables,
&mut primary_indices,
&mut secondary_indices
);
// Create table using DdlExecutor trait
let result = db.create_table(
"users",
&[
("id", DataType::UInt32),
("name", DataType::VarChar),
("age", DataType::UInt8),
("active", DataType::Bool),
],
Some(0) // Primary key is id field
);
// Create table using SQL statement
let result = db.sql_query(
"CREATE TABLE products (id UINT32 PRIMARY KEY, name STRING, price FLOAT32, in_stock BOOL);"
);
// Create index using DdlExecutor trait
let result = db.create_index(
"users",
"name",
IndexType::BTree
);
}
```
## Other Access Methods
### C Language Interface Access
remdb provides a C language interface for C/C++ applications:
```c
#include "remdb_c.h"
int main() {
// Initialize database
remdb_t *db = remdb_init();
// Create table
remdb_create_table(db, "users", ...);
// Insert data
remdb_insert(db, "users", ...);
// Query data
remdb_result_t *result = remdb_query(db, "SELECT * FROM users");
// Process results...
// Free resources
remdb_free_result(result);
remdb_close(db);
return 0;
}
```
### JDBC Access
remdb provides a JDBC driver, allowing Java applications to access remdb databases through JDBC API:
```java
import java.sql.*;
public class RemdbExample {
public static void main(String[] args) {
try {
// Load driver
Class.forName("com.remdb.jdbc.Driver");
// Establish connection
String url = "jdbc:remdb://localhost:8080/dbname";
Connection conn = DriverManager.getConnection(url);
// Create Statement
Statement stmt = conn.createStatement();
// Execute query
ResultSet rs = stmt.executeQuery("SELECT * FROM users");
// Process result set
while (rs.next()) {
System.out.println(rs.getInt("id") + ": " + rs.getString("name"));
}
// Close resources
rs.close();
stmt.close();
conn.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
```
### UDP-based Reliable Data Subscription and Publishing
> Note: Using this feature requires enabling the `pubsub` feature in Cargo.toml
remdb provides a UDP-based reliable data publish/subscribe mechanism, supporting unicast, broadcast, and multicast modes, suitable for data synchronization in distributed systems. The system includes several predefined topics for publishing different types of database events:
#### Predefined Topics
| wal | - | All WAL operations | WAL_LOG_<id>: Operation=<operation_type>, Table=<table_name>, ID=<record_id>, Data=<data> |
| tables | - | Table creation/deletion events | CREATE:table=<table_name>,id=<table_id>,fields=<field_count> or DELETE:table=<table_name>,id=<table_id> |
| metrics | - | Database metrics | JSON-formatted database metrics data |
| healthstatus | - | Health status | JSON-formatted health status data |
| table.<table_name> | - | Table content changes | INSERT:table=<table_name>,id=<record_id>,data=<hex_data> or UPDATE:table=<table_name>,id=<record_id>,data=<hex_data> |
| SYNC_REQUEST | 2 | Sync request topic | Slave sends sync request to master |
| SYNC_DATA_BEGIN | 5 | Sync data begin | Master sends sync metadata to slave |
| SYNC_DATA_CHUNK | 6 | Sync data chunk | Master sends sync data chunks to slave |
| SYNC_DATA_END | 7 | Sync data end | Master signals end of sync data transmission |
| SYNC_ACK | 8 | Sync acknowledgment | Slave sends acknowledgment to master |
#### Usage Example
```rust
use std::time::Duration;
use remdb::pubsub::{PubSub, PubSubConfig, UdpMode};
// Create publish/subscribe configuration
let config = PubSubConfig {
udp_mode: UdpMode::Broadcast,
multicast_addr: None,
port: 5555,
max_topics: 32,
max_subscribers_per_topic: 16,
buffer_size: 4096,
enable_nack: true,
retransmit_timeout: Duration::from_millis(100),
max_retransmits: 3,
heartbeat_interval: Duration::from_secs(10),
frame_pool_size: 128,
};
// Create publish/subscribe instance
let mut pubsub = PubSub::new(config).expect("Failed to create PubSub instance");
pubsub.init().expect("Failed to initialize PubSub");
// Define subscription callback
true
};
// Subscribe to topic
let subscription_id = pubsub.subscribe(0, callback).expect("Failed to subscribe");
// Publish data
let msg = "Hello, PubSub!";
pubsub.publish(0, msg.as_bytes()).expect("Failed to publish");
// Unsubscribe
pubsub.unsubscribe(subscription_id).expect("Failed to unsubscribe");
```
## SQL Query Examples
remdb supports standard SQL SELECT statements to query data in the in-memory database, including various aggregate functions, mathematical functions, and time conversion functions:
```rust
// Execute SQL query to get all users
let result = db.sql_query("SELECT * FROM users").unwrap();
println!("{}", result.to_string());
// Execute SQL query with condition
let result = db.sql_query("SELECT name, age FROM users WHERE age > 25 ORDER BY name ASC LIMIT 10").unwrap();
for row in result {
println!("{}: {}", row.get(0), row.get(1));
}
// Execute SQL query with condition and sorting
let result = db.sql_query("SELECT * FROM users WHERE active = true ORDER BY created_at DESC").unwrap();
for row in result {
println!("ID: {}, Name: {}, Age: {}, Active: {}",
row.get(0), row.get(1), row.get(2), row.get(3));
}
// Use time conversion functions
let result = db.sql_query("SELECT id, name, TO_ISO8601(created_at) AS iso_created FROM users").unwrap();
// Use TO_CHAR function to format time
let result = db.sql_query("SELECT id, name, TO_CHAR(created_at, 'YYYY-MM-DD HH24:MI:SS') AS formatted_date FROM users").unwrap();
// Use TO_EPOCH function to get Unix timestamp
let result = db.sql_query("SELECT id, name, TO_EPOCH(created_at) AS unix_time FROM users").unwrap();
// Combine aggregate functions with time functions
let result = db.sql_query("SELECT TO_CHAR(timestamp, 'YYYY-MM-DD') AS date, AVG(value) AS avg_value FROM sensor_data GROUP BY date").unwrap();
// Use LIKE operator for pattern matching
let result = db.sql_query("SELECT * FROM users WHERE name LIKE 'A%'").unwrap(); // Match names starting with 'A'
let result = db.sql_query("SELECT * FROM users WHERE name LIKE '%son'").unwrap(); // Match names ending with 'son'
let result = db.sql_query("SELECT * FROM users WHERE name LIKE '%mi%'").unwrap(); // Match names containing 'mi'
let result = db.sql_query("SELECT * FROM users WHERE name LIKE 'A__e'").unwrap(); // Match names of length 4 starting with 'A' and ending with 'e'
```
## Time Series Database
remdb provides powerful time series database functionality, specifically designed for efficient storage and querying of time series data:
### Basic Usage
```rust
use remdb::*;
use remdb::time_series::*;
use std::time::{Duration, SystemTime};
// Define time series table structure
remdb::table!(
sensor_data,
5000, // Maximum record count
primary_key: id,
secondary_index: timestamp,
fields: {
id: i32,
sensor_id: str(32), // Sensor ID
sensor_type: str(32), // Sensor type
value: f64, // Sensor value
timestamp: u64, // Timestamp
location: str(64) // Location information
}
);
// Define database configuration
remdb::database!(
DB_CONFIG,
tables: [sensor_data]
);
fn main() {
unsafe {
// Initialize memory allocator
let memory_size = 128 * 1024 * 1024; // 128MB
static mut DB_MEMORY: [u8; 128 * 1024 * 1024] = [0u8; 128 * 1024 * 1024];
memory::allocator::init_global_allocator(
DB_MEMORY.as_mut_ptr(),
DB_MEMORY.len()
).expect("Failed to initialize memory allocator");
// Initialize platform abstraction layer
platform::init_platform(platform::posix::get_posix_platform());
// Initialize global database
let db = init_global_db(&DB_CONFIG).unwrap();
// Get table reference
let table_mut = db.get_table_mut(0).unwrap();
// Simulate inserting sensor data...
// Query data within time range
let start_time = base_time;
let end_time = base_time + 30 * 60000; // 30 minutes
let mut result_buffer = [0u8; 160 * 50]; // Buffer for 50 records
let found_count = table_mut.get_records_in_time_window(
4, // timestamp field index
start_time,
end_time,
result_buffer.as_mut_ptr(),
50
).unwrap();
// Calculate statistics within time range
match table_mut.aggregate_count(4, start_time, end_time) {
Ok(count) => {
println!("Record count within time range: {}", count);
// Calculate average, sum, min, max...
},
Err(e) => println!("Failed to count records: {:?}", e)
}
}
}
### Time Series Pre-Aggregation
remdb now supports time series data pre-aggregation, which automatically calculates and stores aggregated results at different time intervals during data writing, significantly improving query performance.
#### Key Features
- **Automatic Updates**: Pre-aggregated data is automatically updated when new data is written
- **Multiple Aggregation Functions**: Supports SUM, AVG, MIN, and MAX aggregation functions
- **Custom Time Intervals**: Allows configuring different time intervals for pre-aggregation
- **Thread Safety**: Uses Mutex to ensure data consistency during concurrent writes
- **Efficient Storage**: Stores pre-aggregated data in a hash table for fast lookup
#### Usage Example
```rust
use remdb::*;
use remdb::time_series::*;
// Get time series table reference
let ts_table = db.get_time_series_table("sensor_data").unwrap();
// Add pre-aggregation configurations
ts_table.add_pre_aggregation(60, "AVG").unwrap(); // 1-minute average
ts_table.add_pre_aggregation(300, "SUM").unwrap(); // 5-minute sum
// Write data (pre-aggregations will be automatically updated)
let records = vec![
TimeSeriesRecord {
timestamp: 1609459200000, // 2021-01-01 00:00:00
value: 25.5,
tags: vec!["sensor_id=1"],
},
// More records...
];
ts_table.batch_write(&records).unwrap();
// Query pre-aggregated data
let start_time = 1609459200000;
let end_time = 1609462800000; // 1 hour later
// Query 1-minute average data
let avg_result = ts_table.query_pre_aggregated(start_time, end_time, 60, "AVG").unwrap();
for record in avg_result {
println!("Time: {}, Average Value: {}", record.timestamp, record.value);
}
// Query 5-minute sum data
let sum_result = ts_table.query_pre_aggregated(start_time, end_time, 300, "SUM").unwrap();
for record in sum_result {
println!("Time: {}, Sum Value: {}", record.timestamp, record.value);
}
```
#### Benefits
- **Faster Queries**: Directly retrieve pre-calculated results instead of computing on the fly
- **Reduced CPU Usage**: Avoids real-time aggregation calculations
- **Consistent Performance**: Query performance remains stable regardless of data volume
- **Flexible Configuration**: Supports multiple aggregation functions and time intervals
#### Use Cases
- **Real-time Monitoring**: Quickly query recent aggregated data
- **Historical Analysis**: Efficiently query long-term aggregated results
- **Dashboard Display**: Pre-calculate common time interval aggregations
- **Alerting Systems**: Use pre-aggregated data for threshold-based alerts
## Vector Database
remdb provides powerful vector database functionality, supporting native vector types, multiple distance metrics, and efficient vector indexing:
### Basic Usage
```rust
use remdb::*;
use remdb::config::{DbConfig, WALConfig};
// Initialize database configuration
let config = Box::leak(Box::new(DbConfig {
tables: vec![],
total_memory: 16 * 1024 * 1024, // 16MB
low_power_mode_supported: false,
low_power_max_records: None,
memory_allocator: &SimpleAllocator,
wal_config: WALConfig {
log_path: "./wal",
log_mode: remdb::config::LogMode::Async,
checkpoint_interval_ms: 60000,
log_file_size_limit: 16 * 1024 * 1024,
log_prealloc_size: 4 * 1024 * 1024,
log_segment_size: 16 * 1024 * 1024,
retained_checkpoints: 2,
},
time_series_defaults: TimeSeriesConfig {
partition_duration_secs: 3600,
retention_period_secs: 7 * 24 * 3600,
compression: remdb::time_series::compression::CompressionType::None,
max_partitions: 100,
},
}));
// Initialize database
let mut db = RemDb::new(config);
db.init()?;
// Create table with vector field
let create_sql = r#"CREATE TABLE products (
id INT32 PRIMARY KEY,
name TEXT,
embedding VECTOR(4) WITH DISTANCE=IP
)"#;
db.sql_query(create_sql)?;
// Insert vector data
let insert_sql = r#"INSERT INTO products (id, name, embedding) VALUES
(1, 'product1', '[0.1, 0.2, 0.3, 0.4]'),
(2, 'product2', '[1.0, 0.9, 0.8, 0.7]')
"#;
db.sql_query(insert_sql)?;
// Vector similarity query - inner product distance
let similarity_sql = "SELECT id, name, embedding <#> '[0.2, 0.3, 0.4, 0.5]' AS similarity FROM products ORDER BY similarity DESC LIMIT 2";
let similarity_result = db.sql_query(similarity_sql)?;
// Create vector index
let create_index_sql = "CREATE INDEX idx_products_embedding ON products (embedding) USING HNSW WITH (M=16, ef_construction=200)";
db.sql_query(create_index_sql)?;
```
## RBAC Permission Management
remdb provides role-based access control (RBAC), supporting user, role, and permission management for fine-grained data access control.
### Basic Usage
```rust
use remdb::rbac::{Permission, Role, User, RbacManager};
// Create RBAC manager
let mut rbac = RbacManager::new();
// Create role
let role = Role::new("admin", "Administrator");
rbac.create_role(role);
// Create user
let user = User::new("alice", "password_hash");
rbac.create_user(user);
// Assign role to user
rbac.assign_role("alice", "admin");
// Grant permissions
rbac.grant_permission("admin", Permission::Select("*"));
rbac.grant_permission("admin", Permission::Insert("*"));
rbac.grant_permission("admin", Permission::Delete("*"));
// Check permission
let has_permission = rbac.check_permission("alice", &Permission::Select("users"));
```
## AI Model Inference
remdb integrates ONNX runtime, supporting AI model inference with built-in models and custom model loading.
### Basic Usage
```rust
use remdb::model::{OnnxModel, ModelManager, ModelUDF};
use remdb::model::builtin_models::{list_builtin_models, get_builtin_model};
// List built-in models
let models = list_builtin_models();
// Load built-in model
let model = get_builtin_model("bge-m3");
// Create model manager
let mut manager = ModelManager::new();
let model_path = "models/model.onnx";
let model = OnnxModel::load(model_path)?;
manager.register_model("my_model", model);
// Execute model inference
let input = vec![0.1, 0.2, 0.3, 0.4];
let output = manager.infer("my_model", &input)?;
```
## WAL Log Compression
remdb supports LZ4 and Zstd WAL log compression algorithms to effectively reduce log storage space.
### Configuration Example
```rust
use remdb::config::WALCompressionType;
// Use LZ4 compression
let mut config = DbConfig::default();
config.wal_compression = WALCompressionType::Lz4;
// Use Zstd compression
config.wal_compression = WALCompressionType::Zstd;
```
## System Tables
remdb provides system table management for database metadata query and system information monitoring.
### Basic Usage
```sql
-- Query all table information
SELECT * FROM information_schema.tables;
-- Query table structure information
SELECT * FROM information_schema.columns WHERE table_name = 'users';
-- Query database status
SELECT * FROM information_schema.database_status;
```
## Platform Support
### POSIX Platform
Enable POSIX platform support:
```toml
features = ["posix"]
```
### Baremetal Platform
Enable baremetal platform support:
```toml
features = ["baremetal"]
```
## Testing
### Run Core Library Tests
```bash
cargo test --lib
```
### Run Core Library Tests with Specific Features
```bash
cargo test --lib --features "pubsub ha"
```
### Run Full Test Suite
```bash
cargo test
```
### Check Compilation
Check compilation in no_std environment:
```bash
cargo check --tests --no-default-features
```
### Check Compilation in baremetal environment:
```bash
cargo check --no-default-features --features=baremetal
```
### Running Tests in Baremetal Environment
Due to the test framework's dependency on the std library, directly running `cargo test` in a baremetal environment will fail. However, you can verify the correctness of the code in a baremetal environment through the following steps:
1. Ensure the code compiles successfully:
```bash
cargo check --no-default-features --features=baremetal
```
2. For actual baremetal hardware testing, you may need:
- Cross-compilation toolchain
- Test code written for the target hardware
- Appropriate linker script configuration
- Flashing tool to write the executable to hardware
3. Example cross-compilation command (for ARM Cortex-M):
```bash
cargo build --target thumbv7m-none-eabi --no-default-features --features=baremetal
```
### Testing Notes
- Core library tests (`cargo test --lib`) do not depend on specific features and are the best way to verify basic functionality
- The full test suite (`cargo test`) may fail because examples and integration tests depend on specific features
- Tests with features (such as `--features "pubsub ha"`) require that related features are correctly configured
- Some examples and integration tests may require specific runtime environments or configurations
## Examples
Check the examples directory for sample code:
- `basic_usage.rs`: Basic usage example demonstrating table definition, insertion, query, and transaction operations
- `low_power_mode.rs`: Low power mode example demonstrating how to configure and use low power mode
- `incremental_snapshot.rs`: Incremental snapshot example demonstrating how to save and restore incremental snapshots
- `sql_query.rs`: SQL query example demonstrating how to use SQL to query the in-memory database
- `ddl_example.rs`: DDL example demonstrating how to define tables and indexes using DDL macros
- `ddl_runtime_example.rs`: Runtime DDL configuration example demonstrating how to use the runtime DDL API
- `pubsub_example.rs`: Pub/Sub example demonstrating how to use the UDP-based reliable data publish/subscribe functionality
- `time_series.rs`: Time series example demonstrating how to handle time series data
- `vector_example.rs`: Vector database example demonstrating how to use vector fields, insert vector data, and perform vector similarity queries
- `vector_distance_test.rs`: Vector distance test example demonstrating vector similarity calculations with different distance metrics
- `drop_table_example.rs`: DROP TABLE example demonstrating how to use SQL DROP TABLE statements to delete table structures
- `test_remdb_server.rs`: Master-slave replication example demonstrating how to run master and slave servers with synchronous or asynchronous replication mode
### Master-Slave Replication Example
> Note: Using this feature requires enabling the `ha` feature in Cargo.toml
The `test_remdb_server.rs` example demonstrates how to use the master-slave replication feature, supporting setting synchronous or asynchronous replication mode via command line arguments:
#### Master Node Start Command
```bash
# Synchronous mode
cargo run --example test_remdb_server master sync
# Asynchronous mode
cargo run --example test_remdb_server master async
```
#### Slave Node Start Command
```bash
# Synchronous mode
cargo run --example test_remdb_server slave sync <master_ip> <master_port>
# Asynchronous mode
cargo run --example test_remdb_server slave async <master_ip> <master_port>
```
#### Example Output
```
Starting RemDB Server...
Role: Master
Replication Mode: Sync
RemDB Server started successfully!
Listening on UDP port 5555
Topics available:
- WAL_INSERT (ID: 1) - WAL insert operations
- WAL_UPDATE (ID: 2) - WAL update operations
- WAL_DELETE (ID: 3) - WAL delete operations
- WAL_TIMESERIES_INSERT (ID: 4) - WAL timeseries insert operations
- WAL_COMMIT (ID: 5) - WAL commit operations
- WAL_ABORT (ID: 6) - WAL abort operations
- WAL_CHECKPOINT (ID: 7) - WAL checkpoint operations
- WAL_ALL (ID: 8) - All WAL operations
- TABLES (ID: 9) - Table creation/deletion events
- HEARTBEAT - Sent every 5 seconds
```
## Project Structure
```
remdb/
├── src/
│ ├── lib.rs # Main library entry point
│ ├── types.rs # Basic data type definitions
│ ├── config.rs # Compile-time configuration macros
│ ├── table.rs # In-memory table implementation
│ ├── index.rs # Index implementation
│ ├── transaction.rs # Transaction management
│ ├── monitor.rs # Database monitoring module
│ ├── c_api.rs # C language interface implementation
│ ├── compression.rs # Data compression module
│ ├── log.rs # Logging module
│ ├── utf8.rs # UTF8 character support
│ ├── sync.rs # Synchronization primitives
│ ├── system_tables.rs # System tables management
│ ├── wal_compression.rs # WAL log compression
│ ├── sql/
│ │ ├── mod.rs # SQL query module
│ │ ├── query_parser.rs # SQL query parser
│ │ ├── query_executor.rs # SQL query executor
│ │ ├── result_set.rs # Result set handling
│ │ ├── error.rs # SQL error handling
│ │ ├── utils.rs # SQL utility functions
│ │ ├── functions/ # SQL functions
│ │ │ ├── mod.rs
│ │ │ ├── aggregate.rs # Aggregate functions
│ │ │ ├── math.rs # Math functions
│ │ │ ├── string.rs # String functions
│ │ │ ├── time.rs # Time functions
│ │ │ └── json.rs # JSON functions
│ │ └── operations/ # SQL operations
│ │ ├── mod.rs
│ │ ├── expression.rs # Expression handling
│ │ ├── comparison.rs # Comparison operations
│ │ ├── ddl.rs # DDL operations
│ │ └── vector.rs # Vector operations
│ ├── memory/
│ │ ├── allocator.rs # Static memory allocator
│ │ ├── pool.rs # Memory pool
│ │ └── mod.rs
│ ├── platform/
│ │ ├── mod.rs # Platform abstraction layer definition
│ │ ├── posix.rs # POSIX platform implementation
│ │ └── baremetal.rs # Baremetal platform implementation
│ ├── ha/
│ │ ├── mod.rs # High Availability module entry
│ │ ├── manager.rs # HA Manager implementation
│ │ ├── replication.rs # Replication functionality implementation
│ │ ├── heartbeat.rs # Heartbeat monitoring implementation
│ │ ├── role.rs # Role management implementation
│ │ ├── protocol.rs # Sync protocol definitions
│ │ ├── sync_handler.rs # Master sync handler
│ │ └── sync_receiver.rs # Slave sync receiver
│ ├── pubsub/
│ │ ├── mod.rs # Pub/Sub module entry
│ │ ├── protocol.rs # Protocol frame definition and parsing
│ │ ├── udp.rs # Cross-platform UDP socket encapsulation
│ │ ├── subscriber.rs # Subscriber management
│ │ ├── publisher.rs # Publisher management
│ │ ├── topics.rs # Predefined topics
│ │ ├── ttl_ringbuffer.rs # TTL ring buffer
│ │ └── crc32.rs # CRC32 check implementation
│ ├── json/
│ │ ├── mod.rs # JSON module entry
│ │ ├── document.rs # JSON document processing
│ │ ├── path.rs # JSON path query
│ │ └── memory_pool.rs # JSON memory pool
│ ├── rbac/
│ │ ├── mod.rs # RBAC permission management module
│ │ ├── user.rs # User management
│ │ ├── role.rs # Role management
│ │ ├── permission.rs # Permission definitions
│ │ └── manager.rs # Permission manager
│ ├── model/
│ │ ├── mod.rs # AI model module entry
│ │ ├── model_manager.rs # Model manager
│ │ ├── builtin_models.rs # Built-in models
│ │ ├── onnx_runtime.rs # ONNX runtime
│ │ ├── model_udf.rs # Model UDF function
│ │ ├── cache.rs # Model cache
│ │ ├── downloader.rs # Model downloader
│ │ ├── worker_manager.rs # Worker process manager
│ │ └── worker_protocol.rs # Worker process protocol
│ └── time_series/
│ ├── mod.rs # Time series database module entry
│ ├── table.rs # Time series table implementation
│ ├── index.rs # Time series data indexing
│ ├── compression.rs # Compression algorithms implementation
│ ├── partition.rs # Data partitioning implementation
│ ├── lifecycle.rs # Data lifecycle management
│ └── config.rs # Time series database configuration
├── examples/ # Example code
│ ├── api/ # API usage examples
│ ├── sql/ # SQL usage examples
│ └── misc/ # Other examples
├── tests/ # Test code
├── include/ # C language header files
├── models/ # AI model files
├── onnxruntime/ # ONNX runtime files
├── Cargo.toml # Project configuration
└── README.md # Project documentation
```
## License
MIT License
## Contribution
Issues and pull requests are welcome!
## Project Links
- Domestic: https://gitee.com/totaltrust/remdb
- Abroad: https://github.com/bobjia/remdb
- Crates: https://crates.io/crates/remdb
## Notes
1. remdb is designed for embedded systems and is not suitable for large-scale data storage
2. When used in no_std environments, appropriate memory allocator implementation needs to be provided
3. Ensure proper initialization of memory allocator and platform abstraction layer before use
## Future Plans
- Support more data types
- Optimize memory usage
- Provide more index types
- Add more examples and documentation
- Implement more complex memory optimization algorithms
- Implement more flexible memory allocation strategies
- Complete runtime DDL configuration API, supporting full table and index creation functionality
- Support ALTER TABLE statements
- Optimize performance of runtime DDL operations
- Support more complex index configuration options