# AEDB
`aedb` is an embedded Rust storage engine for applications that need:
- transactional writes
- durable WAL + checkpoint recovery
- snapshot-consistent reads
- optional permission-aware APIs for multi-tenant workloads
Primary API entry point: `AedbInstance`.
## Why AEDB
AEDB is designed for local-first and service-side state where you want predictable durability and recovery behavior without running an external database process.
Use AEDB when you want:
- in-process storage with explicit durability controls
- deterministic crash recovery from checkpoint + WAL replay
- table + KV data models in one engine
- operational APIs for checkpoint, backup, restore, and diagnostics
## Installation
```toml
[dependencies]
aedb = "0.2.3"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
```
## Quick Start
```rust
use aedb::AedbInstance;
use aedb::catalog::DdlOperation;
use aedb::catalog::schema::ColumnDef;
use aedb::catalog::types::{ColumnType, Row, Value};
use aedb::commit::validation::Mutation;
use aedb::query::plan::{Expr, Query};
use tempfile::tempdir;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let dir = tempdir()?;
let db = AedbInstance::open(Default::default(), dir.path())?;
db.create_project("demo").await?;
db.create_scope("demo", "app").await?;
db.commit_with_preflight(Mutation::Ddl(DdlOperation::CreateTable {
project_id: "demo".into(),
scope_id: "app".into(),
table_name: "users".into(),
owner_id: None,
columns: vec![
ColumnDef {
name: "id".into(),
col_type: ColumnType::Integer,
nullable: false,
},
ColumnDef {
name: "username".into(),
col_type: ColumnType::Text,
nullable: false,
},
],
primary_key: vec!["id".into()],
}))
.await?;
db.commit_with_preflight(Mutation::Insert {
project_id: "demo".into(),
scope_id: "app".into(),
table_name: "users".into(),
primary_key: vec![Value::Integer(1)],
row: Row::from_values(vec![Value::Integer(1), Value::Text("alice".into())]),
})
.await?;
let query = Query::select(&["id", "username"])
.from("users")
.where_(Expr::Eq("id".into(), Value::Integer(1)));
let result = db.query("demo", "app", query).await?;
assert_eq!(result.rows.len(), 1);
Ok(())
}
```
## Core Concepts
### Action envelopes and typed KV numerics
For single hot-path action commits (effects + metadata in one atomic envelope), use
`AedbInstance::commit_action_envelope(...)` with `ActionEnvelopeRequest`.
Result semantics are explicit:
- `ActionCommitOutcome::Applied`
- `ActionCommitOutcome::Duplicate` (idempotent replay, no writes applied)
Native U256 KV mutation variants are available for strict/soft decrement and bounded updates:
- `Mutation::KvAddU256Ex`
- `Mutation::KvSubU256Ex`
- `Mutation::KvMaxU256`
- `Mutation::KvMinU256`
`CommitResult` also exposes idempotency metadata:
- `idempotency: IdempotencyOutcome`
- `canonical_commit_seq`
### Data model
- Namespace hierarchy: `project -> scope -> table`
- Typed relational tables for structured data
- KV APIs for point lookups, prefix/range scans, and counters
- Atomic KV/table integer updates with commit-time read assertions
Casino-style hot balance updates should use native atomic mutations plus assertions,
not a separate accumulator subsystem. This keeps the safety check in the same commit
envelope while allowing non-conflicting atomic updates to stay parallelizable.
Atomic update safety has four layers:
- `preflight` / `preflight_plan` can reject obviously invalid updates before enqueueing, such as a decrement that would underflow the snapshot it inspected.
- `ReadAssertion`s in a `TransactionEnvelope` are authoritative pre-apply commit-time checks. They are evaluated against the working state for the commit epoch before the write intent applies.
- `Mutation::PostflightCheck` evaluates `ReadAssertion`s after prior mutations in the same write intent have applied to the transaction-local trial state. Use this for hot-key atomic updates when the invariant is about the post-update value; it does not add a pre-apply read dependency that would serialize every writer on the same key.
- Atomic mutation variants such as `KvSubU64Ex`, `KvSubU256Ex`, `KvAddI64Bounded`, and `TableDecU256` apply the numeric update without requiring callers to perform a separate read-modify-write. If the mutation itself cannot be applied, it returns a structured error and the envelope rolls back.
Use `commit_with_preflight` for simple single-mutation UX checks. Use an explicit
`TransactionEnvelope` with `Mutation::PostflightCheck` when the business invariant
depends on the value after an atomic update, for example "balance must remain
above the reserve after this debit".
```rust
use aedb::commit::tx::{ReadAssertion, ReadSet, TransactionEnvelope, WriteClass, WriteIntent};
use aedb::commit::validation::{
CompareOp, KvU64MissingPolicy, KvU64UnderflowPolicy, Mutation,
};
use aedb::query::plan::ConsistencyMode;
let balance_key = b"house/balance".to_vec();
let base_seq = db.snapshot_probe(ConsistencyMode::AtLatest).await?;
db.commit_envelope(TransactionEnvelope {
caller: None,
idempotency_key: None,
write_class: WriteClass::Standard,
assertions: Vec::new(),
read_set: ReadSet::default(),
write_intent: WriteIntent {
mutations: vec![
Mutation::KvSubU64Ex {
project_id: "casino".into(),
scope_id: "app".into(),
key: balance_key.clone(),
amount_be: u64::to_be_bytes(500),
on_missing: KvU64MissingPolicy::Reject,
on_underflow: KvU64UnderflowPolicy::Reject,
},
Mutation::PostflightCheck {
assertions: vec![ReadAssertion::KeyCompare {
project_id: "casino".into(),
scope_id: "app".into(),
key: balance_key,
op: CompareOp::Gte,
threshold: u64::to_be_bytes(10_000).to_vec(),
}],
},
],
},
base_seq,
})
.await?;
// Tier-2 event stream + processor checkpoint primitives
db.emit_event(
"casino",
"app",
"hand_settled",
"hand_42".into(),
r#"{"user_id":"u1","wager":100,"pnl":-120}"#.into(),
)
.await?;
let page = db
.read_event_stream(Some("hand_settled"), 0, 100, ConsistencyMode::AtLatest)
.await?;
db.ack_reactive_processor_checkpoint("points_processor", page.next_commit_seq.unwrap_or(0))
.await?;
let processor_lag = db
.reactive_processor_lag("points_processor", ConsistencyMode::AtLatest)
.await?;
```
For event processors, prefer watermark-batched checkpoint ACKs to reduce write load:
```rust
db.ack_reactive_processor_checkpoint_batched(
"points_processor",
processed_seq,
100, // persist every 100 commits advanced
)
.await?;
```
`ack_reactive_processor_checkpoint_batched_as(...)` isolates watermark state per caller and
only updates cache state after successful commit.
Built-in processor scheduler (period + size limits):
```rust
use std::sync::Arc;
let db = Arc::new(db);
db.start_reactive_processor(
"points_processor",
aedb::ReactiveProcessorOptions {
caller_id: Some("processor_points".into()),
topic_filter: Some("hand_settled".into()),
run_on_interval: false,
max_allowed_lag_commits: Some(2_000),
max_allowed_stall_ms: Some(30_000),
max_events_per_run: 256,
max_bytes_per_run: 2 * 1024 * 1024,
max_run_duration_ms: 250,
run_interval_ms: 20,
idle_backoff_ms: 50,
checkpoint_watermark_commits: 64,
max_retries: 3,
retry_backoff_ms: 100,
},
move |db, events| async move {
// Your derived-state logic (idempotent)
for event in events {
let _ = db
.emit_event(
"casino",
"app",
"points_applied",
format!("points:{}", event.event_key),
event.payload_json,
)
.await?;
}
Ok(())
},
)
.await?;
let status = db
.reactive_processor_runtime_status("points_processor")
.await;
let health = db
.reactive_processor_health("points_processor", ConsistencyMode::AtLatest)
.await?;
let processors = db
.list_reactive_processors(ConsistencyMode::AtLatest)
.await?;
db.stop_reactive_processor("points_processor").await?;
```
`caller_id` defines the explicit auth context for that processor runtime:
- event stream reads run via `read_event_stream_as(...)`
- lag/checkpoint access runs via `reactive_processor_lag_as(...)` and
`ack_reactive_processor_checkpoint_batched_as(...)`
- dead-letter writes run via `commit_as(...)`
In secure mode, `caller_id` is required for `start_reactive_processor(...)`.
For periodic refresh jobs (e.g. weekly/all-time leaderboard snapshots), set
`run_on_interval: true`. AEDB will invoke the processor handler on each
`run_interval_ms` tick even when no new events are present.
Lifecycle controls:
- `pause_reactive_processor(name)` disables in registry and stops runtime.
- `resume_reactive_processor(name)` re-enables and starts runtime using the registered handler.
- `list_reactive_processors(consistency)` returns durable config + running state.
- `reactive_processor_health(name, consistency)` returns lag + runtime counters/timestamps.
- `reactive_processor_slo_status(name, consistency)` returns threshold checks and breach reasons.
- `list_reactive_processor_slo_statuses(consistency)` returns all processor SLO states.
- `enforce_reactive_processor_slos(consistency)` returns `Unavailable` when any enabled processor breaches SLO.
Processor configs are durably persisted in a system registry table when started.
On process restart, register the handler again and AEDB auto-resumes enabled processors:
```rust
let resumed = db
.register_reactive_processor_handler("points_processor", move |db, events| async move {
// same handler logic
Ok(())
})
.await?;
assert!(resumed); // true when durable registry had enabled processor config
```
When handler retries are exhausted, AEDB writes failed events to the durable
`_system.app.reactive_processor_dead_letters` table and advances checkpoint so
poison batches do not stall ingestion.
Secure mode/authenticated flows can use `commit_as`, `commit_envelope_as`,
`commit_as_with_preflight`, and `ack_reactive_processor_checkpoint_batched_as`.
Arcana-oriented engine interface primitives (effect batches, keyed-state helpers,
processor pull/commit/context) are also exposed under `aedb::engine_interface`
and as `AedbInstance` methods:
- `commit_effect_batch(project_id, scope_id, EffectBatch)`
- `keyed_state_read` / `keyed_state_read_field` / `keyed_state_write` / `keyed_state_update` / `keyed_state_delete`
- `keyed_state_query_index` / `keyed_state_index_rank`
- `processor_pull(event_name, processor_id, max_count)`
- `processor_commit(processor_id, checkpoint_seq, mutations)` (atomic state + checkpoint commit)
- `processor_context(project_id, scope_id, processor_id, source_event)` with:
- `pull(max_count)`
- `read` / `query_index`
- `write` / `update` / `delete`
- `accumulate` / `value` / `expose` / `release_exposure`
- `emit`
- `commit`
### Consistency modes
Reads are snapshot-based and configurable via `ConsistencyMode`:
- `AtLatest`
- `AtSeq`
- `AtCheckpoint`
```rust
use aedb::query::plan::{ConsistencyMode, Query, QueryOptions};
let result = db
.query_with_options(
"demo",
"app",
Query::select(&["*"]).from("users"),
QueryOptions {
consistency: ConsistencyMode::AtCheckpoint,
..QueryOptions::default()
},
)
.await?;
println!("snapshot seq = {}", result.snapshot_seq);
```
### Preflight and commits
- `preflight` and `preflight_plan` are advisory; state may change before commit
- `commit_with_preflight` / `commit_as_with_preflight` include the preflight read set in the commit envelope, which catches stale reads before writes apply
- `ReadAssertion`s are commit-time checks for invariants that must hold under concurrency
- `Mutation::PostflightCheck` is a commit-time post-apply check for atomic updates; failed checks abort and roll back the whole envelope before publish
- atomic integer updates should encode their own underflow/overflow/missing-key policy and return structured errors rather than relying on fallbacks
- use `commit_with_finality(..., CommitFinality::Visible)` for low-latency user ack
- use `CommitFinality::Durable` for flows that must wait for WAL durability
Low-latency profile example:
```rust
use aedb::config::AedbConfig;
let config = AedbConfig::low_latency([7u8; 32]);
let db = aedb::AedbInstance::open(config, dir.path())?;
```
## Security and Permissions
AEDB supports permission-aware APIs via `CallerContext` and `Permission`.
- `open_production` and `open_secure` require authenticated `*_as` calls
- `open_secure` enforces hardened durability/recovery settings (`DurabilityMode::Full`, strict recovery, hash chain, HMAC)
- table/KV/query access can be scoped per project/scope/resource
- `authz_audit` and `assertion_audit` system tables provide built-in audit trails
Security/operations docs:
- `docs/SECURITY_ACCEPTANCE_CRITERIA.md`
- `docs/SECURITY_OPERATIONS_RUNBOOK.md`
- `docs/AEDB_SDK_PROCESSOR_MACRO_SPEC.md`
- `docs/AEDB_MIGRATION_SYSTEM.md`
## Operational APIs
- `checkpoint_now()` to force a fuzzy checkpoint (does not block commit/query traffic)
- `backup_full(...)` / restore helpers for backup workflows
- `operational_metrics()` for commit latency, queue depth, durable head lag, and more
CLI helper (`src/bin/aedb.rs`) includes offline dump/parity/invariant tooling:
```bash
cargo run --bin aedb -- dump export --data-dir /tmp/aedb-data --out /tmp/aedb-dump.aedbdump
cargo run --bin aedb -- dump parity --dump /tmp/aedb-dump.aedbdump --data-dir /tmp/aedb-data
cargo run --bin aedb -- check invariants --data-dir /tmp/aedb-data
```
Explorer CLI crate (`crates/aedb-explorer`) provides read-only inspection of projects/scopes/tables, schema, and sample rows:
```bash
cargo run -p aedb-explorer -- projects --data-dir /tmp/aedb-data
cargo run -p aedb-explorer -- tables --data-dir /tmp/aedb-data --project demo --scope app
cargo run -p aedb-explorer -- scan-table --data-dir /tmp/aedb-data --project demo --scope app --table users --limit 25
```
## API Areas
- `aedb::commit`: mutations, envelopes, validation
- `aedb::query`: query planning and execution
- `aedb::catalog`: schema, types, and DDL
- `aedb::repository`: typed repository/pagination helpers
- `aedb::declarative`: declarative schema migration builders
- `aedb::backup`, `aedb::checkpoint`, `aedb::recovery`: durability and restore path
## Development
```bash
cargo build
cargo test
```
Focused suites:
```bash
cargo test --test query_integration
cargo test --test backup_restore
cargo test --test crash_matrix
cargo test --test stress
```
Security acceptance gate (mandatory profile):
```bash
./scripts/security_gate.sh
```
Production readiness gate:
```bash
./scripts/production_readiness_gate.sh
```
Production rollout guidance:
- [docs/PRODUCTION_READINESS.md](docs/PRODUCTION_READINESS.md)
## License
Dual-licensed under:
- MIT (`LICENSE-MIT`)
- Apache-2.0 (`LICENSE-APACHE`)