# BEAM
**A real-time, decentralized, P2P-synced graph database written in Rust — maintaining wire-format compatibility with [Gun.js](https://gun.eco/).**
[](https://crates.io/crates/beamdb)
[](https://docs.rs/beamdb)
[](LICENSE)
[](https://doc.rust-lang.org/edition-guide/)
[](https://www.rust-lang.org/)
---
## Table of Contents
- [What Is BEAM?](#what-is-beam)
- [Install](#install)
- [Quick Start](#quick-start)
- [WASM / Browser Support](#wasm--browser-support)
- [Architecture](#architecture)
- [Data Model](#data-model)
- [Cryptography (SEA Layer)](#cryptography-sea-layer)
- [Storage Backends](#storage-backends)
- [Wire Protocol](#wire-protocol)
- [Configuration](#configuration)
- [Testing](#testing)
- [Benchmarks](#benchmarks)
- [Features](#features)
- [Security](#security)
- [Contributing](#contributing)
- [Maintainers](#maintainers)
- [Credits](#credits)
- [License](#license)
- [Sponsors](#sponsors)
---
## What Is BEAM?
BEAM is a distributed graph database where every node holds a partial replica of the graph and synchronizes with peers in real time. Data flows over WebSocket relays, UDP multicast, or direct WebRTC connections. All cryptographic operations — signatures, key exchange, encryption — use the SEA layer (Security, Encryption, Authorization), providing Gun.js-compatible wire protocol and cryptographic semantics.
BEAM is a maintained fork of [rod](https://github.com/mmalmi/rod) — a from-scratch Rust port of [Gun.js](https://github.com/amark/gun) by Mark Nadal — maintaining wire-format compatibility so BEAM nodes can interop with Gun.js peers. BEAM has since grown into a comprehensive distributed-database system with multiple storage backends, WebRTC direct P2P, observability, and migration tooling.
### Key Properties
- **Decentralized** — no central server; any peer can relay data to any other
- **Real-time** — `on()` subscriptions deliver updates as they propagate through the mesh
- **Eventually consistent** — last-write-wins conflict resolution via timestamps (matching Gun.js)
- **Encrypted** — SEA layer provides Ed25519 signing, X25519 ECDH, and AES-256-GCM encryption
- **Persistent** — `redb` embedded database (default), `fjall` LSM-tree (recommended for multi-node), `Persy` for high-concurrency, or in-memory for ephemeral use
- **Multi-transport** — WebSocket (relay), UDP multicast (LAN discovery), WebRTC (direct P2P)
- **Browser-ready** — compiles to WebAssembly via `wasm-pack`; same engine, same wire protocol, IndexedDB persistence
---
## Install
```toml
[dependencies]
beamdb = "0.16"
```
Or via the CLI:
```bash
cargo add beamdb
```
Feature flags (all off by default):
```toml
# WebRTC direct P2P support
beamdb = { version = "0.16", features = ["webrtc"] }
# Fjall LSM-tree storage backend (recommended for multi-node deployments)
beamdb = { version = "0.16", features = ["fjall"] }
# Persy storage backend (for high-concurrency workloads)
beamdb = { version = "0.16", features = ["persy"] }
```
---
## WASM / Browser Support
BEAM compiles to WebAssembly and runs in the browser. The same graph engine,
crypto stack (SEA), and actor model that power native nodes work in the browser
via `wasm-bindgen` and `web-sys`.
### Build
```bash
# Install wasm-pack (if not already installed)
cargo install wasm-pack
# Build the browser package
wasm-pack build --target web --release
```
This produces a `pkg/` directory containing:
| `beam_bg.wasm` | The compiled WASM binary (~419KB) |
| `beam.js` | JavaScript glue code (auto-generated by wasm-bindgen) |
| `beam.d.ts` | TypeScript type definitions |
| `package.json` | npm-ready package manifest |
### JavaScript API
```js
import init, { Beam } from "./beam.js";
// Initialize the WASM module (must be called once)
await init();
// Create a BEAM node
const beam = new Beam(); // in-memory (lost on reload)
// or: const beam = Beam.new_persistent(); // IndexedDB (survives reload)
// or: const beam = Beam.new_with_opfs(); // OPFS (survives reload, faster)
// Connect to a relay server
beam.connect("wss://relay.example.com/ws");
// Write data (fire-and-forget, dot-separated paths)
beam.put("users.alice.name", "Alice");
beam.put_num("users.alice.age", 30); // numeric
beam.put_bool("users.alice.active", true); // boolean
beam.put_null("users.alice.deleted"); // explicit null
// Read once (returns a Promise)
const name = await beam.get("users.alice.name"); // "Alice"
// Subscribe to child updates (Gun.js .on() semantics)
beam.on("users.alice.name", (value) => {
console.log("name changed:", value);
});
// Data syncs to all connected peers in real time.
// Stop the node and close connections
beam.stop();
```
### Example: Browser Chat
```html
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>BEAM Browser Chat</title>
</head>
<body>
<div id="messages"></div>
<input id="msg" placeholder="Type a message..." />
<button id="send">Send</button>
<script type="module">
import init, { Beam } from "./beam.js";
await init();
const beam = new Beam();
beam.connect("wss://relay.example.com/ws");
beam.on("chat", (value) => {
const div = document.createElement("div");
div.textContent = value;
document.getElementById("messages").appendChild(div);
});
document.getElementById("send").onclick = () => {
const text = document.getElementById("msg").value;
const ts = Date.now();
beam.put(`chat.${ts}`, text);
document.getElementById("msg").value = "";
};
</script>
</body>
</html>
```
### Example: Persistent Key-Value Store
```js
import init, { Beam } from "./beam.js";
await init();
const beam = Beam.new_persistent();
beam.connect("wss://relay.example.com/ws");
// Writes survive page reload via IndexedDB
beam.put("app.settings.theme", "dark");
beam.put_num("app.settings.fontSize", 14);
beam.put_bool("app.settings.notifications", true);
// Read back (returns a Promise)
const theme = await beam.get("app.settings.theme"); // "dark"
// Multi-user: each browser sees the same data via the relay
// beam.put("shared.todo.1", "Buy milk");
// Other browsers on the same relay see this instantly
```
### Architecture
Browser BEAM is a **client-only node** — it connects to relay servers via
WebSocket but does not accept inbound connections. This matches the browser's
security model (no listening sockets).
```text
Browser Relay Server
┌──────────────────────────┐ ┌──────────────┐
│ Beam (JS) │ │ BEAM Native │
│ └── Node (Rust/WASM) │ WebSocket │ Node │
│ ├── WasmWsConn ───┼──────────────►│ WsServer │
│ ├── MemoryStorage │ (same wire │ RedbStorage │
│ └── WasmIdbStorage│ protocol) │ │
└──────────────────────────┘ └──────────────┘
```
The browser node speaks the **same Gun.js-compatible wire protocol** as native
nodes. A BEAM browser node can connect to a Gun.js relay, a BEAM relay, or any
compatible WebSocket peer.
### Storage
| `MemoryStorage` | No (lost on reload) | Default | Ephemeral data, testing |
| `WasmIdbStorage` | Yes (IndexedDB) | Opt-in via `new_persistent()` | Production browser apps |
| `WasmOpfsStorage` | Yes (OPFS) | Opt-in via `new_with_opfs()` | Modern browsers (Chrome 102+, Firefox 111+, Safari 15.2+) |
| `WasmNodeFsStorage` | Yes (Node.js fs) | Node.js only (`--features node-fs`) | Server-side WASM, Electron |
**WasmIdbStorage** uses a write-through cache: writes go to an in-memory `HashMap`
(fast reads) and are simultaneously written through to IndexedDB (persistence).
Data is serialized as postcard bytes (base64-encoded for IDB string storage),
with automatic JSON fallback for backward compatibility with pre-v0.17 databases.
On page reload, data is read back from IndexedDB into the cache.
**WasmOpfsStorage** uses the Origin Private File System for file-based persistence.
Data is stored as postcard-serialized binary files in OPFS, offering better
performance than IndexedDB for larger datasets. Requires a secure context
(HTTPS or localhost).
### Browser Constraints
- **Single-threaded** — all async work runs on the browser's main thread
- **Client-only** — connects to relays, does not accept inbound connections
- **No native file system** — uses IndexedDB or OPFS instead of redb/Persy
- **WebSocket only** — no UDP multicast or WebRTC (browser sandbox limitations)
### Interop with Gun.js
BEAM browser nodes are wire-compatible with Gun.js. A BEAM WASM node can:
- Connect to a Gun.js relay server
- Exchange `Put` / `Get` / `Hi` messages
- Verify Gun.js SEA signatures (P-256 ECDSA, double-hashed)
- Interoperate with the Gun.js graph format
Bidirectional compatibility is verified by Playwright E2E tests
(`tests/e2e/gun-beam-interop.spec.mjs`) covering Gun.js→BEAM, BEAM→Gun.js,
and bidirectional convergence scenarios.
## Quick Start
### Build & Run a Node
```bash
# Build
cargo build --release
# Start with defaults: redb storage, WebSocket server on port 4944
cargo run --release --bin beam -- --port 4944
# With WebRTC support (direct P2P connections)
cargo run --release --bin beam --features webrtc -- --port 4944
# Connect to existing peers
cargo run --release --bin beam -- --port 4944 --peers wss://relay1.example.com:8080/ws,wss://relay2.example.com:8080/ws
# With TLS
cargo run --release --bin beam -- --port 4944 --cert-path /path/cert.pem --key-path /path/key.pem
# In-memory only (no persistence)
cargo run --release --bin beam -- --port 4944 --memory-storage true
# Restrict to signed data only (disable public space)
cargo run --release --bin beam -- --port 4944 --allow-public-space false
```
### Generate a SEA Session Key
```bash
cargo run --release --bin beam-sea-keygen
```
### Use as a Library
```rust
# use beam::{Node, Value};
# #[tokio::main]
# async fn main() {
let mut db = Node::new();
// Write
db.get("greeting").put("Hello World!".into()).await.unwrap();
// Subscribe to live updates
let mut sub = db.get("greeting").on();
if let Value::Text(s) = sub.recv().await.unwrap() {
println!("{}", s); // "Hello World!"
}
// Read once
let val = db.get("greeting").once(None).await;
assert_eq!(val, Some(Value::Text("Hello World!".into())));
db.stop();
# }
```
### Connect Two Nodes Over WebSocket
```rust,no_run
# use beam::adapters::{OutgoingWebsocketManager, WsServer};
# use beam::{Config, Node, Value};
# #[tokio::main]
# async fn main() {
let config = Config::default();
// Peer 1: WebSocket server
let mut peer1 = Node::new_with_config(
config.clone(),
vec![Box::new(beam::adapters::MemoryStorage::new())],
vec![Box::new(WsServer::new(config.clone()))],
);
// Peer 2: WebSocket client connecting to peer 1
let client = OutgoingWebsocketManager::new(
config.clone(),
vec!["ws://localhost:4944/ws".to_string()],
);
let mut peer2 = Node::new_with_config(
config,
vec![Box::new(beam::adapters::MemoryStorage::new())],
vec![Box::new(client)],
);
// Wait for connection
tokio::time::sleep(std::time::Duration::from_secs(1)).await;
// Peer 2 writes, peer 1 receives via mesh sync
peer2.get("hello").put("from peer 2".into()).await.unwrap();
let mut sub = peer1.get("hello").on();
if let Value::Text(s) = sub.recv().await.unwrap() {
println!("Peer 1 received: {}", s);
}
peer1.stop();
peer2.stop();
# }
```
---
## Architecture
BEAM is built on an actor model with a central router. Every component — storage, network, graph nodes — is an actor communicating via typed messages over Tokio channels.
```text
┌─────────────────────────────────────────────┐
│ Node (root) │
│ uid="" ← the root node owns the router │
│ get("key") → child Node (uid="key") │
│ put(value) → broadcasts to on() subscribers │
│ and sends Put to router │
└────────────────────┬────────────────────────────┘
│ Message::Put / Get / Flush
▼
┌─────────────────────────────────────────────┐
│ Router │
│ - Deduplication (Dup: 999 entries, 9s TTL) │
│ - Peer management (known_peers, server_peers) │
│ - Topic subscriptions (subscribers_by_topic) │
│ - Put relay with anti-loop (peer_hop_list) │
│ - Get routing (storage → server → random) │
│ - RtcSignal routing to specific peers │
└──────┬──────────────┬──────────────┬──────────┘
│ │ │
┌──────▼──────┐ ┌──────▼──────┐ ┌─────▼──────┐
│ Storage │ │ Network │ │ WebRTC │
│ Adapters │ │ Adapters │ │ (opt) │
│ │ │ │ │ │
│ MemoryStorage│ │ WsServer │ │ WebRtcPeer │
│ RedbStorage │ │ WsClient │ │ (str0m) │
│ FjallStorage│ │ Multicast │ │ │
│ PersyStorage│ │ │ │ │
└─────────────┘ └─────────────┘ └────────────┘
```
### Module Map
| `types.rs` | Core data types: `Value` (Null/Bit/Number/Text/Link), `NodeData`, `Children`, JSON conversion |
| `utils.rs` | `random_string()` (OS CSPRNG), `BoundedHashMap` (FIFO eviction for dedup) |
| `dup.rs` | `Dup` — Gun.js DAM-style message deduplication (TTL + bounded capacity) |
| `message.rs` | Wire protocol: `Get`, `Put`, `BatchPut`, `Flush`, `RtcSignal`, `Hi` — JSON serialization/deserialization, signature verification |
| `actor.rs` | Actor framework: `Actor` trait, `ActorContext`, `Addr` — built on Tokio unbounded channels |
| `node.rs` | Graph node API: `put()`, `get()`, `on()`, `once()`, `map()`, `batch_put()`, `connect_peer()`, `connect_webrtc_peer()`, `stop()` |
| `router.rs` | Central router: dedup, Get/Put routing, peer management, topic subscriptions, anti-loop relay, flush forwarding, RtcSignal delivery |
| `ack.rs` | Ack protocol: sentinel-driven async ack across put, batch_put, flush, map, quorum |
| `metrics.rs` | Observability: atomic counters for puts, gets, peer connections, message routing |
| `migration.rs` | Storage migration tooling: `beam migrate` subcommand, batch processing, checksum verification |
| `sea/pair.rs` | Key pair generation: ECDSA P-256 (signing) + ECDH P-256 (encryption), Gun.js `x.y` base64 format |
| `sea/sign.rs` | Signature creation (P-256 ECDSA via `ring`) |
| `sea/verify.rs` | Signature verification (sync + async variants) |
| `sea/work.rs` | Proof-of-work / content hashing (PBKDF2, SHA-256, base64) |
| `sea/secret.rs` | ECDH shared secret derivation between key pairs |
| `sea/encrypt.rs` | AES-256-GCM encryption with PBKDF2 key derivation; symmetric and ECDH-based modes |
| `sea/decrypt.rs` | AES-256-GCM decryption; shares `derive_aes_key_sync` with encrypt.rs (DRY) |
| `sea/certify.rs` | Capability certificates: issue, verify, check certificant membership, expiry enforcement |
| `sea/user.rs` | User identity: `create()`, `auth()`, `leave()`, `trust()`, `grant()`, `secret()`, `is()` — Gun.js `user.is` semantics |
| `sea/session/` | Session persistence: `MemorySessionStorage` (ephemeral) and `EncryptedFileSessionStorage` (disk, AES-GCM) |
| `adapters/memory_storage.rs` | In-memory `HashMap` storage (ephemeral, default for `Node::new()`) |
| `adapters/redb_storage.rs` | Persistent storage via `redb` embedded database — `BatchPut` atomic transactions, flush ack |
| `adapters/fjall_storage.rs` | Persistent storage via `fjall` LSM-tree database — WAL journalling, LZ4 compression, recommended for multi-node (feature-gated) |
| `adapters/persy_storage.rs` | Persistent storage via `Persy` segment store — high-concurrency writes, optional `background_ops` |
| `adapters/ws_server.rs` | WebSocket server: accepts inbound connections, spawns `WsConn` per connection, optional TLS, web UI on port+1 |
| `adapters/ws_client.rs` | `OutgoingWebsocketManager` — connects to remote WebSocket peers with retry |
| `adapters/ws_conn.rs` | Per-connection WebSocket actor: bridges wire format ↔ Message types |
| `adapters/multicast.rs` | UDP multicast LAN discovery (224.0.0.123:6969) — syncs with peers on local network |
| `adapters/webrtc.rs` | WebRTC data channel P2P via `str0m` — ICE/DTLS/SCTP, STUN discovery, TURN relay (feature-gated) |
| `adapters/wasm_ws.rs` | Browser WebSocket adapter (WASM only) — `web-sys::WebSocket` for relay connections from browser nodes |
| `adapters/wasm_idb.rs` | IndexedDB persistent storage (WASM only) — write-through cache + async IndexedDB for browser persistence |
| `wasm.rs` | JavaScript bindings (WASM only) — `#[wasm_bindgen]` exports: `Beam` struct with `connect()`, `put()`, `get()`, `on()`, `stop()` |
| `stun.rs` | STUN Binding Request + TURN Allocate Request helpers (feature-gated) |
| `main.rs` | CLI entry point: clap argument parsing, adapter configuration, signal-based graceful shutdown (SIGINT + SIGTERM) |
| `bin/beam-sea-keygen.rs` | Utility binary: generates 32-byte random session key (base64-encoded) |
---
## Data Model
BEAM uses a **key-path graph** — a hierarchical tree of nodes addressed by `/`-separated paths:
```text
root (uid="")
└── "users" (uid="users")
└── "alice" (uid="users/alice")
└── "profile" (uid="users/alice/profile")
└── "name" (uid="users/alice/profile/name")
└── value = Value::Text("Alice")
```
### Node Operations
| `db.get("key")` | Traverse to child node (creates lazily if it doesn't exist) |
| `node.put(value)` | Set a value on this node; propagates to parents and peers |
| `node.batch_put(ops)` | Atomic multi-write: multiple `(path, value)` pairs in one storage transaction |
| `node.on()` | Subscribe to value updates → `broadcast::Receiver<Value>` |
| `node.once(timeout)` | Read current value once (queries storage + peers), returns `Option<Value>` |
| `node.map()` | Subscribe to all children → `broadcast::Receiver<(String, Value)>` — replays existing children from storage |
| `db.connect_peer(url)` | Add a WebSocket peer at runtime |
| `db.connect_webrtc_peer(...)` | Bootstrap a WebRTC direct connection (requires `webrtc` feature) |
| `db.flush_storage(timeout)` | Flush storage adapters to disk (durable persistence) |
| `db.stop()` | Stop the node and all child actors/adapters |
### Path Depth and Data Access Semantics
BEAM's graph operations differ from Gun.js in important ways. Understanding these prevents confusion.
#### One-Level Paths
Both flat (one-level) and nested paths work in BEAM:
```rust
# use beam::{Node, Value};
# #[tokio::main]
# async fn main() {
let mut db = Node::new();
// Flat path — works
db.get("x").put("Hello World!".into()).await.unwrap();
let mut sub = db.get("x").on();
let _ = sub.recv().await; // Ok(Text("Hello World!"))
// Nested path — also works
db.get("x").get("y").put("Hello World!".into()).await.unwrap();
let mut sub = db.get("x").get("y").on();
let _ = sub.recv().await; // Ok(Text("Hello World!"))
db.stop();
# }
```
> **Gun.js difference:** Gun.js prohibits saving primitive values at the root level — `Gun().put("oops")` and `Gun().get("odd").put("oops")` are errors. BEAM does **not** enforce this restriction. Flat-key writes (`db.get("key").put(val)`) are valid and propagate to storage and peers normally.
#### `on()` — Subscribing to a Single Value
`on()` subscribes to a node's value and immediately requests the current value from storage (and peers, if connected). The broadcast receiver yields values in this order:
1. **Local value first** — if a value was already `put()` on this node, it arrives before any remote updates
2. **Streamed values** — new values from peers, storage replay, or subsequent `put()` calls
3. **Linked values** — `Value::Link("path/to/child")` if a child reference exists
#### `map()` — Subscribing to All Children
`map()` returns a stream of `(child_key, value)` pairs. It replays existing children from storage, then streams new ones as they're added. A sentinel `("__beam_replay_complete__", Null)` signals that all existing children have been replayed; subsequent values are **new** children only.
```rust,no_run
# use beam::{Node, Value};
# #[tokio::main]
# async fn main() {
let mut db = Node::new();
let mut sub = db.get("users").map();
while let Ok((key, value)) = sub.recv().await {
if key == "__beam_replay_complete__" {
break;
}
println!("child: {} = {:?}", key, value);
}
db.stop();
# }
```
The `__beam_replay_complete__` sentinel signals that all existing children have been replayed from storage. Subsequent values on the receiver are **new** children added after subscription. To read a child's actual value, call `on()` or `once()` on the child node directly.
#### `once()` — Read-Once with Timeout
`once()` returns the current value with a 66ms timeout (matching Gun.js's default). If no value exists and no peer responds within the window, returns `None`. Use `once()` for one-shot reads; use `on()` for subscriptions.
### Wire-Compatible Leaf Types
BEAM supports five wire-compatible leaf types, matching Gun.js:
| `Value::Null` | `null` | Absent or explicitly null |
| `Value::Bit(bool)` | `true` / `false` | Booleans |
| `Value::Number(f64)` | JSON number | `42`, `3.14` |
| `Value::Text(String)` | JSON string | `"hello"` |
| `Value::Link(String)` | `{"#": "path/to/child"}` | Reference to another node |
---
## Cryptography (SEA Layer)
The SEA (Security, Encryption, Authorization) module implements Gun.js-compatible cryptography. All operations use `ring` for primitives and `pbkdf2` for key derivation.
### Key Pair Generation
```rust
# use beam::sea;
# #[tokio::main]
# async fn main() {
let pair = sea::generate_pair().await.unwrap();
println!("pub: {}", pair.pub_key);
println!("epub: {}", pair.epub_key.as_ref().unwrap());
println!("priv: {}", pair.priv_key);
println!("epriv: {}", pair.epriv_key.as_ref().unwrap());
# }
```
### Signing and Verification
```rust
# use beam::sea;
# use serde_json::json;
# #[tokio::main]
# async fn main() {
let pair = sea::generate_pair().await.unwrap();
let signed = sea::sign(&json!({"msg": "hello"}), &pair).await.unwrap();
let verified = sea::verify_sync(&signed, &pair.pub_key).unwrap();
# }
```
### Encryption and Decryption
```rust
# use beam::sea;
# use serde_json::json;
# #[tokio::main]
# async fn main() {
let pair = sea::generate_pair().await.unwrap();
// Asymmetric (ECDH key exchange + AES-GCM)
let their_epub = pair.epub_key.as_ref().unwrap().clone();
let encrypted = sea::encrypt(&json!({"secret": "message"}), &pair, Some(&their_epub)).await.unwrap();
let decrypted = sea::decrypt(&encrypted, &pair, Some(&their_epub)).await.unwrap();
// Symmetric: raw 32-byte AES-256 key
let key_bytes: &[u8] = &[0u8; 32];
let encrypted = sea::encrypt_symmetric(&json!({"secret": "message"}), key_bytes).await.unwrap();
let decrypted = sea::decrypt_symmetric(&encrypted, key_bytes).await.unwrap();
# }
```
### User Identity
```rust,no_run
# use beam::sea::User;
# use beam::Node;
# use serde_json::json;
# #[tokio::main]
# async fn main() {
let mut node = Node::new();
let user = User::create("alice", "password123", &mut node).await.unwrap();
// Trust another user's public key
user.trust("bob_pub_key", Some("path/prefix"), &mut node).await.unwrap();
// Grant access to encrypted data
user.grant("bob_pub_key", "bob_epub_key", "path/secret", &mut node).await.unwrap();
// Store an encrypted secret
user.secret(&json!({"api_key": "..."}), "wallet/key", &mut node).await.unwrap();
// Check identity
let _identity = user.is(); // Some(Identity { alias, pub_key, epub_key })
// Zeroize keys and invalidate all clones
user.leave();
node.stop();
# }
```
### Three Data Spaces
| **Public** | Anyone (if `allow_public_space=true`) | Anyone | any (e.g. `"data"`) |
| **User** | Key owner only (signature verified) | Anyone | `~{pub_key}` or `~{pub_key}/...` |
| **Frozen** | Nobody (append-only, content-addressed) | Anyone | `#` (content hash = key) |
When `allow_public_space=false`, the node rejects unsigned puts to public space — only user-signed data (`~{pub}`) and content-addressed data (`#` namespace) are accepted. This matches Gun.js `opt.enforce` semantics.
---
## Storage Backends
BEAM supports three persistent storage backends for the embedded database layer. All implement the same `Actor` trait, so the rest of the codebase is unaware of which one is active. The wire protocol is backend-agnostic — nodes with different storage choices converge via the standard mesh.
### redb (Default)
**What**: Embedded ACID B+tree database, single-writer, fsync on every Put.
**When to use**:
- Single-node deployments
- Low-to-moderate write throughput
- You want the most mature, stable option
- You don't want to think about it
**Trade-offs**:
- ✅ Battle-tested, single-crate, well-understood
- ✅ fsync before ack = bulletproof durability — data survives power loss
- ✅ Best read performance (mmap'd B+tree = direct memory access)
- ❌ Single-writer serialization limits concurrent write throughput
- ❌ Not ideal for high-fanout mesh workloads
- ❌ Every Put = fsync (milliseconds, blocking)
### fjall (Recommended for Multi-Node)
**What**: Embedded LSM-tree (RocksDB-like) database in 100% safe Rust. WAL journalling with background compaction and built-in LZ4 compression.
**When to use**:
- Multi-node P2P deployments with high write fanout
- Workloads where peers flood puts during resync
- You want maximum write throughput
**Trade-offs**:
- ✅ 3–4× faster writes than redb (journal append vs fsync per write)
- ✅ Built-in LZ4 compression (free, SSTable-level)
- ✅ WriteBatch — single journal entry for atomic multi-put
- ✅ 100% safe Rust, no unsafe blocks
- ❌ ~1.4× slower random reads than redb (multi-level LSM lookup vs B+tree)
- ❌ Not fsync'd per write — data is crash-safe (WAL) but a power loss may lose recent un-fsync'd writes
- ❌ Background compaction causes read latency variance
**Durability model**: fjall's default matches RocksDB — writes are crash-safe via WAL (survive process crash), but not fsync'd to disk until explicit `persist()`. For a P2P database where peers hold copies of the data, this is the correct trade-off: if one node loses its WAL on power failure, peers resync it. `Flush` triggers `persist(SyncAll)` for full durability.
**Benchmarks** (see [`bench/RESULTS.md`](bench/RESULTS.md)):
| write_storm (sequential) | ~977 elem/s | ~3,000 elem/s |
| concurrent_write_storm (4 tasks) | ~1,195 elem/s | ~4,836 elem/s |
| read_storm (random) | ~610 elem/s | ~447 elem/s |
### Persy (Opt-In)
**What**: Embedded segment-based store with per-transaction isolation and optional `background_ops` fsync offloading.
**When to use**:
- Workloads where many writers hit disjoint keys simultaneously
- You're benchmarking and Persy shows wins on your data
**Trade-offs**:
- ✅ Multiple writers proceed in parallel on disjoint keys
- ✅ Optional `background_ops` for fsync offloading
- ❌ Younger ecosystem, fewer Stack Overflow answers
- ❌ Author has acknowledged crash-safety issues; development has slowed
- ❌ No WASM path (native-only)
- ❌ Performance characteristics need your own benchmarks
### Comparison Summary
| **Structure** | B+tree | LSM-tree | Segment store |
| **Write path** | fsync per Put | WAL journal append | Per-tx isolation |
| **Durability** | Bulletproof (fsync) | Crash-safe (WAL), not power-safe | Per-tx |
| **Read speed** | Fastest (mmap) | Slower (multi-level) | Moderate |
| **Write speed** | Slowest (fsync) | Fastest (journal) | Moderate |
| **Concurrency** | Single-writer | Multi-writer | Multi-writer |
| **Compression** | None | LZ4 (free) | None |
| **WASM** | No | No | No |
| **Maturity** | Most mature | Active dev | Slowing dev |
| **Best for** | Single-node | Multi-node P2P | High-concurrency |
### Selection
Storage backends are **build-time** features, not runtime flags:
```bash
# Default build — redb only
cargo build --release --bin beam
# With fjall support
cargo build --release --bin beam --features fjall
# With Persy and/or fjall support (enables migration subcommand)
cargo build --release --bin beam --features persy
cargo build --release --bin beam --features fjall
# Run with redb (default)
cargo run --release --bin beam -- start --port 4944
# In-memory only (no persistence)
cargo run --release --bin beam -- start --port 4944 --memory-storage true
```
**Library usage** — use any backend programmatically (requires corresponding feature flags):
```rust,ignore
use beam::adapters::{FjallStorage, RedbStorage, MemoryStorage};
// fjall (recommended for multi-node, requires --features fjall)
let storage = FjallStorage::new_with_config(Config::default(), "beam.fjall");
// redb (default, best for single-node)
let storage = RedbStorage::new_with_config(Config::default(), "beam.redb", None);
// in-memory (ephemeral)
let storage = MemoryStorage::new();
```
### Migration Between Backends
The `beam migrate` subcommand converts between all supported storage formats (requires `--features persy` and/or `--features fjall`):
```bash
# Preview without writing
beam migrate --from redb --to persy --source ./data.redb --target ./data.persy --dry-run
# redb ↔ persy
beam migrate --from redb --to persy --source ./data.redb --target ./data.persy
beam migrate --from persy --to redb --source ./data.persy --target ./data.redb
# redb ↔ fjall (fjall uses a directory path, not a file)
beam migrate --from redb --to fjall --source ./data.redb --target ./data.fjall
beam migrate --from fjall --to redb --source ./data.fjall --target ./data.redb
# fjall ↔ persy
beam migrate --from fjall --to persy --source ./data.fjall --target ./data.persy
beam migrate --from persy --to fjall --source ./data.persy --target ./data.fjall
# Overwrite existing target
beam migrate --from redb --to persy --source ./data.redb --target ./data.persy --force
# Custom batch size (default: 1000)
beam migrate --from redb --to persy --source ./data.redb --target ./data.persy --batch-size 5000
```
Migration uses a reader/writer architecture with a canonical intermediate format — each backend has one reader and one writer. Adding a new storage backend requires only two functions, not N² pairwise paths. See `docs/migrations/migration-guide.md` for the full procedure including rollback.
### Mixed Meshes
Nodes with different storage backends interoperate transparently. A redb node, a fjall node, a Persy node, and an in-memory node form a valid mesh. The wire protocol carries the data; storage is a local choice.
**Cross-backend mesh verified** by `tests/cross_backend_mesh_e2e.rs`: 2 redb nodes + 1 Persy node converge correctly under the standard Put/Get protocol.
### Known Limitations
- The `beam_meta_v1` metadata table from redb (last-write timestamps) is not preserved when migrating redb → Persy. This metadata is not currently used by the actor framework, so the loss is cosmetic.
- The migration tool is single-threaded per batch. For datasets larger than ~100k records, run during a maintenance window.
- fjall uses a directory path for storage (LSM-tree), while redb and Persy use single files. Migration involving fjall creates a directory at the target path.
---
## Wire Protocol
BEAM uses Gun.js's JSON wire format. Messages are JSON objects with these fields:
### Put
```json
{
"put": {
"node/id": {
"_": { "#": "node/id", ">": { "child_key": 1653465227430 } },
"child_key": "value"
}
},
"#": "msg_id_8chars",
"##": 123456789,
"><": "peer1,peer2"
}
```
| `put` | Map of node_id → {metadata, child values} |
| `_` | Node metadata: `#` = soul (node ID), `>` = child timestamps |
| `#` (top-level) | Message ID (8-char random, used for dedup) |
| `##` | Content checksum (Java `hashCode` of `put` body) |
| `><` | Peer hop list (anti-loop: comma-separated peer IDs already visited) |
| `@` | Ack ID — if present, this Put is a response to a Get with this ID |
### Get
```json
{
"get": { "#": "node/id", ".": "optional_child_key" },
"#": "msg_id_8chars"
}
```
### Other Messages
- `{"dam": "hi", "#": "peer_id"}` — peer introduction
- `{"dam": "flush", "#": "flush_id"}` — flush storage to disk
- `{"dam": "rtc", "id": "...", "offer": "...", "answer": "...", "candidate": "..."}` — WebRTC signaling
---
## Configuration
### CLI Flags
| `--config` | — | — | Custom config file path |
| `--port` | `PORT` | 4944 | WebSocket server port |
| `--ws-server` | `WS_SERVER` | true | Enable WebSocket server |
| `--cert-path` | `CERT_PATH` | — | TLS certificate path (enables WSS) |
| `--key-path` | `KEY_PATH` | — | TLS private key path |
| `--peers` | `PEERS` | — | Comma-separated peer WebSocket URLs |
| `--multicast` | `MULTICAST` | false | Enable UDP multicast LAN discovery |
| `--memory-storage` | `MEMORY_STORAGE` | false | Use in-memory storage (ephemeral) |
| `--redb-storage` | `REDB_STORAGE` | true | Use redb persistent storage |
| `--redb-path` | `REDB_PATH` | `beam.redb` | Path to redb database file |
| `--allow-public-space` | `ALLOW_PUBLIC_SPACE` | true | Accept unsigned writes to public space |
| `--shutdown-timeout` | `SHUTDOWN_TIMEOUT` | 30 | Graceful shutdown timeout (seconds) |
### Migrate Subcommand Flags
| `--from` | Yes | Source backend: `redb`, `persy`, or `fjall` |
| `--to` | Yes | Target backend: `redb`, `persy`, or `fjall` |
| `--source` | Yes | Path to source database (file for redb/persy, directory for fjall) |
| `--target` | Yes | Path to target database (file for redb/persy, directory for fjall) |
| `--batch-size` | No | Records per batch (default: 1000) |
| `--force` | No | Overwrite target if it already exists |
| `--dry-run` | No | Preview without writing |
### Programmatic Config
```rust
# use beam::Config;
# fn main() {
let config = Config {
allow_public_space: false,
my_pub: Some("x.y".into()),
broadcast_buffer_size: 4096,
ice_servers: vec!["stun:stun.l.google.com:19302".into()],
dedup_capacity: 100_000,
mailbox_capacity: 65536,
child_mailbox_capacity: 256,
};
# }
```
---
## Graceful Shutdown
BEAM performs a graceful shutdown when it receives SIGINT (Ctrl-C) or SIGTERM,
ensuring data integrity before exit.
### Shutdown Sequence
1. **Flush storage** — `Node::flush_storage()` sends a `Flush` message through
the router. Since the router processes messages in FIFO order, all pending
writes ahead of the flush are committed by the storage adapters before the
flush ack returns. Both redb and persy commit inline within `handle()`, so
data is durable by the time the ack arrives.
2. **Signal child tasks** — A `tokio::sync::watch` channel broadcasts `true` to
all child tasks. Long-running loops (WsServer accept loop, WebRtcPeer signal
processor) `select!` on the shutdown signal and break cleanly.
3. **Drain** — A brief wait (5 seconds) allows in-flight messages to complete
and WebSocket Close handshakes to finish. WsConn sends a WebSocket Close
frame with a 2-second timeout.
4. **Force stop** — `Node::stop()` aborts any remaining tasks and sends stop
signals to all child actors as a fallback.
### Timeout
The `--shutdown-timeout` flag (default: 30 seconds, env: `SHUTDOWN_TIMEOUT`)
bounds the total graceful shutdown time. If the flush and drain don't complete
within this duration, the node force-stops and exits.
### Double Signal
A second SIGINT or SIGTERM during shutdown exits immediately with code 1.
### Programmatic Shutdown
```rust
# use std::time::Duration;
# use beam::Node;
# #[tokio::main]
# async fn main() {
let mut node = Node::new();
// ... use node ...
match node.shutdown(Duration::from_secs(30)).await {
Ok(()) => println!("graceful shutdown complete"),
Err(e) => eprintln!("timed out: {}, force-stopped", e),
}
# }
```
---
## Testing
```bash
# Run all tests (includes doctests — README code examples are compiled and run)
cargo test
# With WebRTC tests
cargo test --features webrtc
# With fjall storage tests
cargo test --features fjall
# With Persy tests (includes redb↔persy migration tests)
cargo test --features persy
# With fjall tests (includes redb↔fjall migration tests)
cargo test --features fjall
# With both (includes all migration path tests)
cargo test --features fjall,persy
# Lint (zero warnings required)
cargo clippy -- -D warnings
# Doctests only (verifies README code examples compile)
cargo test --doc
cargo bench
# Run a specific integration test
cargo test --test integration websocket_sync_over_relay_peer
# Wire compatibility tests (all 3 layers)
cargo test --test wire_tests # Layer 1: golden JSON fixtures
cd tests/wire-mirror && npm test # Layer 2: Node.js mirror against real Gun.js
cargo test --test wire_live -- --ignored # Layer 3: live integration (needs Node.js)
```
### Integration Test Categories
| `it_doesnt_error` | Node creation, basic get — no panics |
| `first_get_then_put` | Subscribe-then-write ordering |
| `first_put_then_get` | Write-then-subscribe with storage replay |
| `once_returns_value_or_none` | Read-after-write consistency, Null vs absent |
| `connect_and_sync_over_websocket` | Two-node mesh sync over WS (direct) |
| `websocket_sync_over_relay_peer` | Three-node sync via relay (1 hop) |
| `websocket_sync_over_2_relay_peers` | Four-node sync via 2 relays (2 hops) |
| `redb_storage_persists` | Data survives restart with redb storage |
| `redb_storage_flush_returns_ok` | Flush ack protocol |
| `cross_backend_mesh_e2e` | 2 redb + 1 Persy nodes converge correctly |
| `fjall_e2e` | 6 fjall storage tests: put-get, sequential, nested, LWW, flush, isolation (`--features fjall`) |
| `wire_tests` | 36 golden JSON fixtures — wire protocol spec as tests |
| `wire_live` | Live BEAM ↔ Gun.js bidirectional sync (4 scenarios) |
---
## Benchmarks
BEAM includes a comprehensive benchmarking suite covering relay throughput,
micro-benchmarks for hot-path components, and storage performance.
### Local Put Throughput
Local (non-relay) puts through the actor pipeline — measures the full
`Node::handle → Router::route → MemoryStorage::apply` path with no network I/O:
| 1 sender × 10k | 10,000 | ~24,000–53,000 puts/sec |
Throughput varies with system load. On a dedicated machine with no
competing processes, expect 50,000+ puts/sec. The bottleneck is
`Value` cloning for broadcast channels — further gains require
`Arc<Value>` to make cloning a refcount bump.
### Relay Throughput
Real WebSocket connections through a memory-only relay (no disk I/O):
| 1 sender × 10k | 10,000 | ~5,300 msgs/sec |
| 1 sender × 50k | 50,000 | ~10,600 msgs/sec |
| 10 senders × 5k | 50,000 | ~11,400 msgs/sec |
The relay's internal processing (parse + dedup + route + serialize) runs in
microseconds — the bottleneck is client-side `put().await`, not the relay.
### Micro-Benchmarks (Criterion)
| Parse small Put JSON | 1,067 ns |
| Parse medium Put JSON | 2.00 µs |
| Serialize small Put JSON | 69 ns |
| Parse Get | 677 ns |
| Dedup check (fresh) | 274 µs |
| Dedup check (duplicate) | 41.8 µs |
| Actor mailbox send+recv | 309 µs |
### WASM Benchmarks (Node.js)
| Parse small Put | 7.9 µs | 1,067 ns | ~7.4× |
| Serialize small Put | 8.6 µs | 69 ns | ~125× |
| Parse Get | 4.5 µs | 677 ns | ~6.7× |
Run with: `wasm-pack test --node --no-default-features -- --nocapture`
### WASM Relay Throughput (Browser-only)
`web_sys::WebSocket` callbacks don't fire in Node.js `wasm-bindgen-test-runner`
([known limitation](https://github.com/wasm-bindgen/wasm-bindgen/issues/4921)).
Use the browser benchmark page to measure WASM relay throughput:
```bash
cargo run -- start --port 4944 --memory-storage true --redb-storage false
python3 -m http.server 8080 -d examples/
# Open http://localhost:8080/bench.html in a browser
```
Previous v0.11.0 browser results: ~115–651 msgs/sec depending on batch size.
### Browser Benchmark
An interactive benchmark page is available at `examples/bench.html`:
```bash
# Start a relay
cargo run -- start --port 4944 --memory-storage true --redb-storage false
# Serve the benchmark page
python3 -m http.server 8080 -d examples/
# Open in browser
open http://localhost:8080/bench.html
```
The browser benchmark measures:
- **Relay TPS**: end-to-end throughput through a real relay
- **Put throughput**: local WASM API fire-and-forget puts
- **Get throughput**: local WASM API promise resolution
- **Put→Get round-trip**: full local cycle
### Running Benchmarks
```bash
# Relay throughput (release mode required)
cargo test --release --test relay_throughput_bench -- --ignored --nocapture
# Micro-benchmarks (hot-path components)
# Storage benchmarks (redb only by default)
# Storage benchmarks with fjall (head-to-head comparison)
# Storage benchmarks with persy
# Live metrics endpoint (while relay is running)
curl http://localhost:8080/metrics
```
See [`benches/RESULTS.md`](benches/RESULTS.md) for full results with
methodology and analysis.
## Features
| `webrtc` | No | `dep:str0m`, `dep:stun` — direct P2P connections via WebRTC data channels |
| `fjall` | No | `dep:fjall` — LSM-tree storage backend (recommended for multi-node deployments) |
| `persy` | No | `dep:persy` — Persy storage backend for high-concurrency workloads |
Without `webrtc`, the `stun` module and `WebRtcPeer` adapter are stubbed out (functions return `None`). Without `persy`, the `PersyStorage` adapter is not compiled in and migration to/from Persy is unavailable. Without `fjall`, the `FjallStorage` adapter is not compiled in and migration to/from fjall is unavailable. Migration requires at least one of `persy` or `fjall` features.
**WASM**: When targeting `wasm32-unknown-unknown`, native-only modules (redb, fjall, Persy, tokio-tungstenite, multicast) are cfg-gated out. Browser adapters (`wasm_ws`, `wasm_idb`) are compiled in. Timer functions (`sleep`, `timeout`, `interval`) are provided by `tokio_with_wasm` via the `tokio_time` shim module instead of tokio's `time` feature (which panics on WASM). The `wasm.rs` module provides `#[wasm_bindgen]` JavaScript bindings. Build with `wasm-pack build --target web --release`.
---
## Security
BEAM's SEA layer provides Ed25519-compatible signing (P-256 ECDSA via `ring`), X25519 ECDH key exchange, and AES-256-GCM authenticated encryption. Keys are zeroized on `leave()`. Session storage supports encrypted file persistence.
The `allow_public_space` flag controls whether unsigned writes to public nodes are accepted. Set to `false` to enforce that all data must be either user-signed (`~{pub_key}` prefix) or content-addressed (`#` prefix).
For security vulnerabilities or responsible disclosure, please open a GitHub issue or contact the maintainers directly.
---
## Contributing
PRs welcome. Read [COMPASS.md](COMPASS.md) for architecture context and [DEPLOY.md](DEPLOY.md) for operations guidance.
```bash
# Clone and build
git clone https://github.com/guan-tends/beam.git
cd beam
cargo build
# Before submitting a PR
cargo clippy -- -D warnings
cargo test
```
---
## Maintainers
- **Guan** — development
- **David Newman** — maintenance
---
## Credits
BEAM is a maintained and featureful fork of **[rod](https://github.com/mmalmi/rod)**, originally created by [Martti Malmi](https://github.com/mmalmi) as a from-scratch Rust port of [Gun.js](https://github.com/amark/gun) by Mark Nadal. Malmi wrote rod; BEAM is developed by Guan and David Newman (2026–present). The original Gun.js project is maintained by Mark Nadal.
BEAM builds on rod's foundation with substantial additions: SEA crypto layer (P-256 key generation, signing, verification, ECDH, AES-256-GCM, capability certificates, user system with session persistence), WebRTC P2P transport, persistent storage adapters (redb, Persy) with migration tooling, DAM protocol parity, network fanout ack/quorum, observability, and comprehensive wire compatibility testing against Gun.js. See [CHANGELOG.md](CHANGELOG.md) for the full contribution history.
Deep gratitude to Martti Malmi for rod and to Mark Nadal for Gun.js itself — a visionary approach to decentralized data.
---
## License
MIT — see [LICENSE](LICENSE).
---
## Sponsors
If BEAM saves you or your company time, consider sponsoring ongoing maintenance, dependency updates, and issue triage.
### Donate
| **Solana** | `Eu8wQcW68TKMs1a6eqzZu8znzU52QLqQugAMG8uCD6y6` |
| **Ethereum / EVM** | `0x2733ff7c865C56d565a99BE1DC11B81cc76850A5` |
| **XRP Ledger** | `r4X6e7McAQj7e8vBCeued1RYu4mCJrREDG` |