batpak

Sync-first event sourcing for Rust: append-only segments, causal metadata, policy gates, and typed projections — no async runtime.

cargo add batpak

Choose batpak when you want an embedded event log with typed payloads, causal metadata, policy gates, and projections in one Rust process. It is a library substrate, not a hosted database: callers own the process model, disk placement, and integration boundaries.

Mental Model

coordinate → event → guard → pipeline → store

coordinate: an (entity, scope) pair. Every event lives at a coordinate.

event: a typed payload sealed with a UUID v7 ID, HLC timestamp, per-entity clock, and Blake3 hash chain link.

guard: a Gate evaluates a Proposal and issues a Receipt or Denial. A GateSet composes gates. Evaluation is fail-fast by default.

pipeline: Pipeline::evaluate runs the gates; Pipeline::commit persists through a caller-supplied closure. The Receipt is the unforgeable proof gates passed.

store: the persistence engine — append-only segments, in-memory index, background writer thread, projections, subscriptions.

One Event Through The System

store.append_typed(&coord, &payload) — where payload is any #[derive(EventPayload)] struct — serializes the payload to MessagePack, wraps it in an Event<Vec<u8>>, and sends it as a WriterCommand::Append through a one-shot flume channel. The calling thread parks waiting for the writer's response. (The underlying raw surface, append(&coord, kind, &payload), still exists for callers that compute EventKind dynamically.)

The writer thread receives the command and calls WriterState::handle_append(), which executes a ten-step commit protocol: reads the entity's latest IndexEntry from the in-memory DashMap; runs CAS and idempotency checks; computes prev_hash from the latest entry, or uses the genesis [0u8; 32] for first-ever events; advances the per-entity clock; sets the HLC wall-clock position monotonically; computes the Blake3 event hash chained to prev_hash; encodes the wire frame as [len:u32 BE][crc32:u32 BE][MessagePack FramePayload]; rotates the active .fbat segment if the size threshold is crossed, sealing it and writing the SIDX footer; writes the frame to the active segment; and inserts the IndexEntry into all index structures, calls index.publish(global_seq + 1) to make it visible to readers, then broadcasts a Notification to subscribers.

The event now lives in three index structures: a per-entity BTreeMap<ClockKey, Arc<IndexEntry>> ordered by HLC then clock, an O(1) by_id DashMap, and the latest chain head. Readers access it via store.get(id) (index lookup then disk read), store.query(&region) (index scan, no disk I/O), or store.stream(entity) (BTreeMap range scan, no disk I/O).

Store Internals At A Glance

Eight subdirectories organize the store by concern. Flat files alongside them (append.rs, config.rs, error.rs, fault.rs, gate.rs, hidden_ranges.rs, lifecycle.rs, reactor_typed.rs, stats.rs) hold types that belong to the store root and don't fit neatly into one subdirectory.

store/
├── write/        control corridor + writer rooms (append, batch, fence runtime, publish, runtime)
├── segment/      on-disk .fbat frame format and SIDX footer
├── index/        in-memory query engine: streams, by_id, columnar overlays, interner
├── cold_start/   open/restore: mmap → checkpoint → SIDX rebuild → frame scan
├── platform/     target-sensitive fs/sync/lock/clock/mmap helpers
├── projection/   state reconstruction: replay, cache, watcher
├── ancestry/     causal graph walking: by hash chain or by HLC clock
└── delivery/     push subscriptions (lossy) and pull cursors (ordered)

Public Surface

Typed payload binding — #[derive(EventPayload)] on a named-field struct binds the Rust type to its EventKind at compile time. Every typed write/read sibling below infers the kind from T::KIND, so callsites never write EventKind::custom(...) directly.

Append — append_typed, append_typed_with_options, append_reaction_typed, append_batch (with BatchAppendItem::typed), apply_transition (with Transition::from_payload). Each returns an AppendReceipt. Non-blocking variants via submit_typed / try_submit_typed return an AppendTicket you .wait() later. The raw append, append_reaction, submit, try_submit, append_with_options still exist for callers computing EventKind at runtime.

Query — stream(entity), by_scope(scope), by_fact_typed::<T>(), query(&region), get(event_id), walk_ancestors(id, limit). All return from the in-memory index; only get and walk_ancestors read from disk. by_fact(kind) remains for dynamic-kind lookups.

Projection — project::<T>(entity, &freshness) folds events into any type implementing EventSourced. project_if_changed skips work when nothing changed. watch_projection returns a ProjectionWatcher that re-projects on subscription events from a lossy/prunable watcher canal. Two replay lanes: JsonValueInput (default, ergonomic) and RawMsgpackInput (perf).

Delivery — subscribe_lossy(&region) for push-based broadcast (may drop under load). cursor_guaranteed(&region) for process-local pull-based ordered replay from the in-memory index. Durable at-least-once across restarts is exposed by cursor_worker(..., CursorWorkerConfig { checkpoint_id: Some(CheckpointId::new(..)), .. }) and typed reactors via ReactorConfig::checkpoint_id: Option<CheckpointId>. Checkpoint-backed handlers receive Some(&AtLeastOnce) for exactly-once composition with a caller-supplied IdempotencyKey; process-local handlers receive None. Cursor::with_gap_config(...) plus Cursor::take_gaps() expose in-memory write-to-deliver gap observations without introducing a persisted system event. react_loop is the legacy subscribe-based loop.

Control plane — submit/try_submit for non-blocking fire-and-ticket. outbox() for staged batch assembly. begin_visibility_fence() for atomic write groups. open, close, sync, snapshot, compact for lifecycle. stats() and diagnostics() for observability. diagnostics().open_report exposes the structured cold-start receipt, StoreConfig::with_open_report_observer(...) lets callers export it, and mutable opens append one durable SYSTEM_OPEN_COMPLETED lifecycle event at batpak:store / batpak:lifecycle. The batpak: coordinate prefix is reserved for library-owned lifecycle streams; application code should avoid it. Receipt signing is opt-in via StoreConfig::with_signing_key(...); signed AppendReceipt and DenialReceipt values carry key_id and signature, and verify_append_receipt / verify_denial_receipt re-check them against the store's configured key registry. Gate denials can be persisted as first-class entity-chain events through Store::append_denial(...) using EventKind::SYSTEM_DENIAL.

Commands

Testing Doctrine

HARNESS_DIRECTIVE.md defines the five harness patterns used to classify doctrine-bearing test suites and the module-header rule for new harnesses. cargo xtask structural now enforces the ledger schema, module-header rule, and 500-line split discipline for ledger-listed harnesses, with explicit capped legacy debt entries. HARNESS_LEDGER.md records the current canonical witnesses, including derive compile-fail/parity, deterministic concurrency, chaos, fuzz-chaos feedback, perf gates, and cold-start/replay consistency. cargo xtask mutants policy prints the repo-owned mutation thresholds and critical seams without running cargo-mutants.

What This Is Not

No async runtime in production. No tokio, no async-std, no futures in [dependencies]. Async callers integrate at the edges via spawn_blocking or flume's recv_async.

No product or domain concepts. No users, orders, accounts, or payments in the library. Only coordinates, events, gates, pipelines, and the store.

No external database substrate. Segments are native coordinate-addressed append logs. No LMDB, no redb, no SQLite.

No concurrent owners. One live Store handle owns the directory lock at a time. The directory lock is exclusive-only: a second mutable open or a concurrent read-only open fails with StoreLocked instead of racing the same store directory.

No per-entry integrity. Each frame carries a CRC32. Cold-start artifacts carry a full-file CRC. There is no per-byte or per-field checksum beyond that.

No mixed-version concurrent operation. Stop all writers before upgrading. Different binary versions must not share an open store simultaneously.

See GUIDE.md for human-first workflows and usage patterns. See REFERENCE.md for the full technical reference and invariant catalog.

License

MIT OR Apache-2.0