eventsdb

An append-only event store that lives in your process, on SQLite.

It keeps what an event store is for — immutable facts, per-stream ordering, decisions taken inside the write, schema evolution without rewriting stored bytes — and drops what needs a cluster.

Status

Early. The write path, the global order, cross-stream reads, subscriptions, consumer checkpoints, exactly-once projections and retention are implemented and tested.

Crates

crate	what it is
eventsdb	one name for both: re-exports eventsdb-core, and eventsdb-sqlite behind the sqlite feature
eventsdb-core	the envelope contract, schema versioning, the two traits, and an in-memory backend
eventsdb-sqlite	the durable backend: one file, one writer thread, WAL — plus projections, which need the same connection to be exactly-once

Taking the two directly is the same code; the facade adds nothing but the name. Everything below names the crates the items come from.

The shape of an event

Three keys are yours and three belong to the store:

{
  "kind": "order_placed",          // required, yours
  "meta": { "customer": "c-12" },  // optional, yours — shallow, scalars only
  "data": { "total": 40 },         // optional, yours — any depth

  "seq": 4,                        // the store's: per-stream, from 1
  "epoch_ms": 1757000000000,       // the store's
  "_schema_version": 1             // the store's
}

Any other top-level key is refused. That refusal is the point: the envelope is a stable contract whose keys are columns, meta is readable without knowing the kind, and data is the one place a kind's own shape lives. A SQL view built on the envelope is not broken by a kind changing shape.

kind is an opaque string here. Which kinds exist and what their data must contain belong to the layer above.

Time is a coordinate, not an order

Every event carries a time coordinate (epoch_ms) and log positions (seq, position). The positions order the log; the coordinate never does. Every read is ORDER BY seq or ORDER BY position — nothing sorts by time.

Which moment the coordinate names is decided by the call that wrote it, and there are three:

written by	epoch_ms is
append	the wall clock of that write
append_at — backfill	the moment the change happened where it came from
import — transfer	whatever the source log recorded, unchanged

There is no fourth field recording which: the verb says it. A field would have to be trusted to be accurate; a verb cannot be wrong about itself.

The consequence worth knowing: a log holding backfilled or imported history has a coordinate that is not non-decreasing in position order, so Plan::OlderThan removes a scattered set rather than a prefix. That is safe — the watermark already handles scattered removals — but it is a different answer than the same call gives on an append-only log. Backfilling into an empty stream in chronological order keeps the non-decreasing property by construction, and is the shape to prefer.

Reads do not queue behind writes

One writer thread owns the connection that appends. Reads are served from their own read-only connections (two by default, OpenOptions::readers), so a statement does not wait for a write transaction to finish — which is the whole point of WAL, and was being given away when everything ran on one thread: a read measured at 73µs idle took 8.2 seconds with one write open. It is now 367µs [measured, tests/readers.rs].

A reader sees committed data, which is the right answer for a standalone read and the wrong one inside a write transaction — so three read paths stay on the writer, each because a reader would break it:

path	why it cannot move
append_if's decision	must see the stream under the lock, or it is the compare-and-swap it exists to avoid
TxnContext::read	must see what the same closure already appended, which is uncommitted
a projection's batch	must share a transaction with the completeness check, or retention lands between them

They hold the write lock while they read, so other writes wait for them — other reads do not. A decision over 20 000 events held the lock ~780ms while a concurrent read took 536µs. kinds is the control and the difference is not marginal: the same decision was 706ms reading every kind and 1.05ms naming the one it folded. For projections the knob is with_batch.

An in-memory log has no readers, because each :memory: open is a separate database.

Two things a networked event store cannot give you

Positions have no holes. The backend allocates a global position inside the transaction that commits it, and IMMEDIATE means that transaction holds the write lock from BEGIN — so allocation order and commit order cannot diverge, and a reader never sees n + 1 while n is still uncommitted. Following the log is a range read: no gap detection, no grace window. This holds across connections too, which is measured rather than assumed (tests/two_logs.rs).

A projection can be exactly-once. When the read model lives in the same SQLite file as the log, applying an event and advancing the consumer's checkpoint are one transaction. No dedupe table, and no idempotence requirement on the projection author.

Both come from the same place: one writer, one transaction. Distributing the store forfeits both, which is why this one does not distribute.

Usage

use eventsdb_core::{EventLog, EventStore, Filter, Position};
use eventsdb_sqlite::SqliteEventLog;
use serde_json::json;

let log = SqliteEventLog::open("events.db").await?;

let mut orders = log.stream_handle("order-1");
orders.append(json!({ "kind": "placed", "data": { "total": 40 } })
    .as_object().unwrap().clone()).await?;

// Read across every stream, in commit order.
let batch = log.read_all(Position::BEGINNING, &Filter::all(), 100).await?;

// Or follow it: catch up, then stay live.
let mut events = log.subscribe(Position::BEGINNING, Filter::all())?;

A command with an invariant

Two shapes, and which one you want depends on where the decision was made, not on whether the check is atomic — both check inside the write.

The decision runs at the write — append_if reads the stream, calls your decision and appends its answer inside one transaction, so the check runs against the stream as it is at that instant rather than a head you cached:

ledger.append_if(Some(&["granted", "spent"]), Box::new(|seen| {
    let balance = fold_balance(seen);
    (balance >= 4).then(|| spend(4))
})).await?;

Prefer this wherever it fits. It folds the stream rather than comparing one number, so it produces no false conflicts; a decision that finds nothing to do returns None and is idempotent for free; and there is no retry loop, because there is nothing to retry.

The decision was made before the call, somewhere this process cannot reach — an HTTP client holding an ETag, a form somebody filled in, a message that sat in a queue. Then it is expected-version:

orders.append_expecting(Expected::Seq(4), cancelled()).await?;
// Err(HeadMismatch { expected: Seq(4), actual: Seq(6) })  → HTTP 412

The error carries both coordinates, so the caller folds only what it missed rather than the stream from the start. It is not Busy: nothing is contended, and repeating it unchanged fails the same way.

Expected::Unwritten means nothing has ever been appended here — which is not the same as "the stream reads empty". Retention can empty a stream whose counter stands at 50, and the check is against the counter, so a caller meaning "this is a new order" is not told yes about an order that was archived.

A projection

A projection's apply is handed the transaction the log is being read on, so the fold and the cursor move together:

impl Projection for Totals {
    fn name(&self) -> &str { "totals" }
    fn kinds(&self) -> Option<Vec<String>> { Some(vec!["scored".into()]) }

    fn init(&mut self, tx: &Transaction<'_>) -> Result<()> { /* CREATE TABLE */ }
    fn reset(&mut self, tx: &Transaction<'_>) -> Result<()> { /* DROP TABLE */ }

    fn apply(&mut self, tx: &Transaction<'_>, event: &Recorded) -> Result<()> {
        // write the read model through `tx`
    }
}

let mut runner = log.runner(Totals::new());
runner.init().await?;
runner.catch_up().await?;   // or run_once(), or rebuild()

If apply fails part-way through a batch, neither the read model nor the cursor moves — so the retry neither double-counts nor skips. Writing the read model anywhere other than that transaction gives the guarantee up.

Moving a log

A log that cannot be moved is a log its owner cannot leave, and any system that already has one has to be able to bring it. Export and import are part of the store, not an afterthought:

// Page it out. Feed the last position back in for the next batch.
let batch = log.export(Position::BEGINNING, &Filter::all(), 1000).await?;

// And back in, in one transaction.
let report = target.import(batch).await?;
assert!(report.reproduced_coordinates);

Each record is one JSON object (ExportedEvent::to_json / from_json), so a file of them is JSON Lines and needs no format of its own.

What travels: kind, meta, data, epoch_ms and _schema_version — the whole stored object. What is reassigned: seq and position, because those are allocations of the receiving store.

Keeping _schema_version is the half that matters. An old event re-stamped as current falls out of reach of the upcaster written for it, and is then read as a shape it never had — a migration that looks like it worked and did not. The export is also not upcasted, for the same reason: it gives you the bytes, so the receiving store holds what this one held and runs its own chain.

reproduced_coordinates is the check that the copy is the same log rather than merely the same events. It is true for an in-order import into an empty store, which is what a migration is, and false when merging into a store that already has history — stated rather than left to assume.

Both calls are on the EventLog trait, so a migration is written once against the trait rather than once per backend. ExportedEvent is backend-neutral, and a log that cannot do this declines rather than offering a partial import: a transfer that stopped half way is worse than one that refused, because from the outside there is no way to tell how far it got.

The escape hatch

Without one, anyone needing a query the API does not have opens the database file themselves — and a second writing connection is what this store cannot survive. Position order is guaranteed by there being one writer; a second one commits on its own schedule, and a subscriber can pass a position that is still uncommitted. Deleting through a second connection is worse: it removes events with no retention ledger entry, so nothing downstream learns that a fold is now missing its input.

So the hatch is not a convenience. It is what makes "do not open the file yourself" a reasonable thing to ask.

// Read anything, across the log and your own tables.
let rows = log.query(
    "SELECT stream, json_extract(data, '$.n') AS n FROM events WHERE kind = ?1",
    vec![json!("scored")],
).await?;

// Or take a real transaction on the log's own connection.
log.with_transaction(|tx| {
    tx.execute_batch("CREATE TABLE IF NOT EXISTS my_view (k TEXT PRIMARY KEY)")?;
    tx.execute("INSERT INTO my_view (k) VALUES ('x')", [])?;
    Ok(())
}).await?;

Your tables, your SQL, your schema, committed or rolled back with everything else in that transaction. Inside a projection you already have this — apply is handed the same kind of transaction.

What the hatch refuses, through SQLite's authorizer rather than by reading your SQL: writing events, stream_seq, checkpoints, retention or sqlite_sequence; creating anything that shares one of those names in any schema, TEMP included, since a temp table shadows the real one for every unqualified statement on the connection; attaching another database; setting a pragma. Reading any of them is allowed and often the point, and so is adding your own index to events — that changes no data, and it is the only way to make a read cheap when the shipped indices do not cover what you filter on.

Each refusal is an invariant something else already promised — appends get stamped and ordered by the store, removals leave a ledger, stream_seq keeps seq from rewinding after a removal, user_version belongs to the migration ladder, journal_mode to the concurrency story.

How far that reaches. The authorizer is a property of this API: it is installed on the connection this store owns, for the length of a call, so it covers everything coming through the crate and nothing else. A sqlite3 session on the same file never meets it.

One half of append-only is stronger than that. A trigger in the schema — which every connection that opens the file gets — makes a stored event impossible to rewrite, whoever opened it. Removal is deliberately not covered: retention deletes as its whole purpose, so the same trigger would have to be switched off inside the one transaction allowed to delete, which is the code least worth leaving unguarded.

Benchmarks

cargo bench -p eventsdb-sqlite                        # everything
cargo bench -p eventsdb-sqlite -- decide              # one group
cargo bench -p eventsdb-sqlite -- --save-baseline before
cargo bench -p eventsdb-sqlite -- --baseline before   # compare

Groups: append, read, decide, project, transfer, retention. All of them run against a file-backed log in a temporary directory, because that is what production uses — an in-memory log has no reader connections and no WAL on disk, so its numbers would flatter every path that matters.

What is not benchmarked here, deliberately. Criterion measures steady-state cost: how long an operation takes when nothing is fighting it. "Does a read wait behind a write" is a question about contention — it has one answer rather than a distribution, and a regression there is a defect, not a slower number. Those live as assertions instead:

question	where
how long does this operation take	benches/eventsdb.rs
does a read wait behind a write	tests/readers.rs
does a read see the write that just returned	tests/readers.rs
what does holding the write lock during a read cost	tests/lock_hold.rs
does a second log on one file still order correctly	tests/two_logs.rs

Numbers that should get better go in benches; properties that must not regress stay in tests, where a failure is a failure rather than a slower bar on a chart.

Schema evolution

Two axes, and they are not the same one:

axis	subject	who owns the number	mechanism	marker
event shape	data, meta, the meaning of a kind	the author of the kind	upcaster chain, applied on read	_schema_version
table shape	columns, indices, constraints	this crate	migration ladder, applied at open	PRAGMA user_version

Stored bytes are never rewritten. An upcaster moves the reader forward instead.

The store carries _schema_version; it does not choose it. Whoever owns a kind owns what its data looks like, so they own the number that says which shape it is in — a number defined by this crate's release history would mean nothing to a consumer or to anyone reading an export. Set it on the event, or leave it out and get DEFAULT_SCHEMA_VERSION. This is Axon's arrangement: the revision is declared by the author, persisted in a column beside the type, and absent is a legal value the first upcaster selects on.

Select on (kind, version), never on the version alone — one shared number would mean one author's bump silently bumped everyone else's.

The envelope has a shape too, and that one is the crate's: an upcaster transforms JSON and cannot add a column, so an envelope change is a ladder step. The ladder runs at open before any handle is issued, so a database is homogeneous in envelope shape by the time anything reads it — which is why there is no second per-event number.

Retention

Removal is the one operation that can make a correct-looking read wrong: a fold that starts before a deleted range comes back short, and nothing in the shape of the result says so. So it is not just a DELETE.

// Three shapes. Prefix, age, or whole streams.
log.retain(Plan::Before(Position::new(1000)), Guard::default()).await?;
log.retain(Plan::OlderThan(cutoff_ms), Guard::default()).await?;
log.retain(Plan::Streams(vec!["session-7".into()]), Guard::default()).await?;

log.reclaim().await?;   // give the freed pages back to the filesystem

Two things keep it honest:

• The default guard refuses to overrun a consumer. Guard::RegisteredConsumers fails with ConsumerBehind if any stored checkpoint sits below what the plan would remove. It can only see consumers that have saved a checkpoint, so have yours check in before its first batch. Guard::Force removes anyway.
• What was removed outlives it. Every application writes a row to a retention ledger in the same transaction as the delete, and the highest position removed is a watermark. A projection whose cursor sits below the watermark is refused with Truncated rather than served a short answer, and a rebuild on a truncated log is refused before the old model is emptied. A projection that genuinely does not care — a "last 30 days" view — says so with tolerates_truncation.

Dropping whole streams leaves holes in the global order. That is safe: positions are never reused (AUTOINCREMENT), and nothing waits for a specific one — a subscription reads position > cursor and does not see what is gone.

Limitations

• Subscriptions outside the writing log poll. SQLite has no LISTEN/NOTIFY, so a write anywhere but the subscriber's own log is invisible until someone looks — including a second log opened on the same file in the same process, since the wake-up channel belongs to the log. Nothing is lost, only delayed, by roughly three orders of magnitude [measured: 552µs against 552ms on a 600ms poll].
• Open the file once per process. It is safe not to — the order holds — but two logs mean the polling latency above, and two logs opened with different upcaster chains will read the same bytes differently with nothing to detect it.
• No clustering, replication or network protocol, by design — see above.
• Retention deletes; it does not archive. Taking an export before a retain is what preserves the history — the pieces are here, the policy that decides when to do it is not.
• reclaim needs a database created by this version. It relies on auto_vacuum = INCREMENTAL, which SQLite only accepts before the first table exists. On an older file it does nothing.

Contributing

Three files in the repository, absolute-linked because this README ships inside the crates and they do not: CONTRIBUTING.md has the issue, branch, verification and commit conventions, PUBLIC_DEVELOPMENT.md the disclosure policy that outranks it, and AGENTS.md the same pointers arranged for a coding agent.

License

Licensed under either of Apache License, Version 2.0 or MIT license at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this crate by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.