turndb
A content-addressed columnar store for AI traces. Embedded, single-writer, no daemon — a store is
a directory you can tar, and reading one needs nothing but the files.
Single-writer enforcement is platform-dependent: the engine enforces it on Unix and cannot on
wasm32-wasip1, where the embedder must guarantee it. See One writer per store under
What it does not do and FORMAT.md for the
normative statement.
Status: pre-1.0, format version 2, not frozen. FORMAT.md is normative: where it and the code disagree, one of them is a bug.
What problem it solves
Agent and chat APIs are stateless, so every request re-sends the whole conversation. A trace store therefore receives the same message text over and over — on a real corpus, 38.9× duplication. General stores pay for those bytes repeatedly; archival dedup stores collapse them but stop being queryable.
turndb splits the two planes:
- the fold holds content, addressed by BLAKE3 of its bytes, written once and never rewritten;
- parts hold record ids, named content programs, and typed attribute columns — references, not bytes.
Where the planes meet is the carve: dedup can only recognise repetition where piece boundaries
land, and full-resend traffic repeats exactly at message boundaries. The engine's default carve is
therefore structural, not content-defined — the elements of a top-level JSON array become pieces,
and the punctuation between them stays inline. Turn k's messages and turn k+1's re-sent
messages resolve to the same pieces, so each message is stored once however many later requests
carry it, and the storage cost of a full-resend conversation drops from quadratic in its turn
count to linear. This boundary choice, not content addressing by itself, is what produces the
dedup ratios reported below. Bodies that are not JSON arrays fall back to content-defined
chunking, and the default is not a lock-in: pick a carve strategy per write, or hand the store
your own spans and bypass carving entirely (src/carve.rs).
Because a part holds no content, compaction never touches content: merging rewrites references and columns, which on trace data is a small fraction of the bytes. That is what lets a trace store behave like a database instead of a write-once archive.
Measured on 40k records of real agent traffic (1.89 GiB of message bodies): 38.9× piece dedup, 308× total on-disk collapse, every record byte-exact. Against a production trace store's own SQLite path on 3,000 full-resend calls — same records, their schema and insert code — 320.24 MiB became 1.56 MiB.
On these numbers. They were measured on corpora that are not public, and they are provisional pending a larger and more rigorous run — our measurements, not independently reproducible results. The ratio also depends entirely on how much your traffic re-sends. The reproducible form of the claim is comparative: point turndb at your own traces and measure.
The cardinal invariant
Byte-exact reconstruction. Reading a record reproduces every named content value and attribute
exactly — including content boundaries, attribute order, duplicate keys, NaN payloads, and -0.0.
Everything else is in service of it.
Try it
# JSONL in: each line needs a "body" string; every other scalar becomes an attribute
|
# If MANIFEST is damaged: validates the newest retained commit and permits no rollback by default
As a library:
use ;
let mut s = open?;
s.put_body?; // carved by the engine's default opinion
s.sync?; // the ACK point — durable from here
s.flush?; // seal into an immutable part
Storing gen_ai traces
examples/genai_dogfood.rs is the working mapping from LLM API calls
to turndb records, with the reasoning behind each choice in its doc comment;
examples/genai_query.rs runs a trace UI's actual reads — a member's
page, a responseId lookup, aggregates — against the result, and checks byte-exact reconstruction
first. The shape:
- Three records per API call —
#system/#input/#output— because a record has one body by design. Each body is the message array verbatim, which is what lets the structural carve above resolve re-sent turns to the same pieces. - Ids are
member/ts/responseId#kindwith the timestamp zero-padded, so ids sort lexicographically into member-then-time order — the access pattern a trace UI actually has — and the front-coded id column stays both compressible and range-scannable. - Attributes are flattened to OpenTelemetry
gen_ai.*semantic-convention names, not stored as nested JSON: token usage becomesgen_ai.usage.*integer columns that SQL can filter and aggregate directly. - Arrays become repeated attributes. turndb preserves duplicate keys in order, so
finish_reasonsround-trips without inventing a join separator that could collide. - Custom fields pass through with inferred types — one column per (key, type), so a deployment adding its own fields needs no schema change anywhere.
What it does
| Durability | WAL with an explicit ACK point; batches replay all-or-nothing; one commit point (the manifest) with a checksummed commit log, snapshots, and explicit recovery |
| Query | Bounded structured paging with Rust-owned cursors plus an optional DataFusion lens and read-only SQL-to-Arrow stream — named content is independently projectable, and metadata queries open zero fold blocks |
| Compaction | Total merge at eight parts plus exact input-part/row/byte-bounded work units; merges provably touch zero content bytes |
| Deletion | Tombstone → settle → re-fold removes content and metadata; punch reclaims dead blocks in place without moving a single offset |
| Integrity | Per-piece BLAKE3 on every read, per-section checksums, footer and TOC chains, manifest-pinned parts, and a scrub that walks every frame |
| Shipping | pack puts a whole store in one file that reads — and answers SQL — identically |
The pack command takes the writer role (see FORMAT.md), settles a recovered WAL, fully verifies its staged artifact, and refuses to replace an output path. Restore likewise verifies before extraction and atomically publishes only to a destination that does not exist; see backup and restore. Manifest recovery likewise takes the writer role — excluding a live writer only where that role is enforced — and validates the complete candidate before publication; rollback past the newest retained commit requires explicit authorization. See manifest recovery.
The query layers are independently selectable: --features columnar --no-default-features provides
the Arrow scan lens without DataFusion, while the default sql feature adds DataFusion over that same
lens. Storage, visibility, and content semantics remain in TurnDB either way.
Store::scan is available without either query feature. It provides id-ordered forward/reverse pages,
typed predicates, selected attributes, named-content metadata or bytes, opaque checked cursors, and a
per-call examination bound. Reconstructed pages default to a 32 MiB content ceiling, can override it
per request, never split a row, and return a cursor before the row that would cross the ceiling.
Named-content metadata includes the BLAKE3 identity of the exact whole value without reconstruction
for version-2 records. Writer scans include the memtable;
ReadStore::scan remains pinned to its manifest snapshot.
Committed rows are projected from physical columns: sibling attribute value/dictionary and named
content program sections remain unopened. See
projected structured scans.
Store::explain_scan and ReadStore::explain_scan run the same request/cursor preparation and report
required versus predicate-only fields, work ceilings, effective bounds, and exact physical rows in
scope before visibility resolution. See structured scan explanation.
Rust embedders can classify rich error chains through the stable, domain-neutral
ErrorClass; the native Node binding exposes the same engine codes through
TurnDbError and adds only its actor-owned BUSY/CLOSED states.
Format version 2 adds exact unsigned u64, arbitrary binary metadata, UTC Unix-nanosecond
timestamps, and explicit null to the existing scalar fields. Missing and null remain distinct, and
bindings never route exact integers through JavaScript number; see
general scalar field types.
Writer opens also carry generic admission policy: worst-case framed-WAL bytes per record and atomic batch, batch member count, and UTF-8 identifier/name bytes. Defaults are 64 MiB, 256 MiB, 4,096, and 4 KiB respectively; Rust, native Node, and portable Node can override them. Complete batches are charged before the first fold mutation, and charging is independent of dedup history. See write admission limits.
Reads independently admit every atomic WAL, part-TOC/section, and fold-block frame before stored or
decoded allocation. Both ceilings default to 512 MiB, are configurable per Rust/native/portable
handle, classify refusal as RESOURCE_EXHAUSTED, and are distinct from cache residency. Writers seal
fold blocks early under strict profiles and refuse oversized part outputs before publication; see
atomic frame read admission.
Persistent collection growth is bounded separately within that same per-open profile: 100,000 directory entries, 100,000 physical WAL frames, and 1,000,000 fold blocks by default. Checks precede directory/vector growth and future writer output, including atomic-batch member frames and sparse fold ids; see persistent object-count admission.
Long-running compaction, verification, punching, refold, backup, and restore operations accept
reusable Rust cancellation tokens and absolute deadlines through their controlled variants. The
native Node methods map these to submission-inclusive timeoutMs and AbortSignal options while
preserving each operation's publication and restart invariants; see
lifecycle cancellation and deadlines.
Incremental compaction accepts simultaneous exact physical input-part, row, and file-byte limits,
reports the executed plan and output bytes, and refuses rather than exceeding an insufficient budget;
see bounded incremental compaction.
Reachability-aware storage inventory separates live, retained-only, and unclassified files, while
compaction and refold expose exact source facts plus explicitly advisory staging estimates; see
maintenance space accounting and preflight.
Live immutable parts can be upgraded one atomic, restartable unit at a time, with retained-snapshot
dependencies and advisory stage space reported separately; see
resumable format migration.
Writer handles expose process-local, monotonic lifecycle outcomes and nanosecond totals through a
telemetry-neutral polling surface; see pull-based operation metrics.
With sql enabled, query::sql::SqlQuery runs positional-parameter SQL against the generic
records table under a configurable DataFusion execution-memory ceiling. DDL, DML, and session
statements are refused. Results are pulled one bounded batch at a time as complete Arrow IPC streams;
bindings therefore transport dynamic columnar results without translating them through JSON or
JavaScript objects. The ownership, snapshot, cancellation, and memory semantics are detailed in
Read-only SQL and Arrow IPC streaming.
Point reads, storage-native structured scans, and DataFusion are held to one versioned-record
reference model by the three-path differential gate.
What it does not do
No daemon, no network, no cluster, no consensus. Scale-out is more stores, not a bigger one. No encryption (the format reserves a flag bit and refuses it — see FORMAT.md). No parity/erasure coding: corruption is detected at every level and repair is the storage layer's job. The parser/binding threat model, completed hardening, and remaining availability risks are recorded in the security review; checksums are integrity evidence, not authentication.
One writer per store. Readers need no lock and see a consistent committed snapshot. On Unix
the engine enforces this with flock, which the kernel releases when the process dies — so a
stale lock cannot outlive its owner. Under WASI there is no advisory locking and the engine
cannot enforce it: the lock file is created and gates nothing, so the obligation is the
embedder's — at most one open writer per store directory, across every process and every WASM
instance. The guest cannot enforce or detect a violation. In four measured
overlapping-writer runs on Node 24, both writers received successful sync() acknowledgements and
one writer's complete record set was silently discarded. The surviving store was internally
consistent and every remaining record was readable, so a clean read or verification cannot prove
that an acknowledged write survived. Other overlap patterns may fail differently.
See FORMAT.md for the normative statement.
Platforms
The exact tested targets, Node majors, capability rules, semantic-version policy, and current prototype/publication status are defined in the support and compatibility policy.
The crate's native build is Unix only — it needs positioned reads, flock, and (for punch)
Linux hole punching.
bindings/node is the native server-side release candidate. Its napi-rs addon gives each open
writer a dedicated Rust actor and bounded queue, exposes Promise-based batch/durability/scan/content
operations with Buffer and exact bigint, and refuses to fall back to WASM when a native artifact
is unavailable. The first prebuilt slice is Linux x86-64 glibc across Node 22, 24, and 26; its
package manifests carry npm's "private": true, which makes npm publish refuse outright, so
publication can only happen through the staged release path. The
native prebuild contract states the exact artifact, install, provenance,
and publication gates.
It also builds for wasm32-wasip1, which is what the turndb npm package
ships: one .wasm, no native addon, no prebuild matrix, no postinstall. A store written by either
build is readable by the other, byte for byte; the two-way executable proof is documented under
cross-runtime compatibility. That target gives up three things,
and the first is the one that matters: no advisory locking (see above), no punch (refold
reclaims the same space by rewriting), and no threads, so compression runs inline. src/sys.rs is
the single place that states what turndb needs from an operating system and what happens where it
isn't there.
Testing
The test suite is large relative to the engine, deliberately: a storage engine that loses data is worthless.
STORM_XOR=
The deterministic simulation harness records every write, fsync, rename, link, unlink, and punch, then replays every crash point under a model where file content and directory entries become durable independently, and asserts the recovered store equals some prefix of the acknowledged writes. Six sweeps cover the write path and every publication protocol — backup, restore, manifest recovery with rollback, hole punching, and format migration — and each sweep prints the exact number of crash states it checked, so the suite reports its own coverage. It has found real bugs, including an ACK backed by a WAL whose directory entry was not yet durable, and a declared hole punch whose partially landed deallocation bricked writer recovery.
The corruption storm mutates every on-disk structure and requires errors, never panics. It
found five parser bug classes, including bounds checks of the form at + n > len that overflow
and therefore pass.
License
Apache-2.0. Copyright 2026 Efficacious, Inc. See LICENSE and NOTICE.