Skip to main content

Crate turndb

Crate turndb 

Source
Expand description

turndb — a content-addressed columnar store for AI traces.

§The shape

ingest ── WAL (durability) ── memtable
                                 │ flush
                                 ▼
                         part (immutable, columnar)
                                 │ merge
                                 ▼
                         larger parts

       fold ── stable content locations shared by every part

Two ideas fused. Content addressing: a piece of content is identified by BLAKE3 of its bytes and stored exactly once, however many records reference it. Columnar: the typed metadata around that content lives in per-field columns that compress and scan well.

§Why the fusion is worth it

In a conventional LSM, merging rewrites data — merge two 1 GiB parts, write 2 GiB. That write amplification is the tax every tiering strategy exists to manage.

Here, part merging never touches content. Pieces live in the fold, addressed by hash, written once until refold replaces the generation or declared content punch removes an unreachable payload. A part is a bundle of references plus columns, so merging rewrites references and columns and nothing else — on trace-shaped data that is a small fraction of the bytes. Content addressing is what decouples part-merge cost from data volume, and that is what lets a trace store behave like a database instead of a write-once archive.

§The cardinal invariant

Byte-exact reconstruction. Reading a record reproduces the original bytes exactly — including attribute order and duplicate keys. Every layer below is in service of it, and no change ships that breaks it.

Re-exports§

pub use types::AttrValue;
pub use types::Content;
pub use types::ContentHash;
pub use types::ContentOp;
pub use types::PieceHash;
pub use types::Record;
pub use types::BODY_CONTENT;

Modules§

backup
Online backup and restore result types and refusal contracts.
capabilities
Machine-readable build and platform capabilities.
carve
The carve: how a body becomes spans — an opinion with escape hatches, never a lock-in.
container
A store’s artifacts in one mutable file.
control
Cooperative interruption shared by scans and lifecycle operations.
error
Stable, domain-neutral error classification for embedding boundaries.
fold
The fold — an append-only, content-addressed piece store with block compression.
observability
Pull-based, telemetry-neutral operation metrics.
part
A part: an immutable, self-contained, id-sorted columnar slice of the store.
query
The query lens: parts, read a column at a time, as Arrow.
read_limits
Runtime admission for atomic storage frames and persistent object collections.
readat
Positioned reads behind a trait — the seam every future backend goes through.
scan
Storage-native structured paging.
schema
Feature-independent discovery of the store’s self-described field universe.
store
The store: WAL, pending change set, publication, manifest revisions, and WAL replay — the layer that turns a fold and parts into a database.
types
The vocabulary every layer shares.
vfs
Every WRITE-SIDE filesystem operation, behind one thin seam.