Module file 

.verit — the Veritate file: many messages in one self-contained, mmap-able, appendable file.

Normative definition: docs/Architecture/VERIT - File Format Specification.md. The rationale is docs/decisions/ADR-0002. This module implements that document; where the two disagree, the document wins.

┌─ header (32 B, written once) ───────────────────────────────────────────┐
│ "VRTF" · version · required_features · optional_features                │
├─ generation 1 ──────────────────────────────────────────────────────────┤
│  records · schema section (VRSB) · index · footer (64 B)                │
├─ generation 2 (append / removal — appended, nothing above is rewritten) ┤
│  new records · schema section · index · footer (64 B)  ← authoritative   │
└─────────────────────────────────────────────────────────────────────────┘

Four properties fall out of that shape:

• Self-contained. The schema section holds a schema for every id in the index, so a .verit file plus nothing else is readable. Records are hash-only, so each schema is stored once no matter how many records use it.
• Crash-safe without a journal. A footer is authoritative only once fully written and CRC-valid. A crash mid-append leaves the previous footer intact, so the file reads exactly as it did before. FileView::open finds the newest valid footer by scanning back from the end.
• Snapshot reads with no locking. Committed bytes are never rewritten, so a reader holding a footer has a stable view while a writer appends. One writer, many readers.
• Stable identity. Every record carries a monotonic u64 record id that is never reused and survives removal and compaction. A position is not an identity — it shifts under both — so anything that remembers a record across commits (a consumer checkpoint, a sync cursor) must remember its id. Because ids ascend with the index, FileView::find_by_id is a binary search over bytes already loaded, and FileView::records_after gives a tailing reader “everything since my checkpoint” for free.

Two entry points, sharing one commit-image implementation so they cannot drift:

• FileBuilder builds a complete generation-1 file in memory, returning Vec<u8>. No I/O, so it is what the golden-file corpus and the ports’ conformance suites are written against.
• FileWriter owns a std::fs::File and performs incremental, crash-safe commits — append, remove_id, commit, compact.

Reading stays zero-copy and dependency-free: FileView::open takes a &[u8], so mmap the file with your platform’s facility (or std::fs::read it) and pass the bytes. FileView::get returns a sub-slice you hand straight to Message::parse.

let schema = SchemaBuilder::new()
    .add_struct("Point", vec![(1, "x", Dt::I32), (2, "y", Dt::I32)])
    .build("Point")
    .unwrap();

let mut b = FileBuilder::new();
let first = b.append(&schema, &Value::Struct(vec![(1, Value::I32(3)), (2, Value::I32(4))])).unwrap();
b.append(&schema, &Value::Struct(vec![(1, Value::I32(-1)), (2, Value::I32(0))])).unwrap();
let bytes = b.finish().unwrap();

// From here on, pretend we know nothing but `bytes`.
let f = FileView::open(&bytes).unwrap();
assert_eq!(f.len(), 2);
assert_eq!(f.dump_json(0).unwrap(), r#"{"x":3,"y":4}"#);
// A record is found by id, not by position.
assert_eq!(f.find_by_id(first), Some(0));

What this file is not

One file. Retention — “drop the oldest million records” — costs a full compaction here, and no layout inside a single file avoids that. The answer is segmented files: many .verit files under a naming convention, whole segments dropped. That belongs in a layer above the format, and is deliberately not built into it.

Structs

FileBuilder

Build a complete generation-1 .verit file in memory.

FileReader

A .verit file read into memory, owning its bytes.

FileView

A read-only, zero-copy view over a .verit file’s bytes (typically an mmap). open validates the header, footer, schema section, and every index entry up front, so each later get is a bounds-free slice.

FileWriter

A .verit file open for reading and writing, mutated by crash-safe append-only commits.

Footer

A parsed footer — the authoritative statement of a file’s committed state.

Record

One index entry: a record’s identity, where it lives, and which schema interprets it.

Resolvers

One Resolver per distinct schema in a file, built once by FileView::resolvers.

Constants

FILE_HEADER_LEN

Fixed header length, and the alignment every region starts on.

FILE_MAGIC

File magic: “VRTF”. Deliberately not a VRTC version bump — a 0.1.0 container is rejected here structurally rather than partially misread.

FILE_VERSION

File format version this build reads and writes.

FIRST_RECORD_ID

Record ids start at 1, so 0 is available as “no record”.

FOOTER_LEN

Fixed footer length — one cache line, with room reserved for one more field.

INDEX_ENTRY_LEN

Index entry width: u64 id, u64 offset, u64 length, u128 schema_id.

OPT_RECORD_CRC

optional_features bit 0 — the file carries a CRC-32 per record.