# AGENTS.md
Concise, enduring context for working in `segment-buffer`. Read once, internalize.
## What this is
A single-crate **Rust library** (not a binary): `SegmentBuffer<T>` — a high-throughput **local buffer for cloud sync**. Single-process by design. Spools in-memory batches to zstd-compressed CBOR segment files with ack-based deletion, filename-based crash recovery, configurable durability, and optional encryption. Generic over `T: Serialize + DeserializeOwned + Clone + Send` (`'static` is implied by `DeserializeOwned`, not required by the mutex — see TODO_LIST.md "Investigation"). Extracted from monitor365.
**Product positioning (2026-07-20 reframing):** the cloud is the durable layer; this crate is the local throughput buffer in front of it. The README leads with this. The old framing ("durable bounded queue ... crash recovery first") under-sold the actual target use case and over-promised on durability the code doesn't fully deliver (see [Durability model](#durability-model-shipped-in-v050) below).
## Single-process invariant (enforced since v0.5.0)
**One owner process per buffer directory.** Multiple threads inside that process are supported (MPMC via `parking_lot::Mutex`). Multiple independent processes opening the same directory is **rejected** — they would race on segment filenames, double-deliver, and corrupt `head_seq`/`next_seq`.
- **Mechanism:** `open()` acquires an exclusive `flock` (via `fs4::FileExt::try_lock`) on a `.segment-buffer.lock` sidecar in the directory, fail-fast with [`SegmentError::Locked`] if another process holds it. The lock file handle is held in the `lock_file: Option<std::fs::File>` field on `SegmentBuffer` and released on `Drop` (explicit `unlock()` plus fd close as a belt-and-braces guarantee).
- **Cross-platform:** `fs4` uses `flock(2)` on Linux/macOS, `LockFileEx` on Windows. Pure-Rust via `rustix`, no `libc` dep.
- **Lock failure mode:** `Err(SegmentError::Locked)` — no block, no timeout. Callers retry on their own schedule if they want.
- **Loom tests bypass the lock** via `open_with_store` (loom does not model the filesystem, and a real lock file inside `loom::model` would deadlock). The `open_with_store` constructor passes `lock_file: None`.
- **Subprocess spawned by the owner** that inherits the lock fd via fd-passing is the user's concern, not the library's.
- If you refactor `open()` or `open_with_report()`, the lock acquisition MUST happen after `create_dir_all` and BEFORE any filename parsing or state publication.
## At-least-once delivery model
The library provides the substrate for at-least-once delivery; **idempotency lives in the caller's server**.
- `append()` returns the item's stable sequence number (`u64`, monotonic, gap-free across flushes).
- `delete_acked(seq)` is the commit point — it removes every segment whose `end <= seq` and advances `head_seq`.
- Between `read_from(start, ...)` and `delete_acked(start + count - 1)`, a crash leaves the batch on disk. On restart, `read_from(start, ...)` returns it again.
- `read_from` returns `Vec<T>` — items, not `(seq, T)` pairs. The caller tracks `start` and increments it by `batch.len()`. The starting cursor is `buf.stats().head_sequence` after recovery.
- The library does NOT own a cursor file. Cursor persistence is the caller's concern (see [Layer split vs monitor365](#layer-split-vs-monitor365)).
- Only the unflushed in-memory tail is at risk of loss. `flush()` drains it to disk.
## Layer split vs monitor365
segment-buffer is the **producer-side local buffer**. Everything cloud-facing lives upstream. This split was verified against monitor365's source on 2026-07-20 after the user flagged potential scope creep. Respect it: do not pull upstream concerns into this crate.
| Concern | Owner | Location |
| -------------------------------------------------- | --------------------- | --------------------------------- |
| Queue: `append`/`flush`/`read_from`/`delete_acked` | **segment-buffer** | `src/lib.rs` |
| Sequence numbers (stable, monotonic, gap-free) | **segment-buffer** | `SegmentBuffer::append` return |
| Segment file format, crash recovery | **segment-buffer** | `src/segment.rs` |
| `SyncCursor` (newtype around `u64`) | **monitor365** | `cloud-client/src/sync_cursor.rs` |
| Cursor persistence (SQLite + WAL) | **monitor365** | `cloud-client/src/sync_state.rs` |
| Cloud sync orchestration loop | **monitor365** | `cli/src/cloud_sync.rs` |
| Server-side idempotency (`event_id` dedup) | **monitor365 server** | (not in client) |
**Consequences for this crate:**
- **No cursor file.** The cursor is the consumer's concern. monitor365 stores it in SQLite with its own fsync discipline; pulling cursor persistence into segment-buffer would tangle two durability models, re-introduce the per-ack fsync cost `Throughput` removes, and mis-model the per-device vs per-directory cardinality. See TODO_LIST.md — the cursor-file item is REJECTED with rationale.
- **No backpressure policy.** The crate ships `store_pressure()` / `is_overloaded()` metrics only. The decision to block, sample, drop, or crash on disk-full is the upstream consumer's. segment-buffer just makes the metrics available. See `examples/backpressure.rs` for the canonical pattern.
- **No cloud client.** No HTTP, no retry policy, no auth. The drain loop is the consumer's `read_from → upload → delete_acked` cycle.
- **No server-side dedup.** Idempotency on `(producer_id, seq)` lives in the consumer's server. The library delivers at-least-once; the server makes it effectively-once.
If a proposed feature pulls any of the above into segment-buffer, reject it and document it as upstream's concern.
**Future cloud-sync extraction.** Cloud-sync may one day be extracted from monitor365 into its own crate. That extracted crate would sit _between_ segment-buffer and the cloud — it would consume segment-buffer, not be merged with it. segment-buffer stays the focused producer-side local buffer regardless. Do not use "cloud-sync will eventually be extracted" as a rationale for pulling sync logic, cursors, retry policy, or HTTP into this crate — that is scope creep in either direction.
## Durability model (shipped in v0.5.0)
**Today's default behavior** (`DurabilityPolicy::Segment`, what the crate has always done) fsyncs the segment file's data but NOT the directory inode after rename (`src/segment.rs` `write()` → `src/store.rs` `RealStore::write_atomic`). This means a host crash within the kernel's dir-inode flush window (~5–30s on ext4/xfs defaults) can leave the renamed file's data on disk but unreachable through the directory. SQLite went through this exact lesson. So today's behavior is **already not fully durable** — the framing isn't "weaken durability for speed", it's "make the tradeoff explicit and configurable."
The `DurabilityPolicy` enum shipped in v0.5.0:
| Policy | Fsync file | Fsync dir after rename | Worst-case crash loss |
| ------------ | ---------- | ---------------------- | ------------------------------------------------ |
| `Maximal` | yes | yes | last in-flight flush only |
| `Segment` | yes | no | rename window (~5–30s of flushes) — pre-v0.5.0 |
| `Throughput` | no | no | entire OS dirty window (~30s) — cloud is durable |
- `Throughput` is the correct default for cloud-sync deployments where the cloud endpoint holds the durable copy and the local disk is a throughput buffer.
- `Maximal` is for standalone-queue deployments where this buffer is the last copy.
- Backward compatibility: default stays `Segment` for one release after the enum lands, then flips to `Throughput` with a deprecation note.
- Implementation: `policy: DurabilityPolicy` is threaded through `SegmentStore::write_atomic`; the trait signature now takes it as a third parameter. `RealStore::write_atomic` branches on it. The loom `MockStore` accepts it for signature compatibility and ignores it (loom does not model fsync). The policy is a `Copy` enum, no allocation.
- The `Mutex<Compressor>` invariant ("never held across I/O") is preserved: the fsync happens after compression is done and the mutex is released.
## Commands
The `encryption` feature is **off by default**. Most verification commands must be run **twice** — once without features and once with `--features encryption` — because CI does exactly this (see `.github/workflows/ci.yml`).
```bash
# Tests (CONTRIBUTING.md canonical command — runs both default + encryption tests)
cargo test --no-fail-fast --features encryption
# Lint (warnings are hard errors, both in CONTRIBUTING and CI via RUSTFLAGS=-D warnings)
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --features encryption -- -D warnings
cargo fmt --all -- --check
# Examples (each is a separate binary)
cargo run --example basic_usage
cargo run --example backpressure
cargo run --example encrypted --features encryption # REQUIRES the feature flag
# Benchmarks (criterion, 10 separate targets declared in Cargo.toml)
cargo bench --bench bench_append
cargo bench --bench bench_read_from
cargo bench --bench bench_delete_acked
cargo bench --bench bench_recover
cargo bench --bench bench_stats
cargo bench --bench bench_read_vs_for_each
cargo bench --bench bench_append_all
cargo bench --bench bench_durability_policy
cargo bench --bench bench_segment_size_stats
cargo bench --bench bench_cipher --features encryption # REQUIRES the feature flag
# Docs (CI builds with the feature so AesGcmCipher is visible)
cargo doc --no-deps --features encryption
# Scaling test (NOT part of the gate; takes 15–45s at 100M scale)
cargo run --release --example scaling -- 10000000 # 10M
cargo run --release --example scaling -- 100000000 10000 1 # 100M, batch 10k, zstd-1
# Property tests (run as part of cargo test, but can be increased)
cargo test --no-fail-fast --features encryption -- property
# Fuzz (requires nightly; see fuzz/README.md)
cargo +nightly fuzz run fuzz_corrupted_read -- -max_total_time=60
cargo +nightly fuzz run fuzz_recovery -- -max_total_time=60
# Supply-chain publisher provenance (INFORMATIONAL — not part of the gate).
# `cargo audit` + `cargo deny` flag vulnerabilities and policy violations but
# neither shows WHO can publish the crates in the tree. This lists every
# crates.io account with publish rights over the dependency graph — run it
# when reviewing a Cargo.lock bump to spot unexpected new publishers or
# ownership transfers (the npm-style compromised-maintainer vector). The
# weekly `.github/workflows/supply-chain-report.yml` job runs the same thing.
cargo install cargo-supply-chain --locked
cargo supply-chain publishers
cargo supply-chain publishers --features encryption
```
### Nix (reproducible)
```bash
nix develop # devShell: rustc/cargo/clippy/rustfmt/rust-analyzer + zstd + pkg-config
# + nixfmt/dprint/shfmt/prettier (formatters invoked by buildflow)
nix fmt # treefmt: nixfmt + rustfmt (edition 2021, agrees with `cargo fmt`)
nix flake check # build, test, clippy, fmt, doc — all under the sandbox
nix build .#checks.x86_64-linux.test # run just the test check
```
## Feature flags
- `default = []`
- `encryption` — pulls in `aes-gcm` + `chacha20poly1305` + `rand`, exposing `AesGcmCipher` and `XChaCha20Poly1305Cipher`. The `SegmentCipher` **trait** is always available; only the two built-in cipher impls are gated.
**Known false-positive**: `rust-analyzer` will report `unresolved import segment_buffer::AesGcmCipher` (and `XChaCha20Poly1305Cipher`) in `examples/encrypted.rs` and a "configured out" hint at the cipher `pub use` in `src/lib.rs`. This is **not a bug** — rust-analyzer doesn't enable the feature. Real builds with `--features encryption` compile cleanly.
## Architecture & data flow
```
append(item) ─► unflushed: Vec<T> (in-memory, inside Mutex)
│
▼ (batch full OR flush_interval elapsed OR explicit flush())
take() the batch, compute start_seq/end_seq INSIDE the lock
│
▼ (lock released)
segment::encode_segment ─► CBOR → zstd → [optional cipher.encrypt] → prepend 8-byte SBF1 envelope (pure, src/segment.rs)
│
▼
store.write_atomic(range, bytes) ─► tmp → sync_all → rename to seg_*.zst (src/store.rs)
│
▼ (lock re-acquired)
approx_disk_bytes += len; segment_count += 1
```
`read_from(start, limit)` scans on-disk segments first (sorted by `start`), then drains the in-memory pending tail. `delete_acked(seq)` removes every segment whose `end <= seq` and advances `head_seq`. Read path calls `store.read_bytes` then `segment::decode_segment`, which strips the envelope (auto-detecting legacy v1 files) before decryption.
### Three-layer separation (since 2026-07-20)
The crate has a deliberate three-layer split:
| Layer | Module | Knows about |
| ----------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Format** | `src/segment.rs` | Bytes only — envelope, filename, CBOR/zstd/cipher pipeline. Pure functions, zero `std::fs`. |
| **I/O** | `src/store.rs` | The `SegmentStore` trait + `RealStore` impl. Owns the on-disk representation: `create_dir_all`, `scan`, `clean_tmp`, `segment_size`, `remove_segment`, `write_atomic`, `read_bytes`. |
| **Orchestration** | `src/lib.rs` | `SegmentBuffer`: mutex, flush policy, sequence-number invariants. Holds `Arc<dyn SegmentStore + Send + Sync>` and delegates every filesystem call through it. |
`open()` / `open_with_report()` construct `RealStore` internally; the trait is reachable externally only under the `loom` Cargo feature (via `SegmentBuffer::open_with_store`). The trait is **sealed** (`SegmentStore: private::Sealed + Send + Sync`): downstream crates cannot implement it. The seal marker (`SegmentStoreSealed`) is re-exported under the `loom` feature so in-tree mock stores can opt in. The trait-object approach (~5 ns vtable cost per I/O call, negligible next to zstd+CBOR+file I/O) was chosen over a type parameter `SegmentBuffer<T, S: SegmentStore>` because the latter would force every example, bench, fuzz target, and doc test to spell `<T, RealStore>` (~20 callsites of churn for a testing-only improvement).
### Crash recovery (the defining design choice)
**There is no WAL and no metadata database.** State is fully encoded in filenames:
- Pattern: `seg_{start:012}_{end:012}.zst` (12-digit zero-padded, inclusive range)
- On `open()`: `.tmp` files are deleted (incomplete crash writes), then filenames are parsed to rebuild `head_seq` (min start) and `next_seq` (max end + 1).
- Atomic durability comes from the tmp → `sync_all` → `rename` sequence in `write_segment`.
If you change the filename format, `segment::filename` and `segment::parse_filename` (in `src/segment.rs`) are the two sides of the contract — both must stay in sync, and existing on-disk files from monitor365 must still parse.
## Critical concurrency invariant
**`start_seq`, `end_seq`, and the sequence number returned by `append()` must all be computed inside the same mutex lock that takes ownership of `unflushed` / pushes the event.** Do not refactor these into separate lock acquisitions.
This was a real race fixed post-extraction (see CHANGELOG `[0.1.0]` → "Fixed"). The previous code re-read `next_seq` in a second lock and produced corrupted segment filenames under concurrent `append()`.
The `parking_lot::Mutex` is **never held across file I/O** — `flush()` drops it before `write_segment` and re-acquires it only to bump `approx_disk_bytes`; `recover()` collects all segment metadata (file sizes via `store.segment_size`, head/next seq from filenames) before taking the lock once to publish the rebuilt state. There are no await points; all I/O is synchronous. The `SegmentStore` trait object is invoked outside the mutex exactly as `std::fs` was before the refactor. **Nor is the mutex held across a `for_each_from` user callback** — since 2026-08-04 the in-memory pending window is snapshotted under the lock and the lock is released before the callback runs, eliminating the only panic path in the library (the old `assert_not_reentered` re-entrancy guard, the `iteration_in_progress` flag, and the `IterationGuard` RAII type are all deleted). The public API is now panic-free with no qualifications; re-entrant calls from inside a `for_each_from` callback (`append`, `stats`, `delete_acked`, …) are safe. Tradeoff: the in-memory tail is cloned once (bounded by `limit`), so `for_each_from` is no longer ~21× faster than `read_from` on pure in-memory reads.
### `delete_acked` + `append` interleaving (loom-proven since 2026-07-20)
The clamp at the end of `delete_acked`:
```rust
let pending_start = inner.next_seq.saturating_sub(inner.unflushed.len() as u64);
inner.head_seq = new_head.unwrap_or(inner.next_seq).min(pending_start);
```
is exhaustively proven correct across every schedule of two threads by the loom tests `delete_acked_during_append_never_loses_head`, `delete_acked_past_flush_boundary_with_concurrent_append`, `stats_snapshot_consistent_under_delete_plus_append`, and `delete_acked_idempotent_under_concurrent_append` in `tests/loom.rs`. The proof depends on a `MockStore` (loom-aware in-memory stub) being injected via `open_with_store`; the production `RealStore` shares the same trait, so the proof transfers. The stress test `concurrency_4_writers_1_reader_10k_events` covers the same interleaving _statistically_; loom covers it _exhaustively_.
### `read_from` race windows (documented since 2026-07-23, not fixed by design)
`read_from` has two unlocked gaps that concurrent operations can exploit. **Neither corrupts data.** Both are documented in `docs/DOMAIN_LANGUAGE.md` → "Consistency model" and proven safe by two stress tests in `src/tests.rs`.
1. **Concurrent `delete_acked` race.** Phase 1 scans the directory unlocked, then reads each segment file unlocked. If `delete_acked` removes a segment between scan and read, `store.read_bytes` returns a bare `fs::read` error (`SegmentError::Io(NotFound)`). This is _not_ data loss — the segment was already acked. Caller retries or skips forward.
2. **Concurrent `flush` race.** Phase 1 (scan) and Phase 2 (lock + read `unflushed`) are separated by an unlocked gap. If `flush()` completes during that gap, items leave `unflushed` for a segment file the scan already missed. Result: a transient incomplete read — the items are durable on disk, a retry sees them.
**Why these are not fixed:** making `read_bytes` swallow `NotFound` would mask genuine corruption (a missing segment that was _not_ acked). Holding the mutex across the scan-to-read gap would serialize I/O and break the "never held across file I/O" invariant. The correct response is documentation + retry guidance, not a code change that implies a stronger guarantee than the crate provides.
**Scan-cache TOCTOU (fixed):** the race _windows_ above are by-design, but the scan cache in `scan_segments` used to capture the directory `mtime` _after_ its `readdir`. A rename landing mid-scan paired a post-rename `mtime` with a pre-rename segment list, so the `mtime` guard could not detect the staleness and "a retry sees them" failed until the next directory mutation. The `mtime` is now captured _before_ the scan, so any mid-scan rename leaves the cached `mtime` stale and forces a re-scan next call. Effective only where `mtime` advances (`mtime_supported`); on coarse-granularity filesystems the explicit `invalidate_scan_cache` remains the sole defence. If you refactor `scan_segments`, keep the `mtime` capture strictly before `store.scan()`.
**Tests:** `concurrent_read_and_delete_never_corrupts` and `concurrent_read_and_flush_never_corrupts` in `src/tests.rs` stress both windows, asserting that reads never return wrong, out-of-order, or duplicate items. `src/property_tests.rs` adds machine-checkable property tests: three deterministic (`read_from_surviving_items_correct_after_delete`, `read_from_correct_with_disk_memory_split`, `read_from_all_visible_after_flush_from_split`), four single-mutation concurrent tests (`read_from_invariant_under_concurrent_delete_acked`, `read_from_invariant_under_concurrent_flush`, `for_each_from_invariant_under_concurrent_flush`, `for_each_from_invariant_under_concurrent_delete_acked`), two dual-mutation concurrent tests (`read_from_invariant_under_concurrent_delete_acked_and_flush`, `iter_from_invariant_under_concurrent_flush_and_delete`), and one `segment_count_matches_disk_across_flush_delete_ops` invariant. The scan-cache TOCTOU fix (mtime-before-scan ordering) is proven deterministically by `scan_cache_toctou_mtime_guard_forces_rescan_after_mid_scan_rename` in `src/tests.rs` (a `HookedStore` with two `std::sync::Barrier` sync points forcing the exact `scan → rename → scan-returns-stale` interleaving). Two loom tests in `tests/loom.rs` exhaustively enumerate the scan-cache populate/invalidate interleaving under concurrent `flush` and `delete_acked`. A high-concurrency stress test in `src/tests.rs` (`segment_count_stress_4_writers_2_deleters`) proves the live `segment_count` counter never panics and converges to the actual directory count after `sync_disk_bytes`.
## Backpressure / overload policy
The crate **ships no admission policy**. `store_pressure()` returns `approx_disk_bytes / max_size_bytes ∈ [0.0, 1.0]`; `is_overloaded()` is just `> 0.9`. Callers define their own priority thresholds — see `examples/backpressure.rs` for the canonical pattern. For tuning `FlushPolicy::Batch(N)` against the resulting segment file sizes, `segment_size_stats()` returns an on-demand `count` / `min` / `max` / `mean` / `p50` / `p90` byte-size distribution — an `O(n_segments)` scan outside the mutex (like `sync_disk_bytes`), nearest-rank percentiles, pure query (no cached-counter mutation). See `examples/segment_tuning.rs` for the canonical batch-size tuning loop.
## Flush offloading (pattern, not feature)
The crate **does not ship a background flush worker**. The default `FlushPolicy::Batch(N)` runs the encode pipeline inline on the threshold-crossing `append()`. For p99-sensitive producers, the decoupling is a caller-owned `FlushPolicy::Manual` + timer thread — see `examples/background_flush.rs` for the canonical pattern (atomic shutdown flag, final sync flush before exit). A library-internal worker was considered and rejected: it would add a per-buffer thread (breaking the "synchronous, no hidden threads" identity), make error propagation strictly worse (sticky errors on next call vs immediate), and duplicate what `FlushPolicy::Manual` + a user timer already achieves. See `docs/planning/2026-07-21_08-26_flush-worker-and-tier-0-levers.md` § "Addendum" for the full design rationale.
## Encryption on-disk format
`AesGcmCipher` writes `[12-byte random nonce][ciphertext + 16-byte GCM tag]` as the segment **payload**. This payload is **byte-compatible with monitor365's `EncryptionKey` segment format** — do not change it without a migration story. `segment::decode_segment` rejects encrypted payloads shorter than `NONCE_LEN` (12) as `SegmentError::Integrity` with the offending path.
`XChaCha20Poly1305Cipher` (shipped v0.5.0, behind the same `encryption` feature) writes `[24-byte random nonce][ciphertext + 16-byte Poly1305 tag]`. The 24-byte nonce eliminates the 2³²-message per-key limit of AES-GCM's 12-byte nonce, and ChaCha20 is constant-time in software (no AES-NI dependency). This is the cipher `SegmentConfigBuilder::recommended_cipher(key)` installs for new buffers; legacy AES-GCM segments still decrypt through `AesGcmCipher`. The two formats are byte-distinguishable only by which cipher the buffer was opened with (no envelope marker for the cipher type today — see the envelope v2 design doc for the migration path).
**Cipher evolution (2026-07-20 direction, partly shipped):**
- **AES-256-GCM** stays — legacy byte-compat with monitor365 is a hard constraint. Not deprecated.
- **XChaCha20-Poly1305** shipped in v0.5.0 as the recommended cipher for new buffers via `recommended_cipher()`. Same `SegmentCipher` trait, feature-gated impl alongside AES-GCM.
- **Streaming/incremental cipher** is a long-term direction. Today the whole segment is buffered (CBOR → zstd → encrypt as a blob); a streaming AEAD (e.g. RFC 8450 chunked format) would bound memory on large segments and enable early-stop-at-`limit` reads. Cost: format change. Likely v0.6+.
When adding a cipher: feature-gate it, expose it under `src/cipher.rs` alongside `AesGcmCipher` and `XChaCha20Poly1305Cipher`, add property tests (roundtrip, tamper, short-payload), and document the on-disk format here.
## Segment file envelope (format evolution)
Every segment written by this crate is wrapped in an 8-byte envelope:
```
offset bytes meaning
0..4 4 magic: ASCII "SBF1"
4 1 envelope version (currently 1)
5..8 3 reserved (zero; future: checksum type, compression algo)
8.. payload (zstd(CBOR), optionally encrypted — the v1 layout)
```
On read, the envelope is **auto-detected**: a file is treated as enveloped only when the magic matches **and** the 3 reserved bytes are all zero. Requiring the reserved bytes is what makes the false-positive rate on legacy encrypted files (whose first 7 bytes are random AEAD nonce) **2⁻⁵⁶ per file — negligible even across the full 597M-segment monitor365 corpus**. Files without both conditions are treated as legacy v1 (the original monitor365 format). This makes the envelope strictly additive — no migration needed. The cipher always sees the payload (post-envelope-strip), so cipher byte-compatibility is preserved.
If you need to evolve the format (new checksum, new compression, metadata block), bump `ENVELOPE_VERSION` in `src/segment.rs` and branch on the version in `unwrap_envelope`. The reserved-bytes-zero invariant must keep holding for any new v1-compatible version; repurpose them only when bumping to a version that is allowed to refuse legacy detection.
## Project layout
```
src/
lib.rs SegmentBuffer, SegmentConfig, BufferStats, BufferInner; orchestrates lock + flush policy + Arc<dyn SegmentStore>
segment.rs On-disk format (PURE, no I/O): envelope, SegmentRange, filename/parse, encode_segment, decode_segment, encode_payload, decode_payload, wrap/unwrap_envelope
store.rs SegmentStore trait + RealStore impl: the I/O boundary. create_dir_all / scan / clean_tmp / segment_size / remove_segment / write_atomic / read_bytes
cipher.rs SegmentCipher trait, CipherError (opaque: private fields + `Arc<dyn Error + Send + Sync>` source for chaining), AesGcmCipher + XChaCha20Poly1305Cipher (both feature-gated, impls in `mod private`)
error.rs SegmentError (typed: path + phase + reason), Result alias
tests.rs `mod tests` — unit tests (132 tests; `grep -c '#[test]' src/tests.rs`)
property_tests.rs proptest: filename/payload/envelope bijections, encrypted roundtrip, corrupted/recovery fuzz analogues, append_all / sync_disk_bytes / segment_size_stats / FlushPolicy / consistency-model race-window invariants, publish_disk_stats correctness, delete_acked idempotency (38 properties; `grep -c '#[test]' src/property_tests.rs`)
examples/ basic_usage, backpressure, background_flush, crash_recovery, mpmc, hotpath_profile, cloud_sync, cloud_sync_disk_full, idempotent_server, encrypted (feature-gated), bring_your_own_cipher (feature-gated), scaling (end-to-end 1M–100M lifecycle throughput), batch_or_interval_min (tiny-segment suppression demo), segment_tuning (segment_size_stats batch-size tuning loop)
benches/ 10 criterion targets (append, read_from, read_vs_for_each, delete_acked, recover, stats, append_all, durability_policy, segment_size_stats, cipher[encryption]) + shared support.rs
fuzz/ cargo-fuzz scaffold (fuzz_corrupted_read, fuzz_recovery, fuzz_parse_filename, fuzz_envelope, fuzz_append_all, fuzz_flush_policy, fuzz_for_each_from); requires nightly
FEATURES.md Honest capability inventory by status
TODO_LIST.md Short/mid-term improvement tasks with status
ROADMAP.md Long-term direction and explicit non-goals
docs/LIMITATIONS.md What the crate does NOT do and why (design tradeoffs, scope boundaries, accepted constraints)
flake.nix Reproducible devShell (zstd, pkg-config, Rust toolchain, formatters)
dprint.json dprint config — JSON-only (markdown/yaml skipped, see .buildflow.yml)
.buildflow.yml buildflow config — skips markdown-format and yaml-format (prettier handles markdown)
```
The split between `lib.rs` (in-memory orchestration + locking) and `segment.rs` (byte-level disk format) is deliberate: the buffer doesn't know how segments are encoded, and the segment module doesn't know about the mutex. `SegmentBuffer`'s private `write_segment`/`read_segment`/`scan_segments`/`segment_path` methods are thin instance-bound wrappers over the stateless `segment::` free functions.
## Code conventions
- `#![warn(missing_docs)]` is on — every public item needs a doc comment.
- Doc comment style uses `# Errors` and `# Example` sections (see `SegmentBuffer::open`).
- Tests use `tempfile::TempDir` and a `test_config(max_size_bytes)` helper that builds a `SegmentConfig` via `FlushPolicy::Batch(4)` (small batches for unit-test isolation). Concurrency/stress tests use `FlushPolicy::Manual` instead — see rule 7.
- The private in-memory field is `unflushed: Vec<T>` (items not yet written to a segment) — distinct from the public `pending_count()` backlog metric. Do not confuse the two.
- The top-level doc example is `#![no_run]`-gated.
- Lint posture is strict: `RUSTFLAGS=-D warnings` plus clippy `-D warnings`.
### Lint architecture (namtao-inspired, adopted 2026-08-02)
The crate uses a **two-tier lint strategy** inspired by [namtao's "Strict Lints" philosophy](https://www.namtao.com/rust/#strict-lints):
**Tier 1 — Cargo.toml `[lints.clippy]` (applies to ALL targets):**
`pedantic` (deny), `nursery` (deny), plus the restriction lints `unwrap_used`, `expect_used`, `indexing_slicing`, `arithmetic_side_effects`, `as_conversions`, `string_slice`, `panic_in_result_fn`, `panic`, `exit`, `todo`, `unimplemented`, `unreachable`. (The `unchecked_time_subtraction` / `unchecked_duration_subtraction` lint is covered by the `nursery` group — not listed explicitly because the lint name differs between MSRV 1.86 and stable.)
**Tier 2 — Library code gets real fixes; non-production targets get targeted allows:**
- **Library code** (`src/lib.rs`, `src/segment.rs`, `src/cipher.rs`, `src/store.rs`, `src/error.rs`): fully clippy-clean under `pedantic + nursery + all restriction lints`. All `as` conversions replaced with `try_from`/`unwrap_or`, all arithmetic replaced with `saturating_*`/`wrapping_*`, `#[must_use]` added to all builders, `Debug` uses `finish_non_exhaustive`, `#[track_caller]` removed from methods that no longer panic, explicit `drop(inner)` after the snapshot in `for_each_from` (lint-enforced minimal lock hold). There are zero `panic!`/`unwrap`/`expect`/indexing/string-slicing paths in library code.
- **In-crate test modules** (`src/tests.rs`, `src/property_tests.rs`): override with `#![allow(...)]` covering the panic-prevention lints, `as_conversions`, `arithmetic_side_effects`, and the full `pedantic` + `nursery` groups. Test code uses `unwrap()`, `as u64`, and `count += 1` extensively and safely.
- **Benches, examples, integration tests** (`benches/`, `examples/`, `tests/`): same `#![allow(...)]` block. These are infrastructure and demonstration code where strict lints add noise without safety value.
**`cargo-nextest`** is available in the Nix devShell (`nix develop`) for faster, more readable local test runs. CI continues to use `cargo test` — the suite runs in ~4 seconds, so nextest's parallelism advantage is negligible for CI.
## CI / MSRV
- Matrix: `ubuntu-latest` + `macos-latest` × `stable` + `1.86`.
- **MSRV is 1.86** (also the `rust-version` in `Cargo.toml`). There is a dedicated `msrv` job that runs `cargo check --all-targets --features encryption` on 1.86.0.
- **Local MSRV verification:** `nix develop .#msrv -c cargo check --all-targets --features encryption`. The `devShells.msrv` in `flake.nix` pins `rust-bin.stable."1.86.0"` via rust-overlay.
- **MSRV consistency guard:** `scripts/check-msrv.sh` asserts that `Cargo.toml rust-version`, `ci.yml` matrix + msrv job, `flake.nix` msrv shell pin, and `docs/MSRV.md` headline all agree. Run by the CI `msrv-consistency` job to prevent drift.
- macOS needs `brew install zstd` (CI does this automatically). Under the Nix devShell (`nix develop`), zstd is provided hermetically so no manual install is needed.
- **`Cargo.lock` is committed** (not gitignored) so Nix flake builds are reproducible. This intentionally overrides the global gitignore; use `git add -f Cargo.lock` if it gets dropped.
- **Loom concurrency testing** (`tests/loom.rs`): run with `RUSTFLAGS="--cfg loom" cargo test --features loom --test loom --release`. 14 tests covering the in-memory hot path (`append`/`append_all`/`stats` snapshot), the `delete_acked` + `append` interleaving (4 tests that exhaustively enumerate every two-thread schedule via a loom-aware `MockStore`), the scan-cache populate path under `read_from` (2 tests exercising `flush` and `delete_acked` racing the cache-populate interleaving), `segment_count` self-healing under concurrent flush + delete (proves the atomic counter never panics and is recalibrated by `sync_disk_bytes` after a momentary wrap), AND the `for_each_from` Phase 2 snapshot-then-release-lock pattern + `iter_from` materialising iterator (2 tests proving the v0.5.5 panic-free snapshot is never torn under every interleaving with a concurrent `append`). `flush` (the byte-level encode pipeline) and `recover` still touch byte-level I/O that loom has no interest in enumerating, so they stay covered _statistically_ by the stress test `concurrency_4_writers_1_reader_10k_events` in `src/tests.rs`. Use `--release` — loom's schedule enumeration is slow in debug.
## Releases
**All 12 versions (0.1.0 through 0.5.5) are published on BOTH crates.io and GitHub releases** (synced 2026-07-20, then kept current through v0.5.5). Before that sync, three crates.io versions (0.2.0/0.3.0/0.4.0) and one GitHub release (v0.1.0) were missing — the gap existed because `publish.yml` auto-publishes to crates.io on tag push but does NOT create GitHub releases, and the workflow had not been set up yet when the early tags were pushed.
Two surfaces, two responsibilities:
- **crates.io:** `.github/workflows/publish.yml` publishes automatically on `git push origin v*.*.*` (needs `CARGO_REGISTRY_TOKEN` secret). To backfill a missing version manually, `git worktree add --detach <dir> <tag>` then `cargo publish --features encryption` from that worktree (the tag's `Cargo.toml` version must match the tag). Verify with `cargo publish --dry-run --features encryption` first.
- **GitHub releases:** NOT automated by any workflow. They are created manually. `gh release create` fails on this repo demanding the `workflow` scope (a false-positive scope check); use `gh api --method POST repos/LarsArtmann/segment-buffer/releases -f tag_name=vX.Y.Z -f name=... -f body=...` instead (only `repo` scope needed). Do NOT pass `target_commitish` pointing at a tag name — it 404s; the tag is resolved from `tag_name` alone.
### Release runbook (step by step)
1. **Verify CI is green.** Run `gh run list --limit 4` and confirm every run on the target branch shows `success`. Local-only green is not sufficient (AGENTS.md rule 9).
2. **Run the full local gate.** `scripts/verify-gate.sh` — all 17 gates must pass.
3. **Bump `Cargo.toml` version.** Edit `version = "X.Y.Z"`. If a new release touches the `html_root_url`, update the version segment in `src/lib.rs` `#![doc(html_root_url = "...")]`.
4. **Update `CHANGELOG.md`.** Move `[Unreleased]` entries under a new `## [X.Y.Z] - YYYY-MM-DD` heading. Add a compare link at the bottom.
5. **Commit.** `git commit -am "release vX.Y.Z"`.
6. **Tag.** `git tag vX.Y.Z` (lightweight tag, no `-a` or `-m` needed — the GitHub release body carries the notes).
7. **Draft the GitHub release notes** BEFORE pushing the tag. A tag-without-release window breaks link checkers.
8. **Push.** `git push origin master --tags`. The `publish.yml` workflow will auto-publish to crates.io (now idempotent — safe if the version already exists).
9. **Create the GitHub release.** Use `gh api --method POST repos/LarsArtmann/segment-buffer/releases -f tag_name=vX.Y.Z -f name="vX.Y.Z" -f body="$(cat CHANGELOG-snippet.md)"` (NOT `gh release create` — see above).
10. **Verify.** Check `https://crates.io/crates/segment-buffer/X.Y.Z` and `https://docs.rs/segment-buffer/X.Y.Z` render within ~5 minutes.
11. **Soak.** Never ship two releases in the same day without a soak period between them.
## Documentation health cadence
Living docs (`README.md`, `AGENTS.md`, `FEATURES.md`, `TODO_LIST.md`, `ROADMAP.md`, `CHANGELOG.md`, `docs/DOMAIN_LANGUAGE.md`, `docs/LIMITATIONS.md`) drift against code every release. The `scripts/verify-gate.sh` gate now includes `lychee` (markdown link check), `scripts/check-html-root-url.sh` (catches the `html_root_url` rot vector), and `scripts/check-changelog-links.sh` (validates that every GitHub tag URL in CHANGELOG.md points to a real tag). Run `scripts/verify-gate.sh` **before every release tag** and re-run the docs-health skill after any release that adds/renames/removes a public item, a feature, or a dependency — the three classes of change that most reliably produce doc drift. Historical docs under `docs/{status,planning,perf}/` are point-in-time snapshots and are brought current by the `update-old-docs` skill (non-destructive annotation), never rewritten in place. Fully-resolved reports and completed plans are moved to `docs/status/archived/` and `docs/planning/archived/` respectively (via `git mv`, preserving history) — the active `docs/status/` and `docs/planning/` directories hold only in-flight work.
## Verification discipline (hard rules)
These rules were installed after three consecutive same-day sessions produced
self-reviews that claimed success without running the verification gate,
fabricated working-tree state, and invented baselines. They are non-negotiable
for any future agent (or human) working in this repo.
1. **Never describe working-tree state without a fresh `git status` in the same message.** "8 files staged", "working tree clean", "all committed" — all of these require a literal `git status` invocation in the current response. Re-running `git status` costs 100 ms; the cost of being wrong is a misleading commit message, a broken push, or a false release claim.
2. **Never invent baselines.** Health scores, perf numbers, "was X, now Y", "previously N tests" — if you cannot cite the source of the "previous" value, say "first audit" or "no prior baseline" instead. Numbers without provenance are lies with extra steps.
3. **Line-number citations are banned.** Cite section names, item text, or commit hashes. Line numbers shift the moment any file above the citation is edited; they rot in the same session that wrote them.
4. **Run the verification gate before declaring work done.** The canonical gate is `scripts/verify-gate.sh` (all 17 gates by default: `fmt`, `clippy(default)`, `clippy(encryption)`, `clippy(fuzz)`, `test(default)`, `test(encryption)`, `doc`, `html_root_url`, `cargo-lock`, `msrv-consistency`, `cargo-deny`, `cargo-audit`, `loom`, `lychee`, `changelog-links`, `actionlint`, and `nix flake check`). Use `--list` to print all gate names or `--only=X,Y,Z` to run a subset. The script uses `set -euo pipefail` and the `run()` helper captures the real exit status with `|| rc=$?` so a failing gate does not silently exit 0. Any claim that "tests pass" or "the build is green" must rest on a literal run of this gate (or the equivalent subset if a tool is unavailable) in the current session, with non-zero exit codes captured. The four basic commands (`cargo fmt`, `cargo clippy`, `cargo test`, `cargo doc`) are the minimal subset that can be run by hand when iterating quickly; the full gate is the source of truth for "done" claims.
5. **The supply-chain gate is BOTH `cargo audit` AND `cargo deny check`.** They pull from different advisory sources in edge cases. Running only one is not equivalent to running both. The CI `supply-chain` job runs both; the local pre-commit gate must too.
6. **The loom gate is `RUSTFLAGS="--cfg loom" cargo test --features loom --test loom --release`.** Files gated by `#![cfg(loom)]` are invisible to `cargo test` by default and silently rot without this explicit invocation. The CI `loom` job enforces it. Coverage (14 tests): the in-memory hot path (`append`/`append_all`/`pending_count`/`latest_sequence`/`stats`), the `delete_acked` + `append` interleaving (4 tests, exhaustively enumerating every schedule of two threads via the `MockStore` injected through `open_with_store`), the scan-cache populate path under `read_from` (2 tests exercising `flush`/`delete_acked` racing the cache-populate interleaving), `segment_count` self-healing under concurrent flush + delete, AND the `for_each_from` Phase 2 snapshot + `iter_from` materialising iterator (2 tests proving the v0.5.5 snapshot-then-release-lock pattern is never torn). `flush`/`recover` still touch byte-level encode/decode that loom has no interest in enumerating; their concurrency contracts stay covered _statistically_ by the stress test `concurrency_4_writers_1_reader_10k_events`.
7. **Concurrency tests must use `FlushPolicy::Manual`.** With `Batch(4)` the stress test creates 20 000 segment files (80 000 items / 4), causing pathological I/O under parallel test execution that hung CI for hours. `Manual` keeps items in-memory so the test stresses mutex contention, not the filesystem.
8. **Doctests that need `--features encryption` must be cfg-gated.** A `rust,no_run` code fence referencing `AesGcmCipher` fails to compile under `cargo test` (default features). Use the hidden `#[cfg(feature = "encryption")] fn main() {}` pattern — see the README encryption example.
9. **Before `git tag` for a release, the most recent CI + Nix runs on the target branch must be green.** Run `gh run list --limit 4` and confirm every run on the branch you are tagging shows `success`. Local-only verification (rule 4) is NOT sufficient: v0.4.1 and v0.4.2 both shipped with a "verification gate" that never checked GitHub Actions, leaving CI broken for 48+ hours while status reports claimed "all green". A release tag on an unverified commit is a lie of omission.
10. **CI-red is a stop-work condition.** If `gh run list --limit 4` shows red on the target branch, the first work item is "turn it green", not "add features on top." Local-only green is never a green claim; check `gh run list` before ANY "done" claim, not just before releases. The investigation sweep of 2026-07-20 documented this exact failure mode: a session claimed "all gates green" while CI was on its 5th consecutive red run due to MSRV drift the session had noticed and dismissed as "out of scope."
### Session-end checklist
Before writing any closing summary, status report, or "done" claim:
- [ ] `git status` — clean? Or have I explained every modified/untracked file?
- [ ] `git log --format='%h %ci %s' -10` — do the commits match what I think I did?
- [ ] Verification gate run with non-zero exit codes captured (see rule 4)?
- [ ] **`gh run list --limit 4` — is CI green on the target branch?** (Rule 10.) If red, the first work item is turning it green. Local-only green is never a "done" claim.
- [ ] Every doc claim that says "passes"/"verified"/"green" cites a commit hash or a literal command output in this session?
- [ ] No fabricated numbers — every "was X / now Y" has a citation or has been rewritten to "first audit" / "no baseline"?
- [ ] TODO_LIST updated for anything completed or partially completed this session?
- [ ] **Did I ship a release?** If yes: did the user explicitly approve the release scope? Never ship breaking changes without explicit approval. Never ship two releases in the same day without a soak period.
- [ ] **Before tagging a release: did `gh run list --limit 4` show the latest CI + Nix runs on the target branch as `success`?** (Rule 9.) A local-only green is not a release-ready green.
- [ ] **Did I draft the GitHub release notes BEFORE pushing the tag?** A tag-without-release window (even 2 minutes) breaks link checkers and confuses downstream consumers.
If any of these cannot be checked, the closing summary must say so explicitly. "Working tree clean" without `git status` in the same response is a process failure, not a shorthand.