Crate segment_buffer 

High-throughput local buffer for cloud sync — single-process by design, durability-configurable, optional performant encryption, at-least-once delivery.

Items are accumulated in memory, flushed as zstd-compressed CBOR batches to seg_{start:012}_{end:012}.zst files, and deleted once the consumer acknowledges receipt via SegmentBuffer::delete_acked.

The buffer is generic over any T: Serialize + DeserializeOwned + Clone + Send. (No explicit 'static bound is required: DeserializeOwned already implies it, since a borrowed type cannot satisfy for<'de> Deserialize<'de>.) Crash recovery is filename-based: scanning the directory rebuilds head_seq and next_seq without any WAL or metadata database.

Delivery guarantees

The crate provides at-least-once delivery. append() returns a stable sequence number; delete_acked(seq) is the commit point. Crash before the ack and items are re-delivered on recovery. Making this effectively-once requires server-side idempotency on (producer_id, seq) — see examples/idempotent_server.rs.

Under the canonical single-consumer drain loop (read_from → upload → delete_acked, sequential), the buffer also provides read-your-writes, monotonic reads, and contiguous results. Under concurrent multi-reader operation, two narrow race windows open (spurious Io errors from concurrent delete_acked; transient gaps from concurrent flush) that do not corrupt data but change the result shape. See the Consistency model section of the Domain Language doc for the full guarantee table and practical guidance.

Guarantees

• Panic-free public API. No public method calls panic!, unwrap, expect, direct indexing, or string slicing — enforced in CI by pedantic + nursery + restriction Clippy lints at deny. for_each_from never holds the mutex across the user callback (pending items are snapshotted under the lock then released), so re-entrant calls are safe and cannot deadlock.
• Single-process per directory. Enforced by an exclusive flock at open; a second process gets SegmentError::Locked.
• Crash recovery by filename. No WAL, no metadata database — scanning the directory rebuilds all state from seg_{start}_{end}.zst filenames.

Schema evolution of T

The crate has two versioning layers: the SBF1 envelope (crate-managed, forward-evolvable) and the CBOR payload of T (caller-managed, unversioned). Changing T in a backward-incompatible way will break deserialization of old segment files. See the Schema evolution section for compatible-change patterns and migration strategies.

Limitations

Every limitation here is a deliberate design decision or accepted tradeoff, not an oversight. The full rationale lives in LIMITATIONS.md.

Process model:

• Single-process per directory — enforced by flock; multiple threads are fine, multiple processes get SegmentError::Locked. Use IPC if you need multi-process access.
• Synchronous only — no async methods, no hidden threads, no built-in background flush worker. Decouple flush timing with FlushPolicy::Manual and a caller-owned timer thread.

Delivery semantics:

• At-least-once, not exactly-once — server-side idempotency on (producer_id, seq) is required for effectively-once delivery.
• No cursor persistence — the crate does not own a cursor file; the caller must persist the read cursor independently.

Durability:

• Unflushed items are volatile — the in-memory tail is lost on crash. Call flush() at crash-sensitive boundaries.
• DurabilityPolicy trades durability for throughput — Throughput (default) skips fsync entirely; Segment fsyncs the file only; Maximal fsyncs file + directory. Only Maximal is fully crash-safe.

Concurrent reads (the canonical single-consumer drain loop never hits these):

• Spurious Io(NotFound) under concurrent delete_acked — the segment was already acknowledged; retry the read.
• Transient gaps under concurrent flush — items move to a new segment file the directory scan already missed; a subsequent read_from observes them.

Data model:

• No schema evolution for T — the CBOR payload is unversioned. See Schema evolution of T above.
• No streaming cipher — the whole segment is buffered during encode and decode; a streaming AEAD is tracked under envelope v2.

Scope boundaries:

• No cloud client, retry policy, or backpressure policy — the crate provides SegmentBuffer::store_pressure as a signal; the decision to block, sample, drop, or crash is the caller’s.

Example

use segment_buffer::{SegmentBuffer, SegmentConfig};
use serde::{Serialize, Deserialize};

#[derive(Serialize, Deserialize, Clone)]
struct MyItem { id: u64 }

let buffer = SegmentBuffer::<MyItem>::open("/tmp/my-queue", SegmentConfig::default())?;
let seq = buffer.append(MyItem { id: 1 })?;
let items = buffer.read_from(0, 100)?;

For the full README — install, quickstart, encryption, backpressure, comparison table, and performance notes — see the project README on GitHub or docs.rs.

Examples

The examples/ directory in the source tree holds runnable end-to-end demos keyed by use case. Build and run any of them with cargo run --example <name> (encryption examples need --features encryption):

Example	What it shows
basic_usage	Minimum append/read/delete cycle.
cloud_sync	Full at-least-once drain loop with retry under transient failures.
cloud_sync_disk_full	Drain loop that pushes backpressure up to the producer when store_pressure() exceeds a threshold.
idempotent_server	Server-side (producer_id, seq) dedup pattern that makes at-least-once effectively-once.
crash_recovery	Flushed segments survive a simulated crash; unflushed don’t; open_with_report prints the recovery scan.
backpressure	The canonical pattern for translating store_pressure() into an admission decision.
background_flush	FlushPolicy::Manual + a caller-owned timer thread for p99-sensitive producers.
mpmc	Multi-producer / multi-consumer sharing via Arc<SegmentBuffer<T>>.
hotpath_profile	Latency-histogram harness for the append hot path.
scaling	End-to-end 1M–100M lifecycle throughput.
encrypted	AES-256-GCM and XChaCha20-Poly1305 ciphers end-to-end (requires --features encryption).
bring_your_own_cipher	Implementing the SegmentCipher trait for a custom cipher (requires --features encryption).
batch_or_interval_min	Suppressing tiny segments with the adaptive BatchOrIntervalMin policy.
segment_tuning	Using segment_size_stats() to tune batch size against resulting file sizes.

Structs

AesGcmCipherencryption

AES-256-GCM cipher with a random 12-byte nonce prepended to each ciphertext.

BufferStats

Point-in-time snapshot of buffer state, captured atomically under a single lock acquisition so all fields are mutually consistent.

CipherError

Error returned by SegmentCipher implementations.

RecoveryReport

Summary of the recovery scan performed by SegmentBuffer::open.

SegmentBuffer

High-throughput local buffer for cloud sync, holding items of T in memory and spilling them to compressed segment files for at-least-once delivery to a cloud endpoint.

SegmentConfig

Configuration knobs for SegmentBuffer.

SegmentConfigBuilder

Ergonomic builder for SegmentConfig.

SegmentIter

Owned-item iterator over buffer contents, yielding (seq, item) pairs.

SegmentSizeStats

Size distribution of the on-disk segment files at a point in time.

XChaCha20Poly1305Cipherencryption

XChaCha20-Poly1305 cipher with a random 24-byte nonce prepended to each ciphertext.

Enums

DurabilityPolicy

Per-flush durability tradeoff between throughput and crash safety.

FlushPolicy

When to auto-flush pending items from memory to a segment file.

IoSite

Which filesystem site an SegmentError::Io failure happened on.

SegmentError

Errors produced by segment-buffer operations.

Traits

SegmentCipher

Encrypts and decrypts segment file payloads.

Type Aliases

Result

Result alias used throughout the crate.