Crate segment_buffer 

High-throughput local buffer for cloud sync — single-process by design, durability-configurable, optional performant encryption, at-least-once delivery.

Items are accumulated in memory, flushed as zstd-compressed CBOR batches to seg_{start:012}_{end:012}.zst files, and deleted once the consumer acknowledges receipt via SegmentBuffer::delete_acked.

The buffer is generic over any T: Serialize + DeserializeOwned + Clone + Send. (No explicit 'static bound is required: DeserializeOwned already implies it, since a borrowed type cannot satisfy for<'de> Deserialize<'de>.) Crash recovery is filename-based: scanning the directory rebuilds head_seq and next_seq without any WAL or metadata database.

Delivery guarantees

The crate provides at-least-once delivery. append() returns a stable sequence number; delete_acked(seq) is the commit point. Crash before the ack and items are re-delivered on recovery. Making this effectively-once requires server-side idempotency on (producer_id, seq) — see examples/idempotent_server.rs.

Under the canonical single-consumer drain loop (read_from → upload → delete_acked, sequential), the buffer also provides read-your-writes, monotonic reads, and contiguous results. Under concurrent multi-reader operation, two narrow race windows open (spurious Io errors from concurrent delete_acked; transient gaps from concurrent flush) that do not corrupt data but change the result shape. See the Consistency model section of the Domain Language doc for the full guarantee table and practical guidance.

Schema evolution of T

The crate has two versioning layers: the SBF1 envelope (crate-managed, forward-evolvable) and the CBOR payload of T (caller-managed, unversioned). Changing T in a backward-incompatible way will break deserialization of old segment files. See the Schema evolution section for compatible-change patterns and migration strategies.

Example

use segment_buffer::{SegmentBuffer, SegmentConfig};
use serde::{Serialize, Deserialize};

#[derive(Serialize, Deserialize, Clone)]
struct MyItem { id: u64 }

let buffer = SegmentBuffer::<MyItem>::open("/tmp/my-queue", SegmentConfig::default())?;
let seq = buffer.append(MyItem { id: 1 })?;
let items = buffer.read_from(0, 100)?;

For the full README — install, quickstart, encryption, backpressure, comparison table, and performance notes — see the project README on GitHub or docs.rs.

Examples

The examples/ directory in the source tree holds runnable end-to-end demos keyed by use case. Build and run any of them with cargo run --example <name> (encryption examples need --features encryption):

Example	What it shows
basic_usage	Minimum append/read/delete cycle.
cloud_sync	Full at-least-once drain loop with retry under transient failures.
cloud_sync_disk_full	Drain loop that pushes backpressure up to the producer when store_pressure() exceeds a threshold.
idempotent_server	Server-side (producer_id, seq) dedup pattern that makes at-least-once effectively-once.
crash_recovery	Flushed segments survive a simulated crash; unflushed don’t; open_with_report prints the recovery scan.
backpressure	The canonical pattern for translating store_pressure() into an admission decision.
background_flush	FlushPolicy::Manual + a caller-owned timer thread for p99-sensitive producers.
mpmc	Multi-producer / multi-consumer sharing via Arc<SegmentBuffer<T>>.
hotpath_profile	Latency-histogram harness for the append hot path.
scaling	End-to-end 1M–100M lifecycle throughput.
encrypted	AES-256-GCM and XChaCha20-Poly1305 ciphers end-to-end (requires --features encryption).
bring_your_own_cipher	Implementing the SegmentCipher trait for a custom cipher (requires --features encryption).

Structs

AesGcmCipherencryption

AES-256-GCM cipher with a random 12-byte nonce prepended to each ciphertext.

BufferStats

Point-in-time snapshot of buffer state, captured atomically under a single lock acquisition so all fields are mutually consistent.

CipherError

Error returned by SegmentCipher implementations.

RecoveryReport

Summary of the recovery scan performed by SegmentBuffer::open.

SegmentBuffer

High-throughput local buffer for cloud sync, holding items of T in memory and spilling them to compressed segment files for at-least-once delivery to a cloud endpoint.

SegmentConfig

Configuration knobs for SegmentBuffer.

SegmentConfigBuilder

Ergonomic builder for SegmentConfig.

SegmentIter

Owned-item iterator over buffer contents, yielding (seq, item) pairs.

SegmentSizeStats

Size distribution of the on-disk segment files at a point in time.

XChaCha20Poly1305Cipherencryption

XChaCha20-Poly1305 cipher with a random 24-byte nonce prepended to each ciphertext.

Enums

DurabilityPolicy

Per-flush durability tradeoff between throughput and crash safety.

FlushPolicy

When to auto-flush pending items from memory to a segment file.

IoSite

Which filesystem site an SegmentError::Io failure happened on.

SegmentError

Errors produced by segment-buffer operations.

Traits

SegmentCipher

Encrypts and decrypts segment file payloads.

Type Aliases

Result

Result alias used throughout the crate.