Crate segment_buffer 

High-throughput local buffer for cloud sync — single-process by design, durability-configurable, optional performant encryption, at-least-once delivery.

Items are accumulated in memory, flushed as zstd-compressed CBOR batches to seg_{start:012}_{end:012}.zst files, and deleted once the consumer acknowledges receipt via SegmentBuffer::delete_acked.

The buffer is generic over any T: Serialize + DeserializeOwned + Clone + Send. (No explicit 'static bound is required: DeserializeOwned already implies it, since a borrowed type cannot satisfy for<'de> Deserialize<'de>.) Crash recovery is filename-based: scanning the directory rebuilds head_seq and next_seq without any WAL or metadata database.

Example

use segment_buffer::{SegmentBuffer, SegmentConfig};
use serde::{Serialize, Deserialize};

#[derive(Serialize, Deserialize, Clone)]
struct MyItem { id: u64 }

let buffer = SegmentBuffer::<MyItem>::open("/tmp/my-queue", SegmentConfig::default())?;
let seq = buffer.append(MyItem { id: 1 })?;
let items = buffer.read_from(0, 100)?;

For the full README — install, quickstart, encryption, backpressure, comparison table, and performance notes — see the project README on GitHub or docs.rs.

Examples

The examples/ directory in the source tree holds runnable end-to-end demos keyed by use case. Build and run any of them with cargo run --example <name> (encryption examples need --features encryption):

Example	What it shows
basic_usage	Minimum append/read/delete cycle.
cloud_sync	Full at-least-once drain loop with retry under transient failures.
cloud_sync_disk_full	Drain loop that pushes backpressure up to the producer when store_pressure() exceeds a threshold.
idempotent_server	Server-side (producer_id, seq) dedup pattern that makes at-least-once effectively-once.
crash_recovery	Flushed segments survive a simulated crash; unflushed don’t; open_with_report prints the recovery scan.
backpressure	The canonical pattern for translating store_pressure() into an admission decision.
background_flush	FlushPolicy::Manual + a caller-owned timer thread for p99-sensitive producers.
mpmc	Multi-producer / multi-consumer sharing via Arc<SegmentBuffer<T>>.
hotpath_profile	Latency-histogram harness for the append hot path.
scaling	End-to-end 1M–100M lifecycle throughput.
encrypted	AES-256-GCM and XChaCha20-Poly1305 ciphers end-to-end (requires --features encryption).
bring_your_own_cipher	Implementing the SegmentCipher trait for a custom cipher (requires --features encryption).

Structs

AesGcmCipherencryption

AES-256-GCM cipher with a random 12-byte nonce prepended to each ciphertext.

BufferStats

Point-in-time snapshot of buffer state, captured atomically under a single lock acquisition so all fields are mutually consistent.

CipherError

Error returned by SegmentCipher implementations.

RecoveryReport

Summary of the recovery scan performed by SegmentBuffer::open.

SegmentBuffer

High-throughput local buffer for cloud sync, holding items of T in memory and spilling them to compressed segment files for at-least-once delivery to a cloud endpoint.

SegmentConfig

Configuration knobs for SegmentBuffer.

SegmentConfigBuilder

Ergonomic builder for SegmentConfig.

SegmentIter

Owned-item iterator over buffer contents, yielding (seq, item) pairs.

XChaCha20Poly1305Cipherencryption

XChaCha20-Poly1305 cipher with a random 24-byte nonce prepended to each ciphertext.

Enums

DurabilityPolicy

Per-flush durability tradeoff between throughput and crash safety.

FlushPolicy

When to auto-flush pending items from memory to a segment file.

IoSite

Which filesystem site an SegmentError::Io failure happened on.

SegmentError

Errors produced by segment-buffer operations.

Traits

SegmentCipher

Encrypts and decrypts segment file payloads.

Type Aliases

Result

Result alias used throughout the crate.