spillbuf 0.1.0

Durable disk buffer: append + fdatasync, hub-ACK watermark, oldest-first drain
Documentation
  • Coverage
  • 3.33%
    1 out of 30 items documented0 out of 16 items with examples
  • Size
  • Source code size: 29.1 kB This is the summed size of all the files inside the crates.io package for this release.
  • Documentation size: 404.0 kB This is the summed size of all files generated by rustdoc for all configured targets
  • Ø build duration
  • this release: 2s Average build duration of successful builds.
  • all releases: 2s Average build duration of successful builds in releases after 2024-10-23.
  • Links
  • Homepage
  • chewbaucke/spillbuf
    2 0 0
  • crates.io
  • Dependencies
  • Versions
  • Owners
  • chewbaucke

spillbuf

A durable on-disk spill buffer for publishers that need to ACK faster than object storage can persist.

Object-store-backed logs (s2-lite, SlateDB, S3-class streams) flush on an interval. Callers often want a 200 in milliseconds. spillbuf is the missing layer: append + fdatasync locally, return a sequence number, drain oldest-first to the hub, and only retire a record when the hub ACKs it.

Std-only. No async runtime. No serde. ~600 lines.

publisher ──append+fdatasync──► spillbuf ──drain batch──► hub (s2 / s2-lite / …)
     ▲                              │                         │
     └── seq (local ACK)            └── ack_through(watermark)◄┘

The contract

  1. The hub's ACK, not the local fsync, retires a record. ack_through(watermark) is the only delete path. A crash between local append and hub ACK redelivers — at-least-once, never silent loss.
  2. Crash-mid-append truncates. A partial last frame is discarded on open(). Intact frames survive reboot and drain oldest-first.
  3. CRC on every frame. IEEE CRC-32 over the payload; a corrupt frame stops the scan and truncates from there.
  4. Cap is backpressure, not drop. When depth_bytes + incoming > cap_bytes, append returns SpillError::CapFull. The caller 429s. Nothing in the buffer is evicted to make room.
  5. Acked prefix is gone. After ack_through, those frames are not redelivered on reopen. Idle segment files are deleted.

This is not a database, not a WAL for a store, and not a multi-writer log. One process, one directory, one writer.

Install

[dependencies]
spillbuf = "0.1"

Until crates.io:

[dependencies]
spillbuf = { git = "https://github.com/chewbaucke/spillbuf" }

Usage

use spillbuf::{Options, SpillBuf};

let mut buf = SpillBuf::open("./spill", Options::default())?;

// Local ACK — durable on this disk, not yet on the hub.
let seq = buf.append(br#"{"event":"signup"}"#)?;

// Background drain. Only retire what the hub confirmed.
let batch = buf.drain_batch(100)?;
// … send batch to the hub …
if let Some(last) = batch.last() {
    buf.ack_through(last.seq)?;
}

println!("depth={} cap={}", buf.depth(), buf.cap_bytes());

Environment knobs (also settable on Options):

Variable Default Meaning
SPILLBUF_SEGMENT_BYTES 64 MiB Rotate the segment file at this size
SPILLBUF_CAP_BYTES 64 MiB Reject appends at this pending depth

On cap, return HTTP 429 + Retry-After (or equivalent) — do not drop.

Using with s2 / s2-lite

The crate is hub-agnostic. On the publish side of s2-sdk / s2-lite:

// 1. ACK the caller from spillbuf.append (fsync).
// 2. Drain a batch and stream.append(AppendInput::new(batch)).
// 3. On success, buf.ack_through(last.seq).
// 4. On Connect / 5xx, leave the frames pending and retry.
// 5. On a permanent 4xx (e.g. metered batch too large), split the batch
//    or dead-letter — never ack_through a failed persist.

See examples/mock_hub.rs for the same loop against a slow in-process hub (the 500 ms sleep stands in for an object-store flush).

Why not an existing crate?

Surveyed when this was built:

  • yaque — closest match (disk queue + ack guards), unmaintained, self-described "for non-critical applications".
  • walrus-rust — a real WAL; named fallback if the crate ever outgrows the dumb-file envelope.
  • Vector's disk buffers — the prior art for the pattern, not a consumable library.

spillbuf is the 200-line-of-semantics version of that pattern: frames, CRC, cap, hub watermark.

Provenance

Extracted from the AmneGraph cell spike, where it sits on the webhook ACK path in front of s2-lite on Tigris. Battle-tested there: crash-mid-append, cap-429 (buffer never dropped), kill-9 windows with persist-before-process, and unclean recovery with tens of MB resident in the buffer.

License

MIT