spillbuf
A durable on-disk spill buffer for publishers that need to ACK faster than object storage can persist.
Object-store-backed logs (s2-lite, SlateDB, S3-class streams) flush on an interval. Callers often want a 200 in milliseconds. spillbuf is the missing layer: append + fdatasync locally, return a sequence number, drain oldest-first to the hub, and only retire a record when the hub ACKs it.
Std-only. No async runtime. No serde. ~600 lines.
publisher ──append+fdatasync──► spillbuf ──drain batch──► hub (s2 / s2-lite / …)
▲ │ │
└── seq (local ACK) └── ack_through(watermark)◄┘
The contract
- The hub's ACK, not the local fsync, retires a record.
ack_through(watermark)is the only delete path. A crash between local append and hub ACK redelivers — at-least-once, never silent loss. - Crash-mid-append truncates. A partial last frame is discarded on
open(). Intact frames survive reboot and drain oldest-first. - CRC on every frame. IEEE CRC-32 over the payload; a corrupt frame stops the scan and truncates from there.
- Cap is backpressure, not drop. When
depth_bytes + incoming > cap_bytes,appendreturnsSpillError::CapFull. The caller 429s. Nothing in the buffer is evicted to make room. - Acked prefix is gone. After
ack_through, those frames are not redelivered on reopen. Idle segment files are deleted.
This is not a database, not a WAL for a store, and not a multi-writer log. One process, one directory, one writer.
Install
[]
= "0.1"
Until crates.io:
[]
= { = "https://github.com/chewbaucke/spillbuf" }
Usage
use ;
let mut buf = open?;
// Local ACK — durable on this disk, not yet on the hub.
let seq = buf.append?;
// Background drain. Only retire what the hub confirmed.
let batch = buf.drain_batch?;
// … send batch to the hub …
if let Some = batch.last
println!;
Environment knobs (also settable on Options):
| Variable | Default | Meaning |
|---|---|---|
SPILLBUF_SEGMENT_BYTES |
64 MiB | Rotate the segment file at this size |
SPILLBUF_CAP_BYTES |
64 MiB | Reject appends at this pending depth |
On cap, return HTTP 429 + Retry-After (or equivalent) — do not drop.
Using with s2 / s2-lite
The crate is hub-agnostic. On the publish side of s2-sdk / s2-lite:
// 1. ACK the caller from spillbuf.append (fsync).
// 2. Drain a batch and stream.append(AppendInput::new(batch)).
// 3. On success, buf.ack_through(last.seq).
// 4. On Connect / 5xx, leave the frames pending and retry.
// 5. On a permanent 4xx (e.g. metered batch too large), split the batch
// or dead-letter — never ack_through a failed persist.
See examples/mock_hub.rs for the same loop against a slow in-process hub (the 500 ms sleep stands in for an object-store flush).
Why not an existing crate?
Surveyed when this was built:
- yaque — closest match (disk queue + ack guards), unmaintained, self-described "for non-critical applications".
- walrus-rust — a real WAL; named fallback if the crate ever outgrows the dumb-file envelope.
- Vector's disk buffers — the prior art for the pattern, not a consumable library.
spillbuf is the 200-line-of-semantics version of that pattern: frames, CRC, cap, hub watermark.
Provenance
Extracted from the AmneGraph cell spike, where it sits on the webhook ACK path in front of s2-lite on Tigris. Battle-tested there: crash-mid-append, cap-429 (buffer never dropped), kill-9 windows with persist-before-process, and unclean recovery with tens of MB resident in the buffer.
License
MIT