spillbuf 0.1.0

Durable disk buffer: append + fdatasync, hub-ACK watermark, oldest-first drain
Documentation
# spillbuf

A durable on-disk spill buffer for publishers that need to **ACK faster than object storage can persist**.

Object-store-backed logs (s2-lite, SlateDB, S3-class streams) flush on an interval. Callers often want a 200 in milliseconds. `spillbuf` is the missing layer: `append` + `fdatasync` locally, return a sequence number, drain oldest-first to the hub, and **only retire a record when the hub ACKs it**.

Std-only. No async runtime. No serde. ~600 lines.

```text
publisher ──append+fdatasync──► spillbuf ──drain batch──► hub (s2 / s2-lite / …)
     ▲                              │                         │
     └── seq (local ACK)            └── ack_through(watermark)◄┘
```

## The contract

1. **The hub's ACK, not the local fsync, retires a record.** `ack_through(watermark)` is the only delete path. A crash between local append and hub ACK redelivers — at-least-once, never silent loss.
2. **Crash-mid-append truncates.** A partial last frame is discarded on `open()`. Intact frames survive reboot and drain oldest-first.
3. **CRC on every frame.** IEEE CRC-32 over the payload; a corrupt frame stops the scan and truncates from there.
4. **Cap is backpressure, not drop.** When `depth_bytes + incoming > cap_bytes`, `append` returns `SpillError::CapFull`. The caller 429s. Nothing in the buffer is evicted to make room.
5. **Acked prefix is gone.** After `ack_through`, those frames are not redelivered on reopen. Idle segment files are deleted.

This is not a database, not a WAL for a store, and not a multi-writer log. One process, one directory, one writer.

## Install

```toml
[dependencies]
spillbuf = "0.1"
```

Until crates.io:

```toml
[dependencies]
spillbuf = { git = "https://github.com/chewbaucke/spillbuf" }
```

## Usage

```rust
use spillbuf::{Options, SpillBuf};

let mut buf = SpillBuf::open("./spill", Options::default())?;

// Local ACK — durable on this disk, not yet on the hub.
let seq = buf.append(br#"{"event":"signup"}"#)?;

// Background drain. Only retire what the hub confirmed.
let batch = buf.drain_batch(100)?;
// … send batch to the hub …
if let Some(last) = batch.last() {
    buf.ack_through(last.seq)?;
}

println!("depth={} cap={}", buf.depth(), buf.cap_bytes());
```

Environment knobs (also settable on `Options`):

| Variable | Default | Meaning |
|---|---|---|
| `SPILLBUF_SEGMENT_BYTES` | 64 MiB | Rotate the segment file at this size |
| `SPILLBUF_CAP_BYTES` | 64 MiB | Reject appends at this pending depth |

On cap, return HTTP `429` + `Retry-After` (or equivalent) — do not drop.

## Using with s2 / s2-lite

The crate is hub-agnostic. On the publish side of [s2-sdk](https://crates.io/crates/s2-sdk) / [s2-lite](https://s2.dev/docs/s2-lite):

```rust
// 1. ACK the caller from spillbuf.append (fsync).
// 2. Drain a batch and stream.append(AppendInput::new(batch)).
// 3. On success, buf.ack_through(last.seq).
// 4. On Connect / 5xx, leave the frames pending and retry.
// 5. On a permanent 4xx (e.g. metered batch too large), split the batch
//    or dead-letter — never ack_through a failed persist.
```

See `examples/mock_hub.rs` for the same loop against a slow in-process hub (the 500 ms sleep stands in for an object-store flush).

## Why not an existing crate?

Surveyed when this was built:

- **yaque** — closest match (disk queue + ack guards), unmaintained, self-described "for non-critical applications".
- **walrus-rust** — a real WAL; named fallback if the crate ever outgrows the dumb-file envelope.
- Vector's disk buffers — the prior art for the pattern, not a consumable library.

`spillbuf` is the 200-line-of-semantics version of that pattern: frames, CRC, cap, hub watermark.

## Provenance

Extracted from the [AmneGraph](https://amnegraph.ai) cell spike, where it sits on the webhook ACK path in front of s2-lite on Tigris. Battle-tested there: crash-mid-append, cap-429 (buffer never dropped), kill-9 windows with persist-before-process, and unclean recovery with tens of MB resident in the buffer.

## License

MIT