# Durability and restart
QuePaxa assumes crash-*recovery*: a replica may stop at any instant and come back with
whatever it had made durable. The crate defines exactly what that is — opaque records you
store and hand back — and is safe as long as your storage keeps four promises.
## What your storage must promise
1. **Order.** Records of one [`Persist`](crate::Action::Persist) are durable after those of
every earlier persist. An append-only file written in action order, `fsync`ed before
reporting [`Persisted`](crate::Input::Persisted), keeps this.
2. **Durability before acknowledgement.** Report `Persisted(id)` only when the records of
`id` (and everything before) survive a power loss.
3. **Atomic replacement.** A persist with `replace: true` replaces *all* earlier records
at once: after a crash you see either the old set or the new one. Write a new file,
`fsync` it, rename it over the old one, `fsync` the directory.
4. **Torn tails are not records.** A crash in the middle of an append may leave a partial
last record. Drop it (records are self-checking: a CRC-32C covers each), or pass it on
and let [`Replica::new`](crate::Replica::new) report
[`CorruptRecord`](crate::Error::CorruptRecord) — never hand back bytes that were not a
complete record. Truncate only what a crash can leave behind: a length field or a record
that fails its check *before* the tail is corruption, and cutting the log there would
silently drop acknowledged records. Checksum your own framing, and refuse to start on
corruption.
If storage fails, report [`Input::StorageFailed`](crate::Input::StorageFailed). The replica
**fences** itself: it stops sending and answering, reports every outstanding proposal as
[`Dropped`](crate::Action::Dropped) with [`DropReason::Fenced`](crate::DropReason::Fenced) (their
outcome is unknown — they may still commit, as after a crash), and returns
[`Error::Fenced`](crate::Error::Fenced) from then on. Restart the process to recover.
## Restarting
Pass every durable record, in write order, to [`Replica::new`](crate::Replica::new), with a
**fresh** random seed:
```
use bytes::Bytes;
use quepaxa::{Action, Config, Input, NodeId, Replica, Timestamp};
/// Runs a single-member replica until it is quiet; returns its payload commits.
fn settle(r: &mut Replica, disk: &mut Vec<Bytes>) -> Vec<Bytes> {
let mut committed = Vec::new();
let mut actions = Vec::new();
r.poll_actions(&mut actions);
while !actions.is_empty() {
for action in std::mem::take(&mut actions) {
match action {
Action::Persist { id, records, replace, .. } => {
if replace { disk.clear(); }
disk.extend(records);
r.handle(Timestamp(0), Input::Persisted(id)).unwrap();
}
Action::Commit { entries, .. } => committed.extend(entries.into_iter().map(|e| e.payload)),
_ => {}
}
}
r.poll_actions(&mut actions);
}
committed
}
let mut replica = Replica::new(config([1; 32]), [], Timestamp(0))?;
settle(&mut replica, &mut disk);
replica.propose(Timestamp(0), Bytes::from_static(b"a"))?;
settle(&mut replica, &mut disk);
drop(replica); // crash: memory is gone, `disk` survives
// A fresh seed on every start: priorities must stay unpredictable.
let mut replica = Replica::new(config([2; 32]), disk.clone(), Timestamp(10))?;
// Recovered slots are emitted again, in order, for the application to rebuild its state.
assert_eq!(settle(&mut replica, &mut disk), [Bytes::from_static(b"a")]);
assert_eq!(replica.commit_index(), 1);
# Ok::<(), quepaxa::Error>(())
```
On every start the replica persists a new **incarnation** number (its startup actions are
the recovered commits, then that persist) and sends nothing until it is durable. Proposals are identified by `(origin, incarnation, id)`, so ids from
before a crash are never confused with new ones, and a leader never reuses its reserved
top priority for a slot it may have used it in before.
### What the application sees after a restart
- [`Commit`](crate::Action::Commit) is emitted again for slots the replica recovers from storage,
and for slots it learns again from peers. Track the next slot you expect and skip older
ones (or rebuild from your own snapshot, see [snapshots](crate::guide::snapshots)).
- Proposals made before the crash are not reported: their outcome is unknown. A client
that retries must make the retry idempotent at the application level (for example with a
request id in the payload). The log itself never commits one proposal twice.
- In a quiet cluster a restarted replica learns that it is behind from the next message it
gets: the next write anywhere, or a read barrier or proposal of its own. It then catches
up in bulk, from a peer's snapshot if the log was compacted.
## Things that break safety
- Acknowledging a write before it is durable, or reordering writes.
- Sending a `then` message before its persist is durable.
- Restarting a replica identity with an **empty** data directory (or an old backup) while
the rest of the cluster remembers it: it would reuse its incarnation and leader
reservation. Restore its latest state, or add it back as a new replica
([membership](crate::guide::membership)).
- Reusing the seed across starts (liveness, not safety: priorities become predictable).
Records are versioned, but before 1.0 a version reads only its own formats: after a format
change (listed in the changelog) a replica refuses its old records and snapshots with an
error rather than misreading them. Upgrade such a replica as a new one: a new id with empty
storage, added to the cluster ([membership](crate::guide::membership)).