Expand description
§spacedb-durability — SpaceDB Layer 2 (cold path)
Durability that survives any home dying: a dataset is sealed into a content-addressed, Reed-Solomon erasure-coded snapshot whose shards spread across diverse homes, so losing homes is recoverable rather than fatal.
M4-S1 (here) is the foundation — the erasure math and the verifiable
manifest, over opaque (already-encrypted) snapshot bytes:
encode_snapshot → (Manifest, shards), and reconstruct_snapshot from
any k-of-n survivors with full integrity checking. Higher slices add the
placement and shard-store seams (M4-S2) and the repair loop (M4-S3); MATA
implements those over maestro-disco.
Open-core (MIT): no MATA dependency. Erasure operates on ciphertext, so a hosting home stores shards it cannot read.
Structs§
- Fleet
- A set of homes. Supports killing/reviving nodes to simulate failures.
- Manifest
- The durable description of an erasure-coded snapshot — everything needed to verify and reconstruct it from a sufficient subset of shards.
- MemShard
Store - In-memory content-addressed shard store. For tests and single-machine use; loses everything on drop.
- Node
- One home in the fleet: an id, a failure domain, a shard store, and whether it is currently reachable.
- Placement
- A placement record: which target holds each shard, indexed by shard index.
- Reclaim
Report - The outcome of a reclamation pass.
- Reclaimed
Copy - One surplus copy that was dropped.
- Repair
Report - The outcome of a repair pass.
- Replica
Health - A snapshot’s replica-count health.
- Shard
- One erasure-coded shard: its position in the code and its bytes (a ciphertext fragment). Stored on a host, which cannot read it.
- Shard
Ref - A manifest’s reference to one shard: its index and content hash. This is how a provided shard is verified before use.
- Target
Id - Identifier of a target node (a home).
- Target
Info - A candidate target for placement: its id and failure domain.
Enums§
- Durability
Error - Health
Status - The durability status of a snapshot.
Traits§
- Shard
Store - One host’s content-addressed shard storage.
Functions§
- allocate
- Assign
num_shardsshards acrosstargets, one shard per target, spread across distinct failure domains. Requires at leastnum_shardstargets. - distribute
- Write each shard to the home it was placed on. Errors if a placed target is unknown or offline (placement should only assign online targets).
- encode_
snapshot - Erasure-code
snapshotintodata_shardsdata +parity_shardsparity shards, returning theManifestand thenshards. Anydata_shardsof them reconstruct the original. - health
- Assess a snapshot’s replica-count health against the current fleet.
- reachable_
shard_ count - How many of a snapshot’s shards are currently reachable — placed on an online home that actually holds the bytes.
- reclaim
- Drop surplus copies of a snapshot’s shards — copies held by online homes that
the
placementno longer references — but only for shards whose live placed copy is currently reachable. A no-op (empty report) when there is no safe surplus. - reconstruct_
snapshot - Reconstruct the original snapshot from
availableshards, usingmanifest. Every provided shard is verified against the manifest (a tampered shard is rejected before use), and the rebuilt snapshot is hash-verified. Errors withDurabilityError::InsufficientShardsif fewer thankdistinct valid shards are present. - recover
- Reconstruct the snapshot from the shards currently held by online homes.
Tolerates up to
paritymissing homes; errors withDurabilityError::InsufficientShardsif fewer thankshards survive. - repair
- Restore a snapshot’s replica count by regenerating its missing shards onto
fresh homes. A no-op (empty report) if nothing is missing. Errors with
DurabilityError::InsufficientShardsif fewer thankshards survive (the snapshot can’t be reconstructed to regenerate the rest), orDurabilityError::NotEnoughTargetsif there aren’t enough fresh homes. - surplus_
shard_ count - How many surplus copies of a snapshot’s shards exist — bytes held by an
online home that the placement does not reference (e.g. a repaired-away shard
whose original home rejoined). These are reclaimable cost, orthogonal to the
under-replication
HealthStatus: a fullyHealthysnapshot can still carry surplus copies worth dropping (crate::reclaim).