sleet 0.1.0

A fleet manager for SlateDB databases
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149

# Architecture

Sleet uses object storage as a coordination mechanism. It does not require a
central server or database. Nodes read the same tree of objects and compute
ownership of work locally. The tree contains a registry of databases, a
registry of live nodes, and a configuration file.

## Model

```mermaid
flowchart LR
    subgraph Root["Fleet root"]
        Policy["sleet.toml"]
        Registry["dbs/*.toml"]
        Heartbeats["nodes/*.json"]
    end

    subgraph Nodes["Sleet nodes"]
        N1["sleet run --node-id a"]
        N2["sleet run --node-id b"]
        N3["sleet run --node-id c"]
    end

    DB1["SlateDB database"]
    DB2["SlateDB database"]
    Mirror["Mirror target"]

    N1 --> Root
    N2 --> Root
    N3 --> Root
    N1 --> DB1
    N2 --> DB1
    N2 --> DB2
    N3 --> DB2
    N3 --> Mirror
```

Every node reads the same tree:

```text
<root>/
  sleet.toml
  dbs/<percent-encoded-db-url>.toml
  nodes/<node-id>.<service-letters>.json
```

## Placement

Each node computes ownership locally from:

- the registered database URLs in `dbs/`
- the resolved service config for each database
- the live node set from `nodes/`
- the services each live node offers

Sleet uses rendezvous hashing to keep node assignments stable. Rendezvous
hashing is a deterministic, stateless algorithm that ranks nodes for each
database. Each node computes the same ranking and picks the highest-ranked
live node for each service. The result is a mapping of databases to nodes and
services. Adding or removing a node moves only the assignments affected by that
node's rank.

For `gc` and `compactor-coordinator`, the top-ranked live node owns the
`(database, service)` pair. For `compaction-workers`, the top `count` nodes
run compaction workers for the `(database, service)` pair. For mirroring,
ownership is per `(database, mirror, destination)` triple.

## Heartbeats and liveness

A node writes one heartbeat object every `heartbeat_interval`:

```text
nodes/sleet-1.cgmw.json
```

The suffix letters define the services the node offers:

| Letter | Service                 |
| ------ | ----------------------- |
| `c`    | `compactor-coordinator` |
| `g`    | `gc`                    |
| `m`    | `mirror`                |
| `w`    | `compaction-workers`    |

The placement calculation uses the object name and `LastModified` object
store metadata to determine the services offered and node liveness. It does
not fetch the heartbeat body. `sleet status` reads the body for observability:
`node_id`, heartbeat format version, Sleet version, SlateDB version, and
per-service running/backoff task counts.

A node is live when its heartbeat is younger than `heartbeat_timeout`. A clean
shutdown deletes the heartbeat, so peers can take over without waiting for a
timeout.

## Services

Sleet runs four service types:

| Service                 | Work                                                                         |
| ----------------------- | ---------------------------------------------------------------------------- |
| `gc`                    | Runs SlateDB garbage collection for the database.                            |
| `compactor-coordinator` | Schedules compactions and commits completed results.                         |
| `compaction-workers`    | Claims and executes SlateDB compaction jobs.                                 |
| `mirror`                | Copies a database to configured mirror targets and commits target manifests. |

Sleet only decides which nodes service loops run. SlateDB is responsible for
protecting the database from concurrent access:

- `.manifest` CAS for commits
- compactor epoch fencing for coordinators
- `.compactions` CAS claims for workers

Duplicate service execution can waste work, but will never corrupt a database.

## Failure behavior

Sleet treats placement as an efficiency mechanism. During stale reads, clock
skew, node restarts, or network partitions, two nodes may briefly believe they
own the same assignment. That is acceptable because the underlying SlateDB
operations are fenced or idempotent.

A missing owner is also possible during convergence. In that case work is
delayed until the next config or heartbeat poll. The usual handoff bounds are:

| Change                               | Expected convergence                        |
| ------------------------------------ | ------------------------------------------- |
| Node dies without deleting heartbeat | about `heartbeat_timeout`                   |
| Node exits cleanly                   | next `heartbeat_interval`                   |
| Registry or config changes           | up to `config_poll` plus one heartbeat tick |

Nodes must be able to reach the stores for their offered services.

## Scaling

Coordination cost scales mostly with node count:

- each node PUTs one heartbeat per `heartbeat_interval`
- each node LISTs `nodes/` per `heartbeat_interval`
- each node LISTs `dbs/` every `config_poll`
- assignments are computed in memory

Database work scales with database count and configured poll intervals. At
very large registry sizes, `dbs/` LIST cardinality becomes the pressure
point; [RFC 0001](../rfcs/0001-design.md) tracks an inventory-backed registry
as future work.

## RFCs

See the [rfcs/](../rfcs/) directory for design documents.