# Compaction beside the writer
A container that has been written for months collects objects that are mostly
dead, because blocks were written again or files were deleted. Compaction copies
the few live blocks of such an object somewhere else, so the object can be
pruned. A second handle that publishes its own checkpoint for this is a second
writer: the writer's next publish finds the inode changed, and under `FailFast`
it stops.
Instead, a compactor can *attach* its compaction to the stored inode and leave
it to the writer to take in. It runs beside the writer, on another machine if
need be, and the only thing between the two is the inode's conditional write:
there is no protocol, no extra object and no extra request on the writer's
side. The writer has priority. The compactor does all its work in one go, meets
the writer only when it attaches, and on a conflict tries again without
disturbing the writer. This needs a format with parts (`PartedSegment` or
`StreamSegment`) and no WAL.
## An attached compaction
A compactor works on the version stored when it begins, B0 with number N0. It
copies blocks into new objects and builds C, the map of B0 with those blocks
moved. Everything it writes goes under N0, at parts no object has used yet
([p0, p1)), and is marked `SEGMENT_FLAG_CLEANER` (`0x2`) in its header:
```
0000000010.0 the writer's checkpoint 10: part 0 (index, inode)
0000000010.1 … .k the writer's checkpoint 10: data
0000000010.(k+1) … the compactor's copies and C's nodes, built on 10
0000000011.1 … the writer's checkpoint 11, sent ahead
inode 11, with the compaction of 10 attached
```
The writer never writes under a published number again, because its next
number is above it. So the compactor takes free parts of the head's number
without asking anyone. Every write is create-only: if two compactors pick the
same part, the second finds out and gives up.
Attaching is one conditional write of the inode object as it is stored, number
and all. That write sets `InodeRaw::FLAG_COMPACTION_PENDING` (`0x8`) in the
inode, and puts the compaction in the object's user metadata (`hf-compaction`):
N0, B0's fingerprint, [p0, p1), the number of blocks moved, and C's root. The
inode's content, size, attributes and counters do not change, and nor does
`content_cno`. There is only one compaction at a time: `begin_compaction`
refuses with `ResourceBusy` while one is attached, and an attach that finds
another one there gives up.
## The writer takes it in
Before its next publish of data, the writer merges C into its own map. It does
this whether the publish is the first after an open, or comes after a 412
because the compaction was attached while the writer had changes in flight.
The rule is by number. A pointer in the writer's map is *old* when it is on
storage at a number ≤ N0 and its node is not dirty in memory. Such a pointer is
what B0 had, and where C differs, C's pointer is taken: a whole subtree at once
where C's node differs, a leaf entry where a block moved. Anything else is what
the writer has written since, and is kept. C has B0's shape, so the walk reads
only the nodes on the paths where the writer changed something.
```
writer inode compactor
│ [ 10 ] │ begin: B0 = 10
│ writes, sends 11.1 … │ relocate: copies to 10.(k+1) …
│ flush 11 ───────────────►│ │ write C under 10, no inode
│ [ 11 ] │
│ │◄── If-Match(11) ────────────────│ attach: 11 + bit + metadata
│ [ 11 + C ]
│ next flush: 412 or a fresh open finds C
│ merge C (blocks it changed since 10 keep what it wrote)
│──── If-Match(11 + C) ─────►│
│ [ 12 ] no bit, no metadata
```
A publish under a new number never carries the bit or the metadata, so the
attachment ends there. A write that keeps the number keeps both: attributes
alone, `update_stat_fast`, or the counter written after a failed flush. Every
inode write is checked: a write with the bit and without the metadata, or the
other way round, is refused before it is sent.
The writer may have published any number of times since N0. The attach is
still one write and nothing is redone; the merge keeps every block the writer
changed. A writer's objects already on storage when the compaction is attached
(sent ahead, or partials) stay in use where its blocks point at them.
If the metadata does not read, or the bit is set and the metadata is missing,
the writer takes nothing in and drops the compaction at its next publish of
data. The data is right either way. The copies are left for prune.
## The API
```rust
c.begin_compaction().await?; // B0 = the stored version
c.relocate(&[(start, n)]).await?; // copies blocks, as offline
let out = c.publish_compaction(max_age).await?; // writes C, then attaches
// or c.abort_compaction().await?
```
`Hyper` has the same methods as `fs_begin_compaction`, `fs_publish_compaction`
and `fs_abort_compaction`. A plain flush is refused while a compaction is open.
`CompactionAttached` reports the number the compaction was attached to, N0, the
parts, the blocks moved, and how many conditional writes the attach took.
`publish_compaction` gives up with `TimedOut` once the pass is older than
`max_age`. Until they are attached, the copies are referenced by nothing, and
only prune's grace keeps them. So set `max_age` below that grace, less a
margin. A 412 while attaching means the writer published meanwhile. The
compactor reads the inode again and attaches to the new one, after a short
pause varied between 20 and 200 ms.
`begin_compaction` also refuses with `ResourceBusy` while a checkpoint is on
storage past the stored inode: a writer is between its part 0 and its inode,
or stopped there. Open the compactor's handle with `runtime.take_up_on_open`
off. Otherwise a writable open takes such a checkpoint up once it is a few
seconds old, and writes the inode for it. That is right after a writer crashed
there, and wrong while the writer's own inode write is only slow. With the
setting off, the open takes the checkpoint up in memory only, the compaction
waits with `ResourceBusy`, and the writer publishes. A writer that did crash
there takes its checkpoint up when it next opens.
## What readers see
A reader holds one version from its open. An attached compaction changes no
version anyone opens: readers read through the objects as they were until the
writer takes the compaction in and publishes. After that, a block reads the
same at its copy as at its original, since objects are never changed once
written. A reader opened before reads on from the objects it opened, until they
are pruned.
## What prune has to do
The compaction leaves the old objects in place. They become reclaimable only
once the writer has published past the compaction. Pruning them is the
cleaner's job, which keeps:
1. what the stored inode's map references;
2. what the attached compaction's map references (C's root is in the inode
object's metadata). If the bit is set and the metadata does not read, the
prune stops and deletes nothing;
3. what any version current in the last *T* seconds references;
4. what was written in the last *T* seconds;
5. nothing above the newest published number is taken, because the writer's
next checkpoint is being written there.
`hyperfile-cleaner`'s `prune_with` with `PruneOptions::grace` does this. *T* has
to be longer than any reader holds a version, and longer than any compaction
pass takes (`max_age`).
## Limits
- The writer must be hyperfile 0.12 or later. An older one meets an attach as a
second writer: under `FailFast` its flush stops, and under the default policy
it publishes over the compaction. Its data is right either way, but its
publish keeps the bit without the metadata. Prune then refuses until a
0.12 writer has published.
- A checkpoint number has 16384 parts. A compaction takes the free ones of N0.
When the writer used nearly all of them, `begin_compaction` returns
`ResourceBusy` until the writer publishes again.
- Not with the WAL, whose log the compactor does not read. A WAL writer that
finds a compaction attached refuses with `Unsupported`.
- A copy of the inode object that replaces its metadata drops the compaction.
The writer then publishes without it, and prune refuses until it has.