Module compact 

Offline compaction: rewriting the live blocks of mostly-dead objects so the objects can be reclaimed, and so a read of one file range costs fewer requests.

What prune cannot do

Cleaner::prune reclaims an object nothing points into. An object that is 5% live is not that object: those few blocks hold the file’s only copy of their contents, so the other 95% cannot be given back while they stay there. Rewriting them into a new checkpoint is what turns the old object into one prune can take.

The same rewrite is what makes a read cheaper. A request breaks where the object changes, so blocks of one file range scattered across many objects cost one request each; written out together they cost one. Both goals use the same mechanism and differ only in what they select.

How the rewrite happens

Not by a format this crate writes. Nothing here builds a segment, assigns a pointer or publishes an inode — that is hyperfile’s write path, and a second implementation of it would drift from the first. What this does instead is ask hyperfile to rewrite blocks it already knows how to rewrite:

• Hyper::fs_relocate marks a run of blocks to be rewritten at the next flush with their current bytes. It reads them with the read path’s own coalescing, so a contiguous run costs a few large requests rather than one per block; it skips holes, explicit zero blocks and anything already dirty, so a sparse file stays sparse; and it touches neither mtime, ctime nor the read cache.
• One fs_flush at the end of the pass writes the dirty set as one checkpoint. The dirty set is keyed by block index, so the blocks land in file order and packed — which is both halves of what compaction is for.

A compaction is not a change of content

The pass has to publish a new checkpoint — blocks moved, so pointers, the map and the inode changed — and so last_cno advances. A consumer that reads a new last_cno as “the content changed” would take every compaction for a change. hyperfile therefore marks a publish whose every dirty block came from relocation with InodeRaw::FLAG_COMPACTED, and Hyper::fs_content_cno stays where it was across it. That is the number a consumer should compare.

The mark is only set when the publish’s number directly follows the previous one, so it holds for exactly one checkpoint. Two passes back to back would therefore move content_cno on the second, and S3Cleaner::compact refuses to start on a checkpoint that is already marked unless CompactOptions::allow_consecutive says otherwise.

Why offline

A compactor is a writer, and it writes old bytes — the content it read earlier. hyperfile’s default conflict policy, RetryLastWriterWins, refreshes from storage on a conflict and re-publishes the dirty set, which for a compactor means republishing a stale copy of a block another writer has since changed. So compaction runs with FlushConflictPolicy::FailFast and the caller must be the container’s only writer. A concurrent publish is then detected — the inode carries an If-Match on its ETag — and the pass fails rather than losing the other writer’s data.

A log is the other half of that contract. This refuses to run when the container’s prefix holds keys that are neither the inode nor segment objects, because a WAL is the usual reason for them and a compactor that does not read the log would publish over entries recovery has not replayed yet.

Structs

CompactOptions

Which objects compaction may rewrite, and how much it may do at once.

CompactOutcome

What a compaction pass did.

ObjectUsage

How much of one stored object is still pointed into.

Report

What a container’s storage is made of, and what could be given back.

Constants

UNBOUNDED_PASS_BYTES

The most a pass holds in memory when nothing can write it out early: a parted container with partial segments turned off, or a container that is not parted. There fs_relocate keeps every block dirty until the pass’s one flush.

Functions

compaction_runtime_default

The runtime configuration a compaction uses when the caller gives none.