# Hyperfile Cleaner
Segments cleaner for [Hyperfile](https://crates.io/crates/hyperfile).
## What it does
Hyperfile is a copy-on-write log-structured file format on S3. Each
flush produces a new segment object. Over time, older segments lose
references as the metadata tree gets rewritten. This crate
enumerates and (optionally) deletes segment objects that are no
longer referenced by the latest published checkpoint.
Two operations on the [`Cleaner`] trait:
- **`collect(low, high)`** — dry-run inspection. Walks the metadata
tree, returns `(referenced, unreferenced)` lists of segment
objects.
- **`prune(low, high)`** — `collect` + delete the unreferenced list
via S3 `DeleteObjects`.
`low` and `high` are checkpoint numbers (`Cno`), inclusive at both
ends; `high == 0` is a sentinel for "use the latest checkpoint".
What is reported and deleted is a `SegmentId`, which names a stored
object. Under `BlockPtrFormat::PartedSegment` — the default since
`hyperfile 0.7.0` — and `StreamSegment` (`hyperfile 0.11.0`), one
checkpoint may be several objects, `7.0`, `7.1`, … — and they are
reclaimed separately, because a checkpoint
abandoned partway leaves its partials behind under the same number
the retry then reuses. Under every other format an object is a
checkpoint and prints as the plain number.
## Compaction
`prune` reclaims an object nothing points into. An object that is 5%
live is not that object: rewriting those few blocks elsewhere is what
turns it into one `prune` can take. The same rewrite is what makes a
read cheaper, since a request breaks where the object changes.
Two more operations, on `S3Cleaner` itself:
- **`report(low, high)`** — read-only. Per-object liveness and what
could be given back: `dead()` is `prune`'s work, `compactable(ratio)`
estimates what a rewrite would cost and free.
- **`compact(low, high, &opts)`** — rewrites the live blocks of objects
at or below `max_live_ratio`, as one checkpoint. Deletes nothing;
`prune` does that afterwards.
```rust,no_run
use hyperfile_cleaner::{Cleaner, compact::CompactOptions, s3::S3Cleaner};
# async fn run() -> std::io::Result<()> {
let cleaner = S3Cleaner::new("s3://my-bucket/path/to/file").await?;
let report = cleaner.report(1, 0).await?;
let (objects, rewrite, freed) = report.compactable(0.5);
println!("{objects} objects: rewrite {rewrite} bytes to free {freed}");
let out = cleaner.compact(1, 0, &CompactOptions::default()).await?;
assert!(!out.content_moved());
cleaner.prune(1, 0).await?;
# Ok(())
# }
```
The rewrite is `hyperfile`'s, not this crate's: `fs_relocate` marks the
blocks to be rewritten with their current bytes, reading them with the
read path's own coalescing, and one flush writes them sorted by block
index and packed. Holes and explicit zero blocks are left alone, so a
sparse file stays sparse, and neither `mtime` nor `ctime` moves.
**A compaction is not a change of content.** It has to publish a new
checkpoint, so `last_cno` advances by one. `hyperfile` marks that
checkpoint `FLAG_COMPACTED`, and `fs_content_cno()` stays where it was:
that is the number to compare when asking whether a file changed, not
`last_cno`. The mark holds for one checkpoint, so two passes in a row
would move `content_cno` on the second; `compact` refuses to start on a
checkpoint that is already marked unless `allow_consecutive` is set.
`CompactOutcome::content_moved()` says whether consumers will see the
pass as a change — they will when an earlier flush failed and gave its
number up, which the mark does not span.
**By default compaction requires exclusive write access.** It is a
writer, and it writes bytes it read earlier, so it runs with
`FlushConflictPolicy::FailFast` and a concurrent publish makes the pass
fail rather than overwrite the other writer's newer data. It also
refuses to run when the container's prefix holds keys that are neither
the inode nor segment objects, because a WAL is the usual reason and a
compactor that does not read the log would publish over entries
recovery has not replayed.
**`CompactOptions::beside_writer` compacts while the writer runs.** The
pass writes its copies and their map under the published number, and
*attaches* them to the stored inode: one conditional write that sets a
bit and puts the compaction in the inode object's metadata, nothing else
changed. `last_cno` and `content_cno` both stay. A writer of `hyperfile`
0.12 or later takes the compaction in before its next publish of data,
keeping every block it changed since, rather than failing as on a second
writer; until then readers read the objects as they were. Needs
`PartedSegment` or `StreamSegment`. `CompactOutcome::attached` says what
was attached and to which number. The pass fails with `ResourceBusy`
while another compaction is attached, or while the writer has a
checkpoint on storage it has not yet pointed the inode at, and with
`TimedOut` past `CompactOptions::max_age`; either is worth trying again
later. `Report::attached` shows a compaction not yet taken in. See
`docs/compaction.md` in `hyperfile`.
**`prune_with` keeps what readers may still read.** `prune` takes every
object the newest version does not point into, except what an attached
compaction points into: those are kept by every prune, and a prune
refuses, deleting nothing, when the inode says a compaction is attached
and its metadata does not read. A reader that opened an older version, a
read planned under one and fetched outside the handle, and a compaction
that has written its copies and not yet attached them all need objects
the newest version does not name. `PruneOptions::grace` keeps everything
a version current at any time in the last `grace` points into, and every
object written in it. Make it longer than any reader holds a version,
and set `CompactOptions::max_age` below it. The objects a compaction
empties become reclaimable once the writer has taken it in.
## Usage
```rust,no_run
use hyperfile_cleaner::{Cleaner, s3::S3Cleaner};
# async fn run() -> std::io::Result<()> {
let cleaner = S3Cleaner::new("s3://my-bucket/path/to/file").await?;
// Dry-run: see what would be deleted.
let (refed, unrefed) = cleaner.collect(1, 0).await?;
println!("referenced segments : {:?}", refed);
println!("unreferenced segments: {:?}", unrefed);
// Actually delete.
cleaner.prune(1, 0).await?;
// Beside readers and a compactor: keep what was current in the last hour.
use hyperfile_cleaner::s3::PruneOptions;
let opts = PruneOptions { grace: std::time::Duration::from_secs(3600), ..Default::default() };
let deleted = cleaner.prune_with(1, 0, &opts).await?;
# Ok(())
# }
```
For tools that already have a configured `aws_sdk_s3::Client`, use
`S3Cleaner::with_client(&client, root_uri)` instead of `new`.
## Choosing the hyperfile build
`hyperfile` comes in two builds that cannot share a binary: `blocking`, its
direct-API build, and `reactor`, its default. They link `btree-ondisk` with `rc`
and `arc` nodes respectively. This crate follows whichever is selected:
```toml
# default: hyperfile's `blocking` build
hyperfile-cleaner = "0.10"
# alongside a crate that uses hyperfile with its default features
hyperfile-cleaner = { version = "0.10", default-features = false, features = ["reactor"] }
```
Selecting both, or neither, fails to build — in `btree-ondisk`, which is compiled
first, so its error is the one shown.
`collect`, `prune` and `report` return futures that are `Send` in both builds.
`compact`'s is not, in either: it drives a `Hyper`, which under `blocking` holds
`Rc` nodes and under `reactor` is not `Sync` while its `fs_flush` holds a `&Hyper`
across an await. Await it where it is created rather than spawning it.
## Limitations
- The cleaner does not coordinate with in-flight writers. It
snapshots the inode at the start of the call. New segments
published after that point are outside the configured range and
remain untouched, as are the partials of a checkpoint a writer has
open — those carry the number above the published one.
- Compaction is offline: it needs exclusive write access for the pass,
and a container holding a WAL is refused unless the caller overrides
it. Compacting a container a writer is actively publishing to needs a
background task inside `hyperfile`, not a separate process.
- On a parted container a pass of any size is written out as partial
segments as it goes, so it holds no more than the dirty-data limits
in memory, and it is still one checkpoint. Where no partial can be
written — a container that is not parted, or partials turned off — a
pass holds everything it relocates until its one flush and is capped
at `UNBOUNDED_PASS_BYTES` (512 MiB). A container with more to rewrite
than one pass takes several, and each pass after the first in a row
moves `content_cno` once.
- The handle that rewrites uses `CompactOptions::runtime` if given —
pass the configuration the container's writer uses, and compaction
writes objects of the same shape — and otherwise
`compaction_runtime_default()`, 64 MiB partials. Only
`flush_conflict_policy` is overridden, to `FailFast`.
- An inline file or a device node has no segments; `collect`, `prune`,
`report` and `compact` refuse them.
- An object holding a live metadata node is not a compaction candidate,
except a data part of a `StreamSegment` container. A node is relocated
when the flush rewrites it, which needs something under it dirtied,
and which keys a node covers is not visible from here; the stream
layout is the exception because it keeps every leaf in the object
with its data, so rewriting the data rewrites the leaf.
- A container whose inode records a format this build does not know is
refused. Cleaners up to 0.10.5 instead read it as `MicroGroup`, and
on a `StreamSegment` container that reports every object unreferenced:
**do not run a cleaner older than 0.11 on a container written by
hyperfile 0.11 in the stream format.**
## Status
> :warning: Under development. Not recommended for production.
## License
Apache-2.0.
## Contributing
PRs welcome.