Crate hyperfile_cleaner 

hyperfile-cleaner enumerates and removes unreferenced segments from a hyperfile.

Model

A hyperfile is stored on S3 as a single inode object plus a sequence of segment objects. Each segment is keyed by a monotonically increasing SegmentId and contains both the data blocks written in that flush and the metadata B+tree nodes that index them.

Because hyperfile is copy-on-write, segments accumulate as the file is modified. Once a segment is no longer referenced by the latest published checkpoint’s metadata tree, its storage can be reclaimed.

The Cleaner trait provides two operations:

• Cleaner::collect walks the metadata tree of the latest (or a caller-specified) checkpoint and returns (referenced, unreferenced) segment lists for a [low, high] segid range. Useful for dry-run / inspection.
• Cleaner::prune performs the same walk and additionally deletes the unreferenced segments via S3 DeleteObjects.

Concurrency caveat

The cleaner does not coordinate with in-flight writers. It takes a snapshot of the inode at the start of the call and operates on that. New segments published after that point are always outside the configured range (because high was fixed at snapshot time), so the prune is safe against concurrent appends. It is, however, the caller’s responsibility to ensure low is not also being actively reused.

Example

use hyperfile_cleaner::{Cleaner, s3::S3Cleaner};

let cleaner = S3Cleaner::new("s3://my-bucket/path/to/file").await?;

// Inspect what would be deleted (dry-run): pass high=0 to
// mean "use the latest checkpoint".
let (refed, unrefed) = cleaner.collect(1, 0).await?;
println!("referenced  : {:?}", refed);
println!("unreferenced: {:?}", unrefed);

// Actually delete the unreferenced range.
cleaner.prune(1, 0).await?;

Modules

compact

Offline compaction: rewriting the live blocks of mostly-dead objects so the objects can be reclaimed, and so a read of one file range costs fewer requests.

s3

S3-backed cleaner implementation.

Traits

Cleaner

Operations supported by a hyperfile cleaner backend.