hyperfile_cleaner/
lib.rs

1//! `hyperfile-cleaner` enumerates and removes unreferenced
2//! segments from a [hyperfile](https://crates.io/crates/hyperfile).
3//!
4//! ## Model
5//!
6//! A hyperfile is stored on S3 as a single inode object plus a
7//! sequence of segment objects. Each segment is keyed by a
8//! monotonically increasing `SegmentId` and contains both the
9//! data blocks written in that flush and the metadata B+tree
10//! nodes that index them.
11//!
12//! Because hyperfile is copy-on-write, segments accumulate as
13//! the file is modified. Once a segment is no longer referenced
14//! by the latest published checkpoint's metadata tree, its
15//! storage can be reclaimed.
16//!
17//! The [`Cleaner`] trait provides two operations:
18//!
19//! - [`Cleaner::collect`] walks the metadata tree of the latest
20//!   (or a caller-specified) checkpoint and returns
21//!   `(referenced, unreferenced)` segment lists for a
22//!   `[low, high]` segid range. Useful for dry-run / inspection.
23//! - [`Cleaner::prune`] performs the same walk and additionally
24//!   deletes the unreferenced segments via S3 `DeleteObjects`.
25//!
26//! ## Concurrency caveat
27//!
28//! The cleaner does not coordinate with in-flight writers. It
29//! takes a snapshot of the inode at the start of the call and
30//! operates on that. New segments published after that point are
31//! always outside the configured range (because `high` was fixed
32//! at snapshot time), so the prune is safe against concurrent
33//! appends. It is, however, the caller's responsibility to ensure
34//! `low` is not also being actively reused.
35//!
36//! ## Example
37//!
38//! ```no_run
39//! use hyperfile_cleaner::{Cleaner, s3::S3Cleaner};
40//!
41//! # async fn run() -> std::io::Result<()> {
42//! let cleaner = S3Cleaner::new("s3://my-bucket/path/to/file").await?;
43//!
44//! // Inspect what would be deleted (dry-run): pass high=0 to
45//! // mean "use the latest checkpoint".
46//! let (refed, unrefed) = cleaner.collect(1, 0).await?;
47//! println!("referenced  : {:?}", refed);
48//! println!("unreferenced: {:?}", unrefed);
49//!
50//! // Actually delete the unreferenced range.
51//! cleaner.prune(1, 0).await?;
52//! # Ok(())
53//! # }
54//! ```
55
56use std::io::Result;
57use hyperfile::SegmentId;
58
59pub mod s3;
60
61/// Operations supported by a hyperfile cleaner backend.
62///
63/// Both methods take a `[low, high]` segment-id range. `high == 0`
64/// is a sentinel meaning "use the latest published checkpoint
65/// (`Inode::get_last_seq`)". `low` is inclusive; a typical value is
66/// `1` to cover the entire file history.
67pub trait Cleaner {
68    /// Walk the metadata tree of the checkpoint identified by `high`
69    /// (or the latest if `high == 0`) and return
70    /// `(referenced, unreferenced)` segment id lists within
71    /// `[low, high]`.
72    ///
73    /// Useful for dry-run inspection before invoking [`Cleaner::prune`].
74    fn collect(
75        &self,
76        low: SegmentId,
77        high: SegmentId,
78    ) -> impl Future<Output = Result<(Vec<SegmentId>, Vec<SegmentId>)>> + Send;
79
80    /// Walk the metadata tree as in [`Cleaner::collect`] and then
81    /// delete the unreferenced segment objects from the underlying
82    /// store.
83    ///
84    /// Errors if any individual delete batch fails; in that case
85    /// previously-deleted batches stay deleted (S3 has no
86    /// transactional batch). The returned error message mentions
87    /// the failed object count and a sample failed key.
88    fn prune(
89        &self,
90        low: SegmentId,
91        high: SegmentId,
92    ) -> impl Future<Output = Result<()>> + Send;
93}
hyperfile_cleaner/lib.rs

hyperfile_cleaner/
lib.rs
