Struct SegmentCatalog 

pub struct SegmentCatalog { /* private fields */ }

Segment metadata catalog with manifest persistence.

Tracks all segment metadata organized by shard, along with measurement schemas. Changes are persisted to a manifest WAL file, with periodic snapshots to prevent unbounded manifest growth.

Persistence

Changes are appended to manifest.wal as length-prefixed, CRC-32C framed postcard entries. Periodic snapshots written to manifest.snapshot.bin. On startup: load latest snapshot, replay WAL entries.

Concurrency

The catalog currently uses a single &mut self borrow (i.e. external synchronisation via an RwLock or Mutex) for all shards. This is acceptable because catalog mutations (segment add/remove, schema updates) are infrequent (flush/compaction cadence) and short-lived.

If catalog contention ever becomes measurable under very high shard counts, the internal BTreeMap<ShardId, Vec<…>> can be replaced with a DashMap<ShardId, Vec<…>> to allow per-shard concurrent access without changing the external API.

Implementations

impl SegmentCatalog

pub fn new(manifest_dir: impl Into<PathBuf>) -> Result<Self>

Create a new empty catalog persisted in manifest_dir.

Errors

Returns an error if the manifest directory cannot be created.

pub fn open(manifest_dir: impl Into<PathBuf>) -> Result<Self>

Open an existing catalog, loading from snapshot + replaying manifest WAL.

Errors

Returns an error if the snapshot or manifest cannot be read/parsed.

pub fn add_segment(&mut self, entry: SegmentCatalogEntry) -> Result<()>

Add a segment to the catalog.

Errors

Returns an error if the manifest cannot be written.

pub fn remove_segment( &mut self, segment_id: SegmentId, ) -> Result<Option<SegmentCatalogEntry>>

Remove a segment from the catalog by its ID.

Returns the removed entry if found.

Errors

Returns an error if the manifest cannot be written.

pub fn get_segments_for_shard( &self, shard_id: &ShardId, ) -> &[SegmentCatalogEntry]

Get all segments for a specific shard.

pub fn soft_delete_segment( &mut self, segment_id: SegmentId, now_ms: u64, ) -> Result<bool>

Mark a segment as soft-deleted.

The segment remains in the catalog and is still queryable by code that explicitly opts in to soft-deleted segments — but the default query path (via active_segments_for_measurement) hides it. A subsequent call to remove_segment will hard-delete it.

pub fn active_segments_for_measurement( &self, measurement: &str, ) -> Vec<&SegmentCatalogEntry>

Get all active segments belonging to a specific measurement.

Excludes segments that are soft-deleted (waiting for GC), and every segment of a measurement pending a hard delete — a “drop” that leaves its rows scannable for the whole grace period is not a drop. The segments themselves are untouched, so a restore before the deadline costs nothing and loses nothing.

pub fn retired_segments(&self) -> Vec<&SegmentCatalogEntry>

Segments that have been retired — taken out of every query — but whose files are still on disk.

A retirement unlinks the file in the same step that removes the catalog entry; an entry reaches this state only when a running scan had already been handed the path, so garbage collection is what finishes the job once the reader has gone.

pub fn all_segments(&self) -> Vec<&SegmentCatalogEntry>

Get all segments across all shards.

pub fn segments_for_measurement( &self, measurement: &str, ) -> Vec<&SegmentCatalogEntry>

Get all segments belonging to a specific measurement.

pub fn max_segments_per_shard_measurement(&self) -> usize

Get the total number of segments. The largest number of active segments any one (shard, measurement) holds — the count a compaction pass can actually reduce, and so the number write backpressure is keyed on.

pub fn memory_bytes(&self) -> usize

Approximate heap bytes held by the catalog.

One entry per segment, each carrying a measurement name, a path and its per-column statistics — so this grows with segment count, not with row count, and it is the term that a long-running gateway accumulates between compactions. The schema registry and the tombstone set are counted with it: all three live for the process’s lifetime, which is what makes them worth a number rather than an estimate.

pub fn shard_count(&self) -> usize

Shards holding at least one active segment.

Derived rather than tracked: a TimeIndex was maintained beside the catalog on every flush, compaction and retirement to answer this and nothing else, and its own count drifted — a shard emptied by per-measurement retention kept its entry and stayed counted.

pub fn segment_count(&self) -> usize

Total number of segments the catalog holds, in any state.

pub fn next_segment_id(&mut self) -> SegmentId

Allocate the next segment ID.

pub fn manifest_seq(&self) -> u64

Get the current manifest sequence number.

pub fn rollups(&self) -> &BTreeMap<String, RollupRecord>

Every persisted rollup, by name.

pub fn set_rollup(&mut self, name: &str, definition: Vec<u8>) -> Result<()>

Persist a rollup definition, durably.

Errors

Returns an error if the manifest cannot be written or synced.

pub fn set_rollup_state(&mut self, name: &str, state: Vec<u8>) -> Result<()>

Persist a rollup’s materialisation state, durably.

Called after the rollup’s points are durable, never before: a watermark that outlives the rows it claims is a permanent hole in the tier, because nothing recomputes a bucket the watermark has already passed.

Errors

Returns an error if the manifest cannot be written or synced.

pub fn remove_rollup(&mut self, name: &str) -> Result<bool>

Delete a rollup definition and its state.

Errors

Returns an error if the manifest cannot be written.

pub fn pending_measurement_drops(&self) -> &BTreeMap<String, u64>

Every measurement pending a hard delete, and the unix-ms deadline each was given.

pub fn is_measurement_pending_drop(&self, measurement: &str) -> bool

Whether measurement is pending a hard delete.

pub fn set_measurement_pending_drop( &mut self, measurement: &str, deadline_ms: u64, ) -> Result<()>

Mark measurement pending a hard delete at deadline_ms, durably.

Errors

Returns an error if the manifest cannot be written or synced.

pub fn cancel_measurement_pending_drop( &mut self, measurement: &str, ) -> Result<bool>

Cancel a pending drop \u2014 a restore, or the hard delete that finally acted on it \u2014 durably.

Returns whether measurement was pending.

Errors

Returns an error if the manifest cannot be written or synced.

pub fn tombstones(&self) -> &TombstoneSet

The tombstones recorded by deletes.

pub fn record_tombstones(&mut self, tombstones: &[Tombstone]) -> Result<()>

Record the tombstones produced by a delete, durably.

The manifest append is fsynced before this returns, so a delete that has been acknowledged has been persisted — which is the property the previous design lacked, because it kept tombstones only in memory and in a data WAL that the next flush was free to truncate.

Errors

Returns an error if the manifest cannot be written or synced.

pub fn complete_compaction( &mut self, output: SegmentCatalogEntry, inputs: &[SegmentId], now_ms: u64, ) -> Result<()>

Register a compaction’s output and retire its inputs, as one durable step.

Adds output, soft-deletes every input, and extends every tombstone that named an input to the output. That last part is what closes the window in which a delete issued while the merge ran was undone: the merge applied the tombstones it snapshotted, the late tombstone named only the inputs, and once GC removed them the tombstone was reclaimed with its rows alive in the output.

Errors

Returns an error if the manifest cannot be written.

pub fn reclaim_tombstones(&mut self) -> Result<usize>

Reclaim tombstones whose segments have all left the catalog.

A segment leaves the catalog only by being rewritten — compaction applies tombstones as it merges — or by being deleted outright, so once none of a tombstone’s segments remains, no stored row can still match it. Returns the number reclaimed.

Errors

Returns an error if the manifest cannot be written.

pub fn set_schema(&mut self, schema: MeasurementSchema) -> Result<()>

Register or update a measurement schema.

Errors

Returns an error if the manifest cannot be written.

pub fn remove_schema( &mut self, measurement: &str, ) -> Result<Option<MeasurementSchema>>

Remove a measurement schema.

Errors

Returns an error if the manifest cannot be written.

pub fn wal_floor(&self) -> u64

The WAL floor: the highest data-WAL sequence number known to be fully represented by the segments this catalog registers.

open() replays only records above it.

pub fn set_wal_floor(&mut self, sequence_no: u64) -> Result<()>

Raise the WAL floor to sequence_no.

Persisted before it is applied, like every other catalog change. A floor never moves backwards; a lower value is a no-op.

Errors

Returns an error if the manifest append fails.

pub fn get_schema(&self, measurement: &str) -> Option<&MeasurementSchema>

Look up a measurement schema.

pub fn schema_names(&self) -> Vec<String>

Returns all schema names.

pub fn all_schemas(&self) -> impl Iterator<Item = (&String, &MeasurementSchema)>

Returns an iterator over all (name, schema) pairs.

pub fn force_snapshot(&mut self) -> Result<()>

Force a snapshot of the current catalog state.

Errors

Returns an error if the snapshot cannot be written.

pub fn manifest_syncs(&self) -> u64

How many times the manifest has been fsynced since this catalog was opened.

One per catalog transition, not per append: a compaction retiring four segments is one, and so is a delete however many tombstones it produces. Also exported as chronix_catalog_fsync_total.

pub fn in_one_sync<T>( &mut self, body: impl FnOnce(&mut Self) -> Result<T>, ) -> Result<T>

Run body as one durable manifest transition: the appends it makes are written but fsynced once, when it returns.

A compaction of four segments is six appends — the output, four soft-deletes and the Compacted record — and a delete is one per tombstone. On flash-backed storage the fsync rate is the wear rate, which is why the WAL has a periodic policy.

Everything body appended is durable when this returns, on the error path as much as the success path, so a transition that failed half-way is as persistent as it would be if each append had synced. A crash during body leaves what it always could: the manifest is a replayed log, so a partial transition means an input segment still live beside its compaction output, which reads deduplicate and the next compaction clears.

Only for a transition that ends in a sync, and only where the caller does nothing irreversible until it returns: a file unlinked while the appends are still unsynced would survive a crash as a catalog entry naming a path that is gone.

Errors

Returns body’s error, or the error of the fsync that follows it.

pub fn sync_manifest(&mut self) -> Result<()>

Flush any pending manifest writes to durable storage.

Call this after a batch of mutations to ensure all appended entries survive a power failure. This is a no-op if there are no pending un-synced writes.

pub fn write_snapshot_to(&self, manifest_dir: &Path) -> Result<()>

Write this catalog, as it stands, as the whole catalog of a database rooted at manifest_dir — a directory that is not this one.

This is what a backup captures instead of copying catalog/ file by file. Copying was the defect: a snapshot landing mid-copy replaces manifest.snapshot.bin and truncates manifest.wal, so a reader walking the directory can pair the old snapshot with the emptied log and lose every transition since — silently, and reported as a successful backup. One encode of one consistent in-memory state cannot be torn, and the empty log beside it says there is nothing to replay.

The caller must hold the catalog read lock for as long as it needs the segments this names to still be there.

Errors

Returns an error if the directory cannot be created or written.

Trait Implementations

impl Debug for SegmentCatalog

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.