Struct BucketContainers 

pub struct BucketContainers { /* private fields */ }

The physical files backing the weak-super-k-mer buckets.

One container per atlas rather than one file per bucket, taking a 16,384 bucket build from 16,385 files to 129. Two things make this cheap rather than intricate. Every bucket already writes under its atlas’s mutex (SharedBucketSink::append_bucket), and a container holds exactly one atlas’s buckets, so a container is only ever written by the thread holding that lock and needs no lock of its own. And a flush no longer opens anything: BucketFile::open_existing cost an openat, seven unbuffered reads to re-read a 42-byte header, a revalidation, an lseek and a close on every 64 KiB flush, which is about eleven syscalls times the 14.4 million flushes a full-corpus build performs. A container flush is one pwrite.

The measured prize is smaller than that count suggests – partitioning’s whole system time is 436.7 s of CPU across 64 threads, so the ceiling is a couple of seconds of wall – and larger somewhere unexpected. XFS speculatively preallocates on extending writes, and with 16,385 repeatedly reopened files it held 332.7 GB for 237.8 GB of data. Writing 128 files instead returns that 94.9 GB.

Implementations

impl BucketContainers

pub fn segment_bytes(&self) -> u64

pub fn len(&self) -> usize

pub fn is_empty(&self) -> bool

pub fn release_segments(&self, container: usize, segments: &[u32])

Releases a consumed bucket’s segments without disturbing its neighbours.

This is not optional, which measurement rather than reasoning settled. The per-file layout unlinked each bucket as local contraction consumed it, and containers cannot: a container is only droppable once all 128 of its buckets are done. Containers do start about 94 GB below the per-file layout, because they do not accumulate XFS speculative preallocation, and the expectation was that this covered it. It does not – the work-directory peak moves out of the end of partitioning and into local contraction, where the containers still hold everything while local-unitig buckets, labels and the edge matrix accumulate on top, and peak disk rose 24.5 GB.

Punching restores the incremental release. Segments are reserved at a 4 KiB multiple, so a punch frees whole filesystem blocks rather than leaving partial ones behind – the one thing raw extents, averaging 16.1 KiB and unaligned, could not have done. Adjacent segments are punched in one call.

pub fn paths(&self) -> impl Iterator<Item = &Path>

Trait Implementations

impl Debug for BucketContainers

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.