Struct GenericJoiner 

pub struct GenericJoiner<G, M: JoinMode, const BODY_SIZE: usize = DEFAULT_BODY_SIZE>
where
    G: ChunkGet<BODY_SIZE>,
{ /* private fields */ }

Generic async joiner parameterized by chunk mode.

Implementations

impl<G, M, const BODY_SIZE: usize> GenericJoiner<G, M, BODY_SIZE>

where G: ChunkGet<BODY_SIZE> + 'static, M: JoinMode + MaybeSend + Sync,

pub async fn for_each_chunk<F>(self, f: F) -> Result<()>

where F: FnMut(u64, &Bytes) -> ControlFlow<()>,

Visit each leaf body as it lands, out of order, tagged with its absolute offset. Peak memory is the in-flight width. f returns Break to stop early, which drops the stream and cancels in-flight fetches.

pub async fn try_for_each_window<B, F>( self, window: usize, init: B, f: F, ) -> Result<B>

where F: FnMut(B, &[u8]) -> ControlFlow<B, B>,

Fold contiguous in-order byte runs through f without buffering the file. window bounds peak memory (see into_windowed_reader). f returns Break(acc) to stop early with the carried accumulator; otherwise the final accumulator is returned at stream end.

impl<G, M, const BODY_SIZE: usize> GenericJoiner<G, M, BODY_SIZE>

where G: ChunkGet<BODY_SIZE>, M: JoinMode + MaybeSend + Sync,

pub async fn new(getter: G, input: M::RootRef) -> Result<Self>

Create an async joiner from a root reference.

pub async fn open_streaming(getter: G, input: M::RootRef) -> Result<Self>

Open for a forward streaming download, skipping the eager frontier expansion.

new pre-expands a balanced subtree frontier with a level-synchronous BFS, so one slow intermediate stalls every byte until its whole level resolves. The chunk-granular offset stream descends the tree concurrently from any seed, so a forward download needs no pre-expansion: seed it with the root alone and let the bounded pool walk the rest with no per-level barrier, so a slow intermediate lags only its own subtree rather than the whole download. Use this for download_into and into_offset_stream_chunked; the random-access reads (read_all, into_windowed_reader) want the balanced frontier and should open with new. The root is re-fetched once as the stream descends it (a single chunk).

pub fn with_concurrency(self, concurrency: usize) -> Self

Set concurrency level for prefetching.

pub const fn size(&self) -> u64

Total file size.

pub const fn position(&self) -> u64

Current read position.

pub const fn root(&self) -> &ChunkAddress

Root address.

pub async fn read_range(&self, offset: u64, len: usize) -> Result<Vec<u8>>

Read a range of bytes with concurrent fetching using the cached frontier.

pub async fn read_all(&self) -> Result<Vec<u8>>

Read entire file into memory.

pub fn seek(&mut self, pos: SeekFrom) -> Result<u64>

Update read position (synchronous — just updates internal state).

pub fn into_stream(self) -> impl Stream<Item = Result<Bytes>>

Convert into a stream of leaf chunk bodies.

pub fn into_offset_stream(self) -> impl Stream<Item = Result<(u64, Bytes)>>

Convert into a stream of (byte_offset, leaf_body) pairs fetched with bounded concurrency and yielded out of order as each leaf lands.

Unlike into_stream, which walks subtrees one at a time and emits bytes in file order, this keeps up to concurrency subtree fetches in flight and yields each leaf the moment its subtree resolves, tagged with its absolute byte offset in the file. Reassembling the pairs by offset reproduces the file; nothing is buffered into a single contiguous result, so peak memory is bounded by the in-flight width, not the file size. Set the width with with_concurrency.

pub fn into_offset_stream_chunked( self, ) -> impl Stream<Item = Result<(u64, Bytes)>>

where G: 'static,

Convert into a stream of (byte_offset, leaf_body) pairs fetched with per-chunk bounded concurrency and yielded out of order as each leaf lands.

Where into_offset_stream keeps up to concurrency subtree fetches in flight and walks each subtree as a sequential descent, this walks the tree at chunk granularity: every intermediate and leaf fetch competes for the same bounded in-flight pool, so up to concurrency individual chunks are in flight regardless of how few top-level subtrees the tree has. Effective leaf concurrency is min(concurrency, leaf_count) even for a tree that fans into one wide subtree, which the subtree-granular variant cannot reach.

The walk is a bounded producer/worker model expressed without a spawned task: a queue of pending tree nodes feeds a FuturesUnordered pool refilled to the width each step. An intermediate node resolves to its overlapping children (re-queued); a leaf resolves to its (byte_offset, body) pair. A leaf fetch that fails is re-enqueued (up to a bounded retry budget) and the worker pulls the next node, so a slow or flaky chunk retries without holding its slot or gating ready leaves.

Peak memory is bounded by the pool width plus the pending queue, not the file size. Set the width with with_concurrency. Reassembling the pairs by offset reproduces the file, byte-for-byte equal to read_all.

pub fn into_offset_stream_chunked_range( self, start: u64, len: u64, ) -> impl Stream<Item = Result<(u64, Bytes)>>

where G: 'static,

Like into_offset_stream_chunked but restricted to the byte range [start, start + len).

Only subtrees and intermediate children overlapping the range are walked, and the first and last partial leaves are clipped so each emitted (offset, body) lies inside the range. Offsets stay absolute in the file, so reassembling the pairs over [start, start + len) reproduces read_range byte-for-byte. A whole-file range equals into_offset_stream_chunked; an empty or out-of-bounds range yields an empty stream.

pub fn into_reader(self) -> JoinerReader<G, M, BODY_SIZE>

Convert into an AsyncRead reader.

impl<G, M, const BODY_SIZE: usize> GenericJoiner<G, M, BODY_SIZE>

where G: ChunkGet<BODY_SIZE> + 'static, M: JoinMode + MaybeSend + Sync,

pub fn into_windowed_reader( self, window: usize, ) -> WindowedReader<G, M, BODY_SIZE>

Seek-and-play reader over a window that slides with the read cursor. Peak memory is window leaf bodies: in-flight plus buffered leaves never exceed window. window is clamped to at least 1; a value of at least 2 * concurrency keeps the pool full even while the head leaf is the straggler.

impl<G, M, const BODY_SIZE: usize> GenericJoiner<G, M, BODY_SIZE>

where G: ChunkGet<BODY_SIZE> + 'static, M: JoinMode + MaybeSend + Sync,

pub async fn download_into<S: WriteAt>(self, sink: S) -> Result<()>

Reassemble the whole file into sink, writing each out-of-order leaf at its offset. Peak memory is the in-flight width, never the file size. Cancel-safe: dropping the returned future drops the chunk stream (cancelling in-flight fetches) and the sink; a partially-written sink is sparse-valid and safe to resume.

pub async fn download_into_with_progress<S: WriteAt, F: FnMut(u64, u64)>( self, sink: S, on_progress: F, ) -> Result<()>

As download_into, invoking on_progress(written, total) after each leaf lands.

Trait Implementations

impl<G, M, const BODY_SIZE: usize> Debug for GenericJoiner<G, M, BODY_SIZE>

where G: ChunkGet<BODY_SIZE>, M: JoinMode,

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.