Skip to main content

spool_or_buffer

Function spool_or_buffer 

Source
pub async fn spool_or_buffer<S, E>(
    stream: S,
    dir: &Path,
    memory_cap: usize,
) -> Result<Box<dyn NarSource>, StoreError>
where S: Stream<Item = Result<Bytes, E>> + Send + Unpin, E: Display + Send,
Expand description

Turn a one-shot byte stream into a re-openable NarSource, bounded either way.

Preferred path: spool to a file in dir in NAR_CHUNK_BYTES steps, peak one chunk. Fallback: if the spool file cannot be created โ€” no dir, no permission, a full volume โ€” buffer in memory instead, hard-capped at memory_cap, refusing past it with StoreError::TooLarge.

The fallback is chosen before any bytes are read, deliberately. A spool that fails halfway has already consumed part of a one-shot stream and cannot be restarted, so mid-write faults surface as errors rather than silently switching strategy and truncating the upload.

Tier honesty: the fallback path is bounded, not streaming โ€” a machine with no usable spool directory has a memory_cap-sized worst case per concurrent ingest, and NARs above the cap are refused rather than cached. That is a deliberate trade against the alternative, which is the pod dying and taking every in-flight build with it.

ยงErrors

Propagates a read error from stream, a write error to the spool file, or StoreError::TooLarge when the memory fallback is in use and exceeded.