Skip to main content

process_ndjson_chunked_remote

Function process_ndjson_chunked_remote 

Source
pub async fn process_ndjson_chunked_remote<R: BufRead, W: Write>(
    view_definition: SofViewDefinition,
    input: R,
    output: W,
    content_type: ContentType,
    config: ChunkConfig,
    remote_config: &RemoteResolveConfig,
) -> Result<ProcessingStats, SofError>
Expand description

Streaming NDJSON processing with remote resolve() enabled.

Like process_ndjson_chunked, but for each chunk it prefetches references pointing at trusted (allowlisted) servers and folds them into that chunk’s resolution pool before row generation. A single RemoteResolver is shared across all chunks, so a reference recurring across chunks is fetched once (bounded LRU cache) and SOF_RESOLVE_MAX_FETCHES is a per-stream cap.

In-bundle resolution remains per-chunk (a reference to a resource in another chunk of the same input is not resolved locally). When remote_config is inactive this is equivalent to process_ndjson_chunked.