Skip to main content

run_stream

faucet_stream

Function run_stream 

Source 
pub async fn run_stream<S, Si>(
    pages: S,
    sink: &Si,
    options: RunStreamOptions,
) -> Result<PipelineResult, FaucetError>
where
    S: Stream<Item = Result<StreamPage, FaucetError>> + Unpin,
    Si: Sink + ?Sized,
Expand description

Run a streaming pipeline, writing each StreamPage to the sink as it arrives and persisting bookmarks per page.

This keeps memory usage bounded — only one page of records is held at a time. The stream comes from Source::stream_pages (or any Stream<Item = Result<StreamPage, FaucetError>> a caller assembles directly).

Bookmark semantics: whenever a page carries Some(bookmark), the sink is flushed and the bookmark is persisted (when state_store and state_key are both Some) before the next page is polled. Sources that only know their bookmark after seeing every record emit Some on the final page; CDC-style sources emit Some per committed transaction and get per-transaction durability automatically.

Returns the cumulative PipelineResult — records_written is the sum across all pages and bookmark is the last per-page bookmark observed.