Struct RestStream 

pub struct RestStream { /* private fields */ }

A configured REST API stream that handles pagination, auth, and extraction.

Implementations

impl RestStream

pub fn new(config: RestStreamConfig) -> Result<Self, FaucetError>

Create a new stream from the given configuration.

Returns FaucetError::Transform immediately if any RenameKeys transform contains an invalid regex pattern — fail-fast before any HTTP requests are made.

pub async fn fetch_all(&self) -> Result<Vec<Value>, FaucetError>

Fetch all records across all pages as raw JSON values.

When partitions are configured, the stream is executed once per partition and all results are concatenated.

When replication_method is Incremental and replication_key + start_replication_value are both set, records at or before the bookmark are filtered out.

pub async fn fetch_all_as<T: for<'de> Deserialize<'de>>( &self, ) -> Result<Vec<T>, FaucetError>

Fetch all records and deserialize into typed structs.

pub async fn infer_schema(&self) -> Result<Value, FaucetError>

Infer a JSON Schema for this stream’s records.

If a schema is already set on the config, it is returned immediately without making any HTTP requests.

Otherwise the stream fetches up to schema_sample_size records (respecting max_pages) and derives a JSON Schema from them. Fields that are absent in some records, or that carry a null value, are marked as nullable (["<type>", "null"]).

Set schema_sample_size to 0 to sample all available records.

pub async fn fetch_all_incremental( &self, ) -> Result<(Vec<Value>, Option<Value>), FaucetError>

Fetch all records in incremental mode, returning the records along with the maximum value of replication_key observed across those records.

The returned bookmark should be persisted by the caller and passed back as start_replication_value on the next run.

If no replication_key is configured, this behaves identically to fetch_all and the bookmark is None.

pub fn stream_pages( &self, ) -> Pin<Box<dyn Stream<Item = Result<Vec<Value>, FaucetError>> + Send + '_>>

Stream records page-by-page, yielding one Vec<Value> per page as it arrives.

Unlike fetch_all, this does not wait for all pages to be fetched before returning — callers can process each page immediately.

Note: partitions are not supported by stream_pages. Use fetch_all for multi-partition streams.

use faucet_stream::{RestStream, RestStreamConfig};
use futures::StreamExt;

let stream = RestStream::new(RestStreamConfig::new("https://api.example.com", "/items"))?;
let mut pages = stream.stream_pages();
while let Some(page) = pages.next().await {
    let records = page?;
    println!("got {} records", records.len());
}