Struct ParquetSource 

pub struct ParquetSource { /* private fields */ }

Available on crate features source-parquet and source-rest only.

A source that reads Parquet files into JSON records.

Implementations

impl ParquetSource

pub async fn new( config: ParquetSourceConfig, ) -> Result<ParquetSource, FaucetError>

Build a new Parquet source from config.

Performs eager validation (concurrency > 0, mutually exclusive S3 key/prefix) and pre-builds the S3 client when applicable so it can be reused across concurrent file reads.

Trait Implementations

impl Source for ParquetSource

fn stream_pages<'a>( &'a self, context: &'a HashMap<String, Value>, _batch_size: usize, ) -> Pin<Box<dyn Stream<Item = Result<StreamPage, FaucetError>> + Send + 'a>>

Stream RecordBatches from each resolved file, yielding one StreamPage per Arrow RecordBatch so client-side memory is bounded at O(batch_size * row_width) regardless of total file size.

The trait-level batch_size argument is ignored in favour of ParquetSourceConfig::batch_size — the config is the user-facing knob the README documents, and routing the pipeline-supplied hint through it would silently override an explicit config value.

Cadence:

• batch_size > 0 — passed to ParquetRecordBatchStreamBuilder::with_batch_size. Arrow may emit a smaller batch at row-group boundaries, so an emitted page can be smaller than batch_size.
• batch_size == 0 — the sentinel skips with_batch_size, so the file’s native row-group size drives the page cadence (one page per row-group).

Multi-file scans (glob / S3 prefix) iterate sequentially in sorted order. The first file’s Arrow schema is the reference; any subsequent file with a different schema surfaces as FaucetError::Source naming both paths and the first diverging field — matching the eager fetch_with_context behaviour.

Every page carries bookmark: None — the Parquet source has no incremental-replication mode.

fn fetch_with_context<'life0, 'life1, 'async_trait>( &'life0 self, context: &'life1 HashMap<String, Value>, ) -> Pin<Box<dyn Future<Output = Result<Vec<Value>, FaucetError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, ParquetSource: 'async_trait,

Primary fetch method. Receives context from a parent source’s records.

fn config_schema(&self) -> Value

Return a JSON Schema describing the configuration this source accepts.

fn fetch_all<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Result<Vec<Value>, FaucetError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Self: 'async_trait,

Convenience: fetch with no parent context.

fn fetch_with_context_incremental<'life0, 'life1, 'async_trait>( &'life0 self, context: &'life1 HashMap<String, Value>, ) -> Pin<Box<dyn Future<Output = Result<(Vec<Value>, Option<Value>), FaucetError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Incremental fetch with parent context support.

fn fetch_all_incremental<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Result<(Vec<Value>, Option<Value>), FaucetError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Self: 'async_trait,

Convenience: incremental fetch with no parent context.

fn state_key(&self) -> Option<String>

Stable key under which this source’s incremental-replication bookmark should be persisted in a StateStore.

fn apply_start_bookmark<'life0, 'async_trait>( &'life0 self, _bookmark: Value, ) -> Pin<Box<dyn Future<Output = Result<(), FaucetError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Self: 'async_trait,

Apply a bookmark loaded from a StateStore as this run’s starting point.

fn connector_name(&self) -> &'static str

Stable identifier used as the connector label on metrics and the connector attribute on spans. Defaults to the final segment of std::any::type_name::<Self>(), e.g. "RestSource". Built-in connectors override with a short, friendly snake_case name (e.g. "rest"). Must return a non-empty string; observability decorators fall back to "unknown" in release builds if it is empty (and debug_assert! in debug builds).

fn check<'life0, 'life1, 'async_trait>( &'life0 self, ctx: &'life1 CheckContext, ) -> Pin<Box<dyn Future<Output = Result<CheckReport, FaucetError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Run a fast, non-mutating preflight probe (used by faucet doctor).