Struct ArrowSource 

pub struct ArrowSource { /* private fields */ }

A TableSource over Arrow batches a caller already has.

SparkConnectSource needs a Spark Connect endpoint and IcebergSource needs a local skade warehouse directory. Neither can be pointed at a warehouse reached some other way — a remote skade over its Iceberg-REST front, an Iceberg REST catalog, a Spark whose client lives in the calling process. All of those already produce Arrow, and this is the source that takes it.

use knut_bifrost::source::ArrowSource;
// A one-shot build from batches already in hand.
let source = ArrowSource::from_batches("my-warehouse", "lake.db.people", batches);

What it is NOT

It is a batch read, not change-data-capture. CDC and delta-watching in bifröst (SparkIcebergCdc, SparkNativeCdc, watch_iceberg_delta, IcebergSource’s incremental path) all rest on a snapshot lineage — a before-id and an after-id the engine can diff. A bag of Arrow batches has no such lineage and no engine behind it to ask, so:

SyncMode	what this source does
Full	re-reads the producer every round (idempotent MERGE in the sink makes that safe)
Incremental	ships once, then reports nothing — there is no delta to compute
Cdc	refuses, typed, naming the reason and the sources that can

cdc_capability is None, exactly as the file source’s is. A warehouse reached this way feeds bifröst for a full build, not for streaming.

Implementations

impl ArrowSource

pub fn new( producer: impl Into<String>, batches: impl ArrowBatches + 'static, ) -> Self

Wrap any ArrowBatches producer. producer names it for diagnostics ("skade-remote", "iceberg-rest", "spark41", …).

pub fn from_batches( producer: impl Into<String>, table: impl Into<String>, batches: Vec<RecordBatch>, ) -> Self

A one-shot source over batches already in hand, for exactly one table. Sugar over crate::rel2graph::ArrowTables — the same holder the offline engine reads from, not a second one.

pub fn allowing_empty(self) -> Self

Declare that a zero-row read is legitimate for this source. Say it on purpose; the default refuses. See EmptyRead.

pub fn producer(&self) -> &str

The producer name this source reports in its errors.

pub fn empty_policy(&self) -> EmptyRead

What a zero-row read means here.

Trait Implementations

impl TableSource for ArrowSource

fn cdc_capability(&self, _table: &str) -> Option<CdcCapability>

No CDC story at all — the same honest None the file source gives.

fn fetch_new<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, spec: &'life1 TableSpec, state: &'life2 TableState, ) -> Pin<Box<dyn Future<Output = Result<Option<(Vec<RecordBatch>, TableState)>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Return rows new since state and the state to persist once they are sunk, or None when nothing changed.