Struct FileSource 

pub struct FileSource { /* private fields */ }

A TableSource over a watched directory tree of GraphAr/columnar file drops — the local half of the Airflow AIP-82 story.

Each mapped table maps to a subdirectory root/<table>/; every file in it with a recognised extension (parquet, orc, csv, json, arrow) is a chunk. The set of already-ingested file names lives in the table’s TableState::done_files ledger, so a new file drop triggers exactly one sync round and nothing is re-read.

use knut_bifrost::source::FileSource;
let source = FileSource::new("/data/drops");

Implementations

impl FileSource

pub fn new(root: impl Into<PathBuf>) -> Self

Watch root; table lake.foo.bar reads from root/lake.foo.bar/.

Trait Implementations

impl TableSource for FileSource

fn fetch_new<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, spec: &'life1 TableSpec, state: &'life2 TableState, ) -> Pin<Box<dyn Future<Output = Result<Option<(Vec<RecordBatch>, TableState)>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Return rows new since state and the state to persist once they are sunk, or None when nothing changed.

fn cdc_capability(&self, table: &str) -> Option<CdcCapability>

Which CDC backend this source resolved to for table, and why.