Trait OriginConnector 

pub trait OriginConnector: Send + Sync {
    // Required methods
    fn name(&self) -> &str;
    fn fetch_all<'life0, 'life1, 'life2, 'async_trait>(
        &'life0 self,
        sql: &'life1 str,
        key_column: &'life2 str,
    ) -> Pin<Box<dyn Future<Output = Result<Vec<RawRow>, OversyncError>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait,
             'life1: 'async_trait,
             'life2: 'async_trait;
    fn test_connection<'life0, 'async_trait>(
        &'life0 self,
    ) -> Pin<Box<dyn Future<Output = Result<(), OversyncError>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait;

    // Provided method
    fn fetch_into<'life0, 'life1, 'life2, 'async_trait>(
        &'life0 self,
        sql: &'life1 str,
        key_column: &'life2 str,
        batch_size: usize,
        tx: Sender<Vec<RawRow>>,
    ) -> Pin<Box<dyn Future<Output = Result<usize, OversyncError>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait,
             'life1: 'async_trait,
             'life2: 'async_trait { ... }
}

An origin connector fetches rows from an external data source.

Implementations exist for PostgreSQL, MySQL, Trino, ClickHouse, HTTP APIs, GraphQL, and Apache Arrow Flight SQL. Each connector is created via an OriginFactory from a JSON config object.

Lifecycle

1. Factory creates the connector (connection pool established)
2. [fetch_all] or [fetch_into] called once per cycle
3. Connector is reused across cycles (connection pooling)

Required Methods

fn name(&self) -> &str

Human-readable name of this connector instance.

fn fetch_all<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, sql: &'life1 str, key_column: &'life2 str, ) -> Pin<Box<dyn Future<Output = Result<Vec<RawRow>, OversyncError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Fetch all rows matching the query. Returns the full result set in memory.

fn test_connection<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Result<(), OversyncError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Provided Methods

fn fetch_into<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, sql: &'life1 str, key_column: &'life2 str, batch_size: usize, tx: Sender<Vec<RawRow>>, ) -> Pin<Box<dyn Future<Output = Result<usize, OversyncError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Stream rows in batches into a channel. Memory bounded by batch_size * channel buffer. Default: calls fetch_all, chunks, sends.

Implementors