Struct SnowflakeSource 

pub struct SnowflakeSource { /* private fields */ }

Available on crate features source-rest and source-snowflake only.

A source that streams the rows of a Snowflake SQL statement via the SQL REST API.

Implementations

impl SnowflakeSource

pub fn new( config: SnowflakeSourceConfig, ) -> Result<SnowflakeSource, FaucetError>

Create a new Snowflake source. Initialises the underlying HTTP client and does no I/O; it fails only on an invalid config (an out-of-range batch_size).

pub fn with_auth_provider( self, provider: Arc<dyn AuthProvider>, ) -> SnowflakeSource

Attach a shared AuthProvider. When set, the provider supplies the credential for every request (taking precedence over inline auth), so several sources can share one OAuth token with single-flight refresh. Used by the CLI to resolve auth: { ref }, and by library callers who inject a provider directly.

The provider must yield a Bearer or Token credential, which maps onto SnowflakeAuth::OAuth. Key-pair JWT cannot be supplied via a provider (JWT is minted locally from the RSA key).

pub fn with_endpoint_base(self, base: impl Into<String>) -> SnowflakeSource

Override the base URL used to reach the SQL REST API.

Pass http://127.0.0.1:1234 to point the source at a mock server — the source will issue requests to {base}/api/v2/statements and {base}/api/v2/statements/{handle}?partition=N. Useful for tests and proxy/private-link setups.

Trait Implementations

impl Source for SnowflakeSource

fn discover<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Result<Vec<DatasetDescriptor>, FaucetError>> + Send + 'async_trait>>

where 'life0: 'async_trait, SnowflakeSource: 'async_trait,

Enumerate every base table in the configured database (from information_schema.columns / information_schema.tables, excluding INFORMATION_SCHEMA itself), with column types mapped to JSON-Schema fragments and a row estimate from information_schema.tables.row_count. Runs one catalog statement through the source’s ordinary SQL REST API path (async-202 poll and partition paging included) — catalog metadata only, no data scan.

fn check<'life0, 'life1, 'async_trait>( &'life0 self, ctx: &'life1 CheckContext, ) -> Pin<Box<dyn Future<Output = Result<CheckReport, FaucetError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, SnowflakeSource: 'async_trait,

Preflight probe for faucet doctor. Overrides the default (which pulls a page via stream_pages and would execute the configured query). Instead submits a cheap SELECT 1 so doctor validates auth, warehouse, and connectivity without running the user’s real statement.

fn stream_pages<'a>( &'a self, context: &'a HashMap<String, Value>, _batch_size: usize, ) -> Pin<Box<dyn Stream<Item = Result<StreamPage, FaucetError>> + Send + 'a>>

Stream rows partition-by-partition without buffering the full result set. Rows are accumulated into a per-page buffer of SnowflakeSourceConfig::batch_size entries and yielded as soon as the buffer is full.

The trait-level batch_size argument is ignored in favour of the config field — the config is the user-facing knob the README documents, and routing the pipeline-supplied hint through it would silently override an explicit config value.

batch_size = 0 drains every partition and emits the full result set as a single page. The source has no incremental-replication mode today, so every emitted page carries bookmark: None.

fn connector_name(&self) -> &'static str

Stable identifier used as the connector label on metrics and the connector attribute on spans. Defaults to the final segment of std::any::type_name::<Self>(), e.g. "RestSource". Built-in connectors override with a short, friendly snake_case name (e.g. "rest"). Must return a non-empty string; observability decorators fall back to "unknown" in release builds if it is empty (and debug_assert! in debug builds).

fn config_schema(&self) -> Value

Return a JSON Schema describing the configuration this source accepts.

fn dataset_uri(&self) -> String

Logical dataset identity for lineage emission, following OpenLineage naming conventions (https://openlineage.io/docs/spec/naming).

fn supports_discover(&self) -> bool

Whether this source can enumerate the datasets behind its connection via discover. Default: false. Sources backed by an introspectable catalog (database information_schema, MongoDB collections, Elasticsearch indices, object-store prefixes) override this to true.

fn fetch_with_context<'life0, 'life1, 'async_trait>( &'life0 self, context: &'life1 HashMap<String, Value>, ) -> Pin<Box<dyn Future<Output = Result<Vec<Value>, FaucetError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, SnowflakeSource: 'async_trait,

Primary fetch method. Receives context from a parent source’s records.

fn fetch_all<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Result<Vec<Value>, FaucetError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Self: 'async_trait,

Convenience: fetch with no parent context.

fn fetch_with_context_incremental<'life0, 'life1, 'async_trait>( &'life0 self, context: &'life1 HashMap<String, Value>, ) -> Pin<Box<dyn Future<Output = Result<(Vec<Value>, Option<Value>), FaucetError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Incremental fetch with parent context support.

fn fetch_all_incremental<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Result<(Vec<Value>, Option<Value>), FaucetError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Self: 'async_trait,

Convenience: incremental fetch with no parent context.

fn state_key(&self) -> Option<String>

Stable key under which this source’s incremental-replication bookmark should be persisted in a StateStore.

fn apply_start_bookmark<'life0, 'async_trait>( &'life0 self, _bookmark: Value, ) -> Pin<Box<dyn Future<Output = Result<(), FaucetError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Self: 'async_trait,

Apply a bookmark loaded from a StateStore as this run’s starting point.

fn capture_resume_position<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Result<Option<Value>, FaucetError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Self: 'async_trait,

Capture the source’s current replication position without consuming any changes, ensuring any server-side resource (e.g. a logical replication slot) needed to later resume from that position exists.

fn supports_exactly_once(&self) -> bool

Whether this source deterministically replays the same page sequence from a given bookmark — the requirement for the atomic-watermark effectively-once path (a non-deterministic replay could cause the pipeline to skip a page whose contents differ from the one already committed). Default: false.

fn replay_guarantee(&self) -> ReplayGuarantee

The typed replay capability this source advertises — see ReplayGuarantee.

fn is_shardable(&self) -> bool

Whether this source can split its work into independent shards for clustered (Mode B) execution. Default: false (single whole-dataset shard). Sources with a natural partition (object-store prefixes, table primary-key ranges) override this to true and implement enumerate_shards + apply_shard.

fn enumerate_shards<'life0, 'async_trait>( &'life0 self, _target: usize, ) -> Pin<Box<dyn Future<Output = Result<Vec<ShardSpec>, FaucetError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Self: 'async_trait,

Enumerate the shards this source splits into, aiming for roughly target of them (a hint — the source may return fewer, e.g. when the dataset is small, or one per natural partition regardless of target).

fn apply_shard<'life0, 'life1, 'async_trait>( &'life0 self, _shard: &'life1 ShardSpec, ) -> Pin<Box<dyn Future<Output = Result<(), FaucetError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Narrow this source instance to a single shard before streaming.