Struct BigQuerySink 

pub struct BigQuerySink { /* private fields */ }

Available on crate feature sink-bigquery only.

A sink that writes JSON records to a BigQuery table using the streaming insert API (tabledata.insertAll).

Implementations

impl BigQuerySink

pub async fn new( config: BigQuerySinkConfig, ) -> Result<BigQuerySink, FaucetError>

Create a new BigQuery sink from the given configuration.

This initialises the BigQuery client and authenticates with GCP. Returns a FaucetError::Auth if authentication fails.

Trait Implementations

impl Sink for BigQuerySink

fn check<'life0, 'life1, 'async_trait>( &'life0 self, ctx: &'life1 CheckContext, ) -> Pin<Box<dyn Future<Output = Result<CheckReport, FaucetError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, BigQuerySink: 'async_trait,

Preflight check (faucet doctor).

Runs a single read-only tables.get against the configured project_id.dataset_id.table_id using the already-authenticated client built in BigQuerySink::new. This mints/uses the access token and confirms the credentials can read the target table’s metadata — without inserting any rows. Auth, missing-dataset, missing-table, and permission errors all surface as a Fail probe with a remediation hint. The access token is never included in the reason or hint.

fn write_batch<'life0, 'life1, 'async_trait>( &'life0 self, records: &'life1 [Value], ) -> Pin<Box<dyn Future<Output = Result<usize, FaucetError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, BigQuerySink: 'async_trait,

Write records to BigQuery.

When config.batch_size > 0 and the input slice is larger than batch_size, the slice is split into chunks of batch_size rows and each chunk is sent as a separate tabledata.insertAll call. When config.batch_size == 0, the entire slice is sent in a single insertAll request — useful when upstream StreamPages are already sized for BigQuery’s per-request limits.

fn write_batch_partial<'life0, 'life1, 'async_trait>( &'life0 self, records: &'life1 [Value], ) -> Pin<Box<dyn Future<Output = Result<Vec<Result<(), FaucetError>>, FaucetError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, BigQuerySink: 'async_trait,

Write records to BigQuery, returning a per-row outcome vector.

Unlike write_batch, which collapses all insertErrors into a single FaucetError, this method maps each row to Ok(()) if BigQuery accepted it or Err(FaucetError::Sink(...)) if BigQuery reported a per-row error for it. This allows the pipeline’s DLQ router to quarantine only the rows that BigQuery actually rejected while keeping already-committed siblings out of the dead-letter queue.

Transport-level or HTTP-level failures (e.g. network errors, 4xx/5xx responses) are still returned as an outer Err because no rows can be considered committed in that case.

Chunking follows the same batch_size semantics as write_batch: batch_size == 0 sends the entire slice in one call; batch_size > 0 splits the slice into chunks and concatenates the per-row outcomes in input order.

fn connector_name(&self) -> &'static str

Stable identifier used as the connector label on metrics and the connector attribute on spans. See Source::connector_name.

fn config_schema(&self) -> Value

Return a JSON Schema describing the configuration this sink accepts.

fn flush<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Result<(), FaucetError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Self: 'async_trait,

Flush any buffered data to the destination.