Struct BigQueryLoader 

pub struct BigQueryLoader {
    pub project: String,
    pub dataset: String,
    pub partition_by: Option<String>,
    pub cluster_by: Vec<String>,
    pub run_id: Option<String>,
    pub max_partitions_per_job: usize,
}

Loads Rivet Parquet into a BigQuery dataset via the bq CLI.

Fields

project: String

dataset: String

partition_by: Option<String>

Partition expression for table creation, e.g. DATE(created_at) or a DATE/TIMESTAMP column. Applied only when the table is created.

cluster_by: Vec<String>

Up to 4 clustering columns. Applied only when the table is created.

run_id: Option<String>

Load-run correlation id, emitted as the automatic rivet_run:<id> job label so every job of one rivet load invocation shares a run key — cost slices per run (across tables) as well as per table. None omits the label entirely.

max_partitions_per_job: usize

Max distinct partitions a single load job may create — BigQuery’s hard limit is 4,000. When a daily-partitioned, Hive-prefixed input (<col>=YYYY-MM-DD/…, as rivet’s partition_by writes) spans more than this, the free load is split into several LOAD DATA jobs, each under the cap.

Implementations

impl BigQueryLoader

pub fn new(project: impl Into<String>, dataset: impl Into<String>) -> Self

pub fn partition_by(self, expr: impl Into<String>) -> Self

pub fn run_id(self, id: impl Into<String>) -> Self

Set the load-run correlation id, emitted as the rivet_run job label.

pub fn cluster_by(self, columns: Vec<String>) -> Self

Trait Implementations

impl Clone for BigQueryLoader

fn clone(&self) -> BigQueryLoader

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for BigQueryLoader

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl TargetLoader for BigQueryLoader

fn fqtn(&self, table: &str) -> String

Fully-qualify table for this warehouse (project.dataset.t / db.schema.t).

fn materialize( &self, table: &str, specs: &[TargetColumnSpec], uris: &[String], ) -> Result<u64>

Overwrite table with the Parquet at uris, materializing the native column types in specs. Returns the rows the load landed.

fn append_changelog( &self, table: &str, specs: &[TargetColumnSpec], uris: &[String], pk: &[String], ) -> Result<u64>

Append the CDC change Parquet into <table>__changes (created if absent), prepending the __op / __pos / __seq meta columns to specs. Returns the rows this call appended.

fn warehouse(&self) -> Warehouse

The warehouse this adapter targets — lets the shared driver build the current-state view SQL (dialect keyword + identifier quoting) in ONE place per mode instead of once per adapter.

fn create_view(&self, table: &str, view_sql: &str) -> Result<()>

CREATE OR REPLACE the current-state view <table> from pre-built view_sql (the driver builds it via cdc::dedup_view_sql for CDC or cdc::inc_dedup_view_sql for incremental). The adapter only executes it its way (e.g. Snowflake prefixes a QUERY_TAG).