bigquery 0.10.0

Library provides a simple API for Google BigQuery based on the gRPC Storage Read/Write and v2 APIs
Documentation

Cargo tests and formatting security audit

BigQuery for Rust

Library provides a simple API for Google BigQuery using gRPC for every call:

  • Fluent high-level and strongly typed API;
  • Read and write rows as Rust structures using Serde, with own codecs for every BigQuery type, including NUMERIC/BIGNUMERIC, INTERVAL, RANGE, nested STRUCT and ARRAY, and JSON columns mapped by the field type (String, serde_json::Value or your own structure);
  • Support for:
    • Table reads through the Storage Read API, in parallel streams, with column projection and typed filters;
    • Queries with named and positional parameters, including ARRAY and STRUCT parameters, always bound on the server side;
    • Short query mode, so BigQuery can answer short queries without creating a job;
    • Large query results read through the Storage Read API automatically;
    • Query results written into a destination table of your own and read back the same way;
    • Raw Arrow record batches for table reads and query results;
    • DML statements with affected row counts, and dry runs;
    • Writes through the Storage Write API with built-in batching and backpressure, of your structures or of raw Arrow record batches: at least once, exactly once, atomic (all rows or none), or buffered (rows readable once you flush them);
    • Updates and deletes by the primary key through change data capture (CDC), with fluent update() and delete() or a lower-level CDC writer;
    • Declarative table schemas, inferred from your structures or declared by hand, planned and synced with one call: new columns, renames, drops, widening, partitioning, clustering, primary key, recreating an empty table;
    • Datasets, tables and jobs management;
    • Bytes processed and billed, slot milliseconds and cache hits for every query, as span fields and as results;
  • Retries with jitter for the errors where retrying makes sense, including BigQuery rate limits;
  • Full async based on Tokio runtime;
  • Macros that help you use your structure fields as column paths;
  • Google client based on gcloud-sdk library that automatically detects GCE environment or application default accounts for local development;

Documentation

Please follow to the official website: https://bigquery-rust.abdolence.dev.

Quick start

Cargo.toml:

[dependencies]
bigquery = "0.10"
use bigquery::*;
use serde::{Deserialize, Serialize};

#[derive(Debug, Clone, Deserialize, Serialize)]
struct Order {
    id: i64,
    customer: String,
    total: f64,
    placed_at: jiff::Timestamp,
}

#[derive(Debug, Deserialize)]
struct CustomerTotal {
    customer: String,
    total: f64,
}

const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
    let db = BigQueryDb::new("my-gcp-project-id").await?;

    // Create a dataset
    db.fluent()
        .schema()
        .dataset(SHOP)
        .create()
        .location(BigQueryLocation::from_static("EU"))
        .execute()
        .await?;

    // Create the table, or bring an existing one to this schema, with the columns of `Order`
    let report = db
        .fluent()
        .schema()
        .table(SHOP.table(ORDERS))
        .columns(|columns| columns.from_type::<Order>())
        .primary_key([path!(Order::id)])
        .partition_by_day(path!(Order::placed_at))
        .sync()
        .await?;
    println!("{report}");

    // Insert through the Storage Write API
    let orders = vec![
        Order {
            id: 1,
            customer: "Alice".to_string(),
            total: 120.0,
            placed_at: jiff::Timestamp::now(),
        },
        Order {
            id: 2,
            customer: "Bob".to_string(),
            total: 15.5,
            placed_at: jiff::Timestamp::now(),
        },
    ];
    let summary = db
        .fluent()
        .insert()
        .into(SHOP.table(ORDERS))
        .objects(&orders)
        .execute()
        .await?;
    println!("Written {} rows", summary.rows_written);

    // Read the table through the Storage Read API, with a typed filter
    let large_orders: Vec<Order> = db
        .fluent()
        .select()
        .from(SHOP.table(ORDERS))
        .filter(|filter| {
            filter.for_all([
                filter.field(path!(Order::customer)).eq("Alice"),
                filter.field(path!(Order::total)).gt(100.0),
            ])
        })
        .obj()
        .query()
        .await?;

    // Query with parameters, and what the query cost. A longer statement can live in its own
    // .sql file instead, with `.query(sql_file!("sql/totals.sql", customer))`
    let (totals, stats): (Vec<CustomerTotal>, BigQueryJobStats) = db
        .fluent()
        .query(
            "SELECT customer, SUM(total) AS total FROM shop.orders \
             WHERE customer = @customer GROUP BY customer",
        )
        .param("customer", "Alice")
        .obj()
        .query_with_stats()
        .await?;

    println!(
        "{large_orders:?} {totals:?}, billed {:?} bytes",
        stats.total_bytes_billed
    );

    Ok(())
}

If you see no process-level CryptoProvider available, see Crypto provider error.

Migrating from 0.9

0.10 moves to jiff 1.0 and gcloud-sdk 0.34. bigquery::jiff and BigQueryInstant are jiff 1.0 types, so an application that names them through its own jiff dependency needs jiff = "1". Two copies of jiff are not interchangeable, and a Timestamp of jiff 0.2 does not convert into one of jiff 1.0.

jiff 1.0 renames and changes a few items:

jiff 0.2 jiff 1.0
Zoned ZonedDateTime
Timestamp::to_zoned Timestamp::to_zoned_datetime
Timestamp - Timestamp returns a Span returns a SignedDuration
SystemTime::from(timestamp) SystemTime::try_from(timestamp)

The rest of the jiff changes are listed in its changelog. The BigQuery wire encodings of DATE, TIME, DATETIME and TIMESTAMP values are unchanged.

google-cloud-auth 1.19 depends on jiff 0.2, so a build compiles both versions. No type of bigquery or bigquery::gcloud_sdk exposes the one of jiff 0.2.

Migrating from 0.8

0.9 moves to gcloud-sdk 0.33, which authenticates with google-cloud-auth. The token sources of gcloud-sdk are gone, and with_options_token_source is replaced by with_options_auth, which takes anything that converts into bigquery::gcloud_sdk::GoogleAuthHeaders:

0.8 0.9
with_options_token_source(options, scopes, TokenSourceType::Default) with_options_auth(options, GoogleAuthHeaders::from_adc_with_scopes(scopes).await?)
with_options_token_source(options, scopes, TokenSourceType::File(path)) with_options_service_account_key_file(options, path) for the default scopes; with_options_auth(options, GoogleAuthHeaders::from_service_account_key(key, scopes).await?) for other scopes
TokenSourceType::Json(json) with_options_auth(options, GoogleAuthHeaders::from_service_account_key(key, scopes).await?) for a service account key; the user_account, impersonated and external_account builders for other credentials
TokenSourceType::MetadataServer with_options_auth(options, mds::Builder::default().build()?)
TokenSourceType::ExternalSource(source) implement CredentialsProvider and pass Credentials::from(provider)

The library re-exports gcloud-sdk as bigquery::gcloud_sdk, so there is no need to add it to your dependencies. The builders, CredentialsProvider and Credentials are in bigquery::gcloud_sdk::google_cloud_auth::credentials. BigQueryDb::new, with_options, for_default_project_id and with_options_service_account_key_file keep their signatures.

Also changed:

  • MSRV is 1.91, the MSRV of google-cloud-auth;
  • the new default feature auth-default-crypto gives google-cloud-auth the aws-lc-rs rustls provider it signs service account keys with. With default-features = false, enable it or install a rustls provider yourself, see Crypto provider;
  • service account keys sign a JWT that is sent as the token itself, without the exchange at token_uri;
  • for_default_project_id also reads GOOGLE_CLOUD_PROJECT, and the project_id of the application default credentials file before its quota project.

Examples

All examples available in the examples directory.

To run an example with environment variables:

PROJECT_ID=<your-google-project-id> cargo run --example insert

Why this crate

Google has its own Rust crate for BigQuery, google-cloud-bigquery (0.18.0 at the time of writing, October 2026). Both crates work with the same APIs in different ways, so below is what each of them provides that the other does not.

What this crate provides

  • High-level typed API. Fluent builders for table reads, queries, writes, updates and deletes, and table schemas. Rows are your own structures in both directions, through Serde, for table reads, query results, writes and CDC. Dataset and table IDs are checked types, column paths come from your structure fields with path!/paths!, and table schemas are declared once and planned or synced with .plan()/.sync().
  • gRPC throughout. Queries, jobs, datasets and tables go through the BigQuery v2 API over gRPC, as well as Storage Read and Write. The official crate sends queries over REST and reads their rows as JSON pages.
  • Performance. Measured on 2026-10-05 against the official crate and Python on one home connection in Sweden to europe-north2, full details in the benchmarks:
    • small queries take 0.117 s for SELECT 1, about the same as the official crate, which also uses BigQuery's short query mode, and faster than Python's 0.270 s, since Python creates a job;
    • a 200,000-row query result as typed rows takes 1.16 s against 4.39 s, because the library reads a large result through Storage Read and the official crate pages it as JSON;
    • a 1M-row table scan as typed rows takes 9.6 s. The official crate has no typed Storage Read, so its typed path is a SELECT * query, which took 95.2 s and billed the whole table every run;
    • a 1M-row Storage Write from structures takes 34.0 s against 35.4 s, with 18% fewer bytes sent as protobuf than the official crate's Arrow.
  • Observability. Every query, read and write span carries what BigQuery reports: bytes processed and billed, slot milliseconds, cache hits, rows and bytes read, rows appended and bytes sent, retries. query_with_stats() returns a query's figures together with its rows.
  • Safety. Query parameter values are always bound on the server side and never written into the SQL text. The typed filter and the generated DDL write values as escaped literals and names as quoted identifiers, tested against a corpus of hostile values and on BigQuery itself. The calls that lose data say so in their names: dangerously_delete_with_contents(), dangerously_recreate_with_data_loss(). No unsafe code.
  • Testing. A fake BigQuery for your tests, in bigquery::testing. The code under test runs its real client against a loopback gRPC server, which answers queries, serves table rows and fails calls as the test scripts it with your own structures. No credentials or network.

What the official crate provides and this one does not

Checked against google-cloud-bigquery 0.18.0 and google-cloud-bigquery-v2 1.0.0:

  • It is maintained by Google as part of google-cloud-rust, and its v2 API crate is already 1.0;
  • It uses the REST API for v2, which is GA. The v2 API over gRPC this crate uses works for every call the library makes, but Google does not document it and it is pre-GA, so be aware it can change without notice;
  • It has stub traits to mock its clients in your tests. This crate has no mocks of its clients and provides a fake BigQuery server instead, see Testing with the fake.

Google authentication

Looks for credentials in the following places, preferring the first location found:

  • A JSON file whose path is specified by the GOOGLE_APPLICATION_CREDENTIALS environment variable.
  • A JSON file in a location known to the gcloud command-line tool using gcloud auth application-default login.
  • On Google Compute Engine, it fetches credentials from the metadata server.

For local development don't confuse gcloud auth login with gcloud auth application-default login, since the first authorize only gcloud tool to access the Cloud Platform.

A service account key file goes to BigQueryDb::with_options_service_account_key_file. Other credentials go to BigQueryDb::with_options_auth, see Google authentication.

Service account keys sign a JWT, which needs a rustls crypto provider:

  • auth-default-crypto: default feature, the aws-lc-rs provider of google-cloud-auth, built from C sources;
  • with default-features = false and without auth-default-crypto, install a rustls provider before creating a client, see Crypto provider.

How this library is tested

There are unit tests next to the code, many of them against a fake BigQuery gRPC server, and the code in the book is compiled as doctests. The integration tests in the tests directory run for every push to master against a real BigQuery project allocated for testing purposes, or locally when GCP_PROJECT is set. Other branches run only the unit tests and doctests. All of them work in one dataset, bigquery_rs_ci, under an account whose BigQuery access is that dataset and nothing else: it can run jobs and read sessions, but it cannot create a dataset or read or write data outside this one. Each test makes its own tables there, named after its run, and drops them at the end; the dataset expires anything left behind after an hour. Be aware not to introduce huge reads, writes or queries there, since they are billed.

Licence

Apache Software License (ASL)

Author

Abdulla Abdurakhmanov