Skip to main content

Crate bigquery

Crate bigquery 

Source
Expand description

§BigQuery for Rust

A client for Google BigQuery that uses gRPC throughout: the Storage Read API for table scans and large results, the Storage Write API for inserts, and the v2 API for queries, jobs, datasets and tables.

Full documentation, with a chapter per topic: https://bigquery-rust.abdolence.dev

§Example

use bigquery::*;
use serde::{Deserialize, Serialize};

#[derive(Debug, Clone, Deserialize, Serialize)]
struct Order {
    id: i64,
    customer: String,
    total: f64,
}

const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");

let db = BigQueryDb::new("my-project-id").await?;

// Insert through the Storage Write API
let orders = vec![Order {
    id: 1,
    customer: "Alice".to_string(),
    total: 120.0,
}];
db.fluent()
    .insert()
    .into(SHOP.table(ORDERS))
    .objects(&orders)
    .execute()
    .await?;

// Read the table back through the Storage Read API, with a typed filter
let large_orders: Vec<Order> = db
    .fluent()
    .select()
    .from(SHOP.table(ORDERS))
    .filter(|filter| filter.field(path!(Order::total)).gt(100.0))
    .obj()
    .query()
    .await?;

// Query with parameters
let alice_orders: Vec<Order> = db
    .fluent()
    .query("SELECT id, customer, total FROM shop.orders WHERE customer = @customer")
    .param("customer", "Alice")
    .obj()
    .query()
    .await?;

Re-exports§

pub extern crate struct_path;

Re-exports§

pub use gcloud_sdk;
pub use arrow_array;
pub use arrow_schema;
pub use jiff;
pub use url;

Modules§

errors
The error types of this crate, built around BigQueryError. The crate’s error type and the pieces it is built from.
serialize_as_date
#[serde(with = "bigquery::serialize_as_date")] for a jiff::civil::Date field, the same as BigQueryDate.
serialize_as_datetime
#[serde(with = "bigquery::serialize_as_datetime")] for a jiff::civil::DateTime field, the same as BigQueryDateTime.
serialize_as_decimal
#[serde(with = "bigquery::serialize_as_decimal")] for a bare decimal field, the same as BigQueryDecimal.
serialize_as_json
#[serde(with = "bigquery::serialize_as_json")] for a bare field of a JSON column, the same as BigQueryJson.
serialize_as_optional_date
#[serde(with = "bigquery::serialize_as_optional_date")] for an Option<jiff::civil::Date> field, the same as Option<BigQueryDate>.
serialize_as_optional_datetime
#[serde(with = "bigquery::serialize_as_optional_datetime")] for an Option<jiff::civil::DateTime> field, the same as Option<BigQueryDateTime>.
serialize_as_optional_decimal
#[serde(with = "bigquery::serialize_as_optional_decimal")] for an Option decimal field, the same as Option<BigQueryDecimal<T>>.
serialize_as_optional_json
#[serde(with = "bigquery::serialize_as_optional_json")] for an Option field of a JSON column, the same as Option<BigQueryJson<T>>.
serialize_as_optional_time
#[serde(with = "bigquery::serialize_as_optional_time")] for an Option<jiff::civil::Time> field, the same as Option<BigQueryTime>.
serialize_as_optional_timestamp
#[serde(with = "bigquery::serialize_as_optional_timestamp")] for an Option<jiff::Timestamp> field, the same as Option<BigQueryTimestamp>.
serialize_as_time
#[serde(with = "bigquery::serialize_as_time")] for a jiff::civil::Time field, the same as BigQueryTime.
serialize_as_timestamp
#[serde(with = "bigquery::serialize_as_timestamp")] for a jiff::Timestamp field, the same as BigQueryTimestamp.
testing
A fake BigQuery for your tests.

Macros§

path
Builds the BigQuery column path for one field of Struct as a String.
path_camel_case
Builds the BigQuery column path for one field of Struct, converting it to camelCase.
paths
Builds the BigQuery column paths for several fields of Struct as a Vec<String>.
paths_camel_case
Builds the BigQuery column paths for several fields of Struct, converting each to camelCase.
sql_file
Embeds a GoogleSQL file as a BigQuerySql and checks, as the calling crate compiles, that the names listed after the path are exactly the file’s @name parameters.

Structs§

BigQueryBatchRows
The rows of one Arrow RecordBatch decoded into T, one item per row, in order.
BigQueryCdcWriter
A writer of CDC changes: each row carries _CHANGE_TYPE and, optionally, _CHANGE_SEQUENCE_NUMBER, and BigQuery applies it to the row with the same primary key.
BigQueryChange
One CDC change of a row.
BigQueryChangeSequenceNumber
A CDC _CHANGE_SEQUENCE_NUMBER, which orders changes to one key. From<u64> writes it in hex; any other text is sent as it is, and BigQuery checks its form.
BigQueryDataset
A dataset as GetDataset returns it.
BigQueryDatasetBuilder
One dataset, from BigQuerySchemaBuilder::dataset. End it with one of its calls.
BigQueryDatasetCreateBuilder
A new dataset’s settings, from BigQueryDatasetBuilder::create.
BigQueryDatasetId
A BigQuery dataset ID, checked only for what would let it reshape a path or a query.
BigQueryDatasetListBuilder
A listing of the datasets in a project, from BigQuerySchemaBuilder::datasets.
BigQueryDatasetRef
A dataset, by project and dataset ID.
BigQueryDatasetSummary
A dataset as ListDatasets returns it, which is less than BigQueryDataset: read one with BigQueryDatasetBuilder::get for the rest.
BigQueryDatasetUpdateBuilder
Changes to a dataset, from BigQueryDatasetBuilder::update. What it does not change stays as the dataset has it.
BigQueryDate
A DATE column with the codecs’ integer fast path; any other format sees it as a plain jiff::civil::Date, or i32 days since 1970-01-01 in formats that are not human-readable.
BigQueryDateTime
A DATETIME column with the codecs’ integer fast path; any other format sees it as a plain jiff::civil::DateTime, or i64 civil microseconds since 1970-01-01T00:00:00 in formats that are not human-readable.
BigQueryDb
The main entry point for working with Google BigQuery.
BigQueryDbOptions
Configuration options for the BigQueryDb client.
BigQueryDbOptionsInit
Required fields of BigQueryDbOptions; convert with BigQueryDbOptions::from or .into().
BigQueryDecimal
A NUMERIC or BIGNUMERIC value held in any decimal type that round-trips through its Display and FromStr text, such as bigdecimal::BigDecimal or rust_decimal::Decimal.
BigQueryDecimalParams
The declared precision and scale of a NUMERIC or BIGNUMERIC column. A declared precision without a scale has scale 0, so NUMERIC(10) equals NUMERIC(10, 0).
BigQueryDeleteInitialBuilder
The first stage of a delete, from BigQueryExprBuilder::delete.
BigQueryDeleteKeysBuilder
A delete of rows by primary key values, written as CDC deletes of rows that hold only the key columns.
BigQueryDeleteObjBuilder
A delete of rows by primary key, written as CDC deletes through the table’s default stream.
BigQueryDeleteTableBuilder
A delete with its table; continue with the keys of the rows to delete, as plain values or as rows.
BigQueryDmlStats
The rows a DML statement changed, by kind.
BigQueryDryRunResult
What a dry run reports without running the statement.
BigQueryExprBuilder
The entry point for fluent BigQuery operations, obtained from BigQueryDb::fluent().
BigQueryFieldSchema
One column, or one field of a STRUCT.
BigQueryFilter
A row filter from BigQueryFilterBuilder. It holds the first error any of its parts met, which the read’s terminal returns.
BigQueryFilterBuilder
Builds row filters: the closure given to filter receives one.
BigQueryFilterFieldExpr
A column targeted by BigQueryFilterBuilder::field: pick the condition.
BigQueryFinalizedStream
A finalized pending stream, ready for a commit together with other streams.
BigQueryInsertChangesBuilder
An insert of CDC changes through the default stream.
BigQueryInsertInitialBuilder
The first stage of an insert, from BigQueryExprBuilder::insert.
BigQueryInsertObjBuilder
An insert of rows. The default writes through the table’s default stream, at least once.
BigQueryInsertParams
What an insert or a streaming writer writes to and how.
BigQueryInsertParamsInit
Required fields of BigQueryInsertParams; convert with BigQueryInsertParams::from or .into().
BigQueryInsertRecordBatchesBuilder
An insert of Arrow record batches. The default writes through the table’s default stream, at least once.
BigQueryInsertTableBuilder
An insert with its table; continue with the rows or the changes to write.
BigQueryInterval
An INTERVAL value as its three independent parts, each with its own sign, the way BigQuery and Arrow’s MonthDayNano hold it.
BigQueryJob
A job as GetJob or ListJobs returns it.
BigQueryJobId
A BigQuery job ID.
BigQueryJobRef
A BigQuery job.
BigQueryJobStats
What a query used, from the responses the query already received; the _with_stats terminals return it with the rows.
BigQueryJson
A JSON value carried as its JSON text, whatever the serde format.
BigQueryLabels
The labels of a dataset, a table or a job: keys and values, ordered by key.
BigQueryLabelsIntoIter
The owning iterator of BigQueryLabels, ordered by key.
BigQueryListJobsParams
Which jobs BigQueryDb::stream_jobs lists, newest first.
BigQueryLocation
A BigQuery location, such as US, EU or europe-west1.
BigQueryParamType
The type of a query parameter, for a value whose type cannot be inferred.
BigQueryQueryBuilder
A GoogleSQL query being built, from BigQueryExprBuilder::query: add parameters and job settings, then pick a terminal.
BigQueryQueryDestination
The table a query writes its result into, and what happens to the rows the table already holds.
BigQueryQueryId
The ID BigQuery reports for a query it answered, with or without a job.
BigQueryQueryObjBuilder
A typed query: pick a terminal.
BigQueryQueryParams
What a query sends: the statement, its parameters and the job settings, as BigQueryQueryBuilder collects them.
BigQueryQueryParamsInit
Required fields of BigQueryQueryParams; convert with BigQueryQueryParams::from or .into().
BigQueryRange
A RANGE value of DATE, DATETIME or TIMESTAMP.
BigQueryReadOptions
How a Storage Read session is opened.
BigQueryReadParams
What a table read asks Storage Read for.
BigQueryReadParamsInit
Required fields of BigQueryReadParams; convert with BigQueryReadParams::from or .into().
BigQueryRecreate
A recreate .sync() runs because a change is impossible in place and the declaration opts in to recreating.
BigQueryRequestId
The idempotency key of a Query call: a retry that repeats it does not run the statement twice, within the window BigQuery keeps keys for.
BigQuerySchemaBuilder
The schema namespace, from BigQueryExprBuilder::schema.
BigQuerySchemaColumn
One declared column, or one field of a declared RECORD, as BigQuerySchemaColumnsBuilder::field starts it or from_type infers it.
BigQuerySchemaColumns
The columns a columns closure declares, in table order: the list fields collects, or the one from_type infers.
BigQuerySchemaColumnsBuilder
Builds the column list for columns and record.
BigQuerySelectBuilder
A table read with its table named: add a filter or session settings, then pick a terminal, obj for typed rows or record_batches for raw Arrow.
BigQuerySelectInitialBuilder
The first stage of a table read, from BigQueryExprBuilder::select: pick the columns, or name the table.
BigQuerySelectObjBuilder
A typed table read: pick a terminal.
BigQuerySql
The GoogleSQL statement that query runs: text built at run time, or a .sql file embedded and checked by sql_file!.
BigQueryStreamingWriteOptions
How a streaming writer opens its stream, batches rows and bounds what is in flight.
BigQueryStreamingWriter
A writer that streams rows of T into one table over one AppendRows connection.
BigQueryTable
A table as GetTable returns it.
BigQueryTableDeclaration
One table’s declared schema and settings, with the sync options, checked.
BigQueryTableId
A BigQuery table ID, checked only for what would let it reshape a path or a query.
BigQueryTableListBuilder
A listing of one dataset’s tables, from BigQueryDatasetBuilder::tables.
BigQueryTablePlan
What .sync() would do to one table, as .plan() returns it.
BigQueryTableRef
A table, by project, dataset and table ID.
BigQueryTableSchema
A table schema, normalised so that schemas from the v2 API, the Storage API and Arrow compare with ==.
BigQueryTableSchemaBuilder
One table’s declaration. End it with plan or sync, or with one of the plain calls get and delete, which act on the table and ignore whatever the chain declared.
BigQueryTableSummary
A table as ListTables returns it, without its schema and sizes: read one with BigQueryTableSchemaBuilder::get for those.
BigQueryTableSyncReport
What .sync() did to one table.
BigQueryTableTarget
A whole table as .sync() creates or recreates it: what is declared, plus whatever the table already has that the declaration leaves undeclared and is not pruned.
BigQueryTime
A TIME column with the codecs’ integer fast path; any other format sees it as a plain jiff::civil::Time, or i64 microseconds since midnight in formats that are not human-readable. Sub-microsecond digits are dropped on write.
BigQueryTimestamp
A TIMESTAMP column with the codecs’ integer fast path; any other format sees it as a plain jiff::Timestamp: RFC 3339 text in human-readable formats, microseconds since the epoch in the others.
BigQueryTraceId
The trace_id a writer sends with its stream, which BigQuery keeps for its own diagnostics; Google suggests the client’s name and version.
BigQueryUpdateInitialBuilder
The first stage of an update, from BigQueryExprBuilder::update.
BigQueryUpdateObjBuilder
An update of whole rows by primary key, written as CDC upserts through the table’s default stream.
BigQueryUpdateTableBuilder
An update with its table; continue with the rows to write.
BigQueryWithheldChange
A change .sync() leaves out until the declaration opts in to it.
BigQueryWriteResponse
The outcome of one acknowledged batch.
BigQueryWriteStreamName
The name of a write stream, as CreateWriteStream returned it: projects/{p}/datasets/{d}/tables/{t}/streams/{id}.
BigQueryWriteSummary
What a finished writer wrote.

Enums§

BigQueryChangeType
What a CDC change does to the row with its primary key.
BigQueryDestinationWrite
What a query does when its destination table already holds rows. Each one applies only if the job succeeds, as one atomic update of the table.
BigQueryDroppedData
Data a .sync() deleted.
BigQueryFieldMode
The mode of a column.
BigQueryFieldType
The type of a column, with the parameters that are part of it.
BigQueryJobCreation
Whether a query runs as a BigQuery job.
BigQueryJobState
Where a job is in its run, as BigQuery names it in JobStatus.state: PENDING, RUNNING or DONE.
BigQueryJobType
What kind of work a job does, as BigQuery names it in JobConfiguration.job_type: QUERY, LOAD, EXTRACT, COPY or UNKNOWN.
BigQueryMissingValue
What BigQuery stores for a field that a row leaves out.
BigQueryPartitionUnit
The unit of time partitioning.
BigQueryPartitioning
How a table is partitioned. Partition expiration is separate, see partition_expiration, because it can change in place while the partitioning cannot.
BigQueryRangeElementType
The element type of a RANGE column.
BigQueryReadCompression
The compression of the Arrow record batch buffers a read session sends.
BigQueryRecreateMethod
How a table is recreated.
BigQueryRecreatePolicy
What .sync() may do when a change is impossible in place.
BigQueryRefusal
Why .sync() refuses a plan, writing nothing.
BigQuerySchemaChange
One difference between a declaration and the table, and how it is applied.
BigQueryStatementType
The kind of statement a query ran, as BigQuery names it in statement_type.
BigQueryTableType
What kind of table a table is, as BigQuery names it in Table.type: TABLE, VIEW, MATERIALIZED_VIEW, EXTERNAL or SNAPSHOT.
BigQueryWithheldReason
What a withheld change waits for.
BigQueryWriteMode
The write stream a writer uses.

Constants§

BIGQUERY_API_URL
Google’s BigQuery v2 API endpoint: datasets, tables, jobs and queries.
BIGQUERY_STORAGE_API_URL
Google’s BigQuery Storage Read and Write API endpoint.

Traits§

BigQueryFilterExpr
What BigQueryFilterBuilder::for_all, for_any and not take: a filter, or an optional one, where None stands for no condition.

Type Aliases§

BigQueryInstant
An exact instant in time, used by this library for the times BigQuery reports about a dataset, a table, a job or a write: creation and last modified times, a job’s start and end, a table’s expiration, a commit time, and a read’s snapshot time.
BigQueryQueryOutcome
What a statement did, as execute reports it: the same figures a query’s BigQueryJobStats carries.
BigQueryResult
The result type of every fallible call in this crate.