bigquery 0.10.0

Library provides a simple API for Google BigQuery based on the gRPC Storage Read/Write and v2 APIs
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
# Testing with the fake

The library provides a fake BigQuery for your tests, in the `bigquery::testing` module.
`BigQueryFake` runs a gRPC server on a loopback port and hands out a real `BigQueryDb` connected
to it, so the code under test runs unchanged, through the same requests, codecs, retries and
errors as against BigQuery. It needs no credentials and no network.

A test scripts the fake with your own serde types:

- a query rule answers the statement it matches with rows, DML counts or a failure;
- a table holds rows, which reads serve and writes append to, and which the test reads back;
- a fault makes the calls of one RPC fail.

The tests need an async runtime in your dev-dependencies:

```toml
[dev-dependencies]
tokio = { version = "1", features = ["macros", "rt"] }
```

The examples below share a row type and the code under test, and each of them is the body of a
`#[tokio::test] async fn ...() -> BigQueryResult<()>`:

```rust
use bigquery::*;
use serde::{Deserialize, Serialize};

#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
struct Order {
    id: i64,
    customer: String,
    total: f64,
}

const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
const ORDERS_OF: &str = "SELECT id, customer, total FROM shop.orders WHERE customer = @customer";

async fn orders_of(db: &BigQueryDb, customer: &str) -> BigQueryResult<Vec<Order>> {
    db.fluent().query(ORDERS_OF).param("customer", customer).obj().query().await
}

async fn save(db: &BigQueryDb, orders: &[Order]) -> BigQueryResult<BigQueryWriteSummary> {
    db.fluent().insert().into(SHOP.table(ORDERS)).objects(orders).execute().await
}
```

## Queries

There is no SQL engine in the fake. A query answers only as a rule scripts it:

```rust
# use bigquery::*;
# use serde::{Deserialize, Serialize};
# #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
# struct Order { id: i64, customer: String, total: f64 }
# const ORDERS_OF: &str = "SELECT id, customer, total FROM shop.orders WHERE customer = @customer";
# async fn orders_of(db: &BigQueryDb, customer: &str) -> BigQueryResult<Vec<Order>> {
#     db.fluent().query(ORDERS_OF).param("customer", customer).obj().query().await
# }
# #[tokio::main(flavor = "current_thread")]
# async fn main() -> BigQueryResult<()> {
use bigquery::testing::BigQueryFake;

let fake = BigQueryFake::start().await?;
let alice = vec![Order { id: 1, customer: "Alice".into(), total: 120.0 }];
let rule = fake
    .when_query_match(ORDERS_OF)
    .param("customer", "Alice")
    .returns_rows(|columns| columns.from_type::<Order>(), &alice)?;

assert_eq!(orders_of(fake.db(), "Alice").await?, alice);
assert_eq!(rule.calls(), 1);
# Ok(())
# }
```

The rows are any `Serialize` type, and their columns are declared as a table schema declares them,
here with `|columns| columns.from_type::<Order>()`. A rule matches:

- the SQL text exactly, with no whitespace or case folding;
- a statement from `sql_file!` by the text of its file;
- any parameters, unless the rule declares some with `.param(..)`, `.params(..)` or
  `.positional_param(..)`. Then the call's named parameters must equal them as a set, and its
  positional ones in order.

Rules are tried in the order they were registered, and the first that matches answers.
`.times(n)` limits a rule to `n` calls, retries included, so a later rule answers the calls after
them. Besides `returns_rows`, a rule answers with `returns_dml` for a DML statement and the rows it
changed, `returns_statement` for DDL, `fails` for a refused call and `fails_job` for a job that
fails once it runs. One rule answers every terminal of a call: the rows for `query()`, the stats
for `execute()` and the schema for `dry_run()`, with `.bytes_processed(..)` if set.

`fake.when_query(|query| ...)` matches with your own code instead of the SQL text. The closure
gets the call's SQL and its parameters, read back as any `Deserialize` type:

```rust
# use bigquery::*;
# use serde::{Deserialize, Serialize};
# #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
# struct Order { id: i64, customer: String, total: f64 }
# const ORDERS_OF: &str = "SELECT id, customer, total FROM shop.orders WHERE customer = @customer";
# async fn orders_of(db: &BigQueryDb, customer: &str) -> BigQueryResult<Vec<Order>> {
#     db.fluent().query(ORDERS_OF).param("customer", customer).obj().query().await
# }
# #[tokio::main(flavor = "current_thread")]
# async fn main() -> BigQueryResult<()> {
use bigquery::testing::BigQueryFake;

let fake = BigQueryFake::start().await?;
let alice = vec![Order { id: 1, customer: "Alice".into(), total: 120.0 }];
let rule = fake
    .when_query(|query| {
        query.sql().contains("FROM shop.orders")
            && query.param::<String>("customer").is_some_and(|name| name.starts_with('A'))
    })
    .returns_rows(|columns| columns.from_type::<Order>(), &alice)?;

assert_eq!(orders_of(fake.db(), "Alice").await?, alice);
assert_eq!(rule.calls(), 1);
# Ok(())
# }
```

`query.param::<T>(name)` reads a named parameter, with the name compared ignoring case as
BigQuery compares it, and `query.positional_param::<T>(index)` a positional one, the first at 0.
Both return `None` when the call has no such parameter or its value does not read as `T`. A STRUCT
reads as a struct or a map, an ARRAY as a `Vec`, and a RANGE as `BigQueryRange`. `.param(..)` and
`.positional_param(..)` narrow a `when_query` rule as they narrow a SQL one.

## Tables, reads and writes

A table on the fake holds rows. `fake.table(..)` creates it with its columns, and its dataset if
that is missing, and `.rows(..)` adds rows a read is served:

```rust
# use bigquery::*;
# use serde::{Deserialize, Serialize};
# #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
# struct Order { id: i64, customer: String, total: f64 }
# const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
# const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
# #[tokio::main(flavor = "current_thread")]
# async fn main() -> BigQueryResult<()> {
use bigquery::testing::BigQueryFake;

let fake = BigQueryFake::start().await?;
let orders = vec![
    Order { id: 1, customer: "Alice".into(), total: 120.0 },
    Order { id: 2, customer: "Bob".into(), total: 80.5 },
];
fake.table(SHOP.table(ORDERS), |columns| columns.from_type::<Order>())
    .rows(&orders)
    .create()?;

let read: Vec<Order> = fake.db().fluent().select().from(SHOP.table(ORDERS)).obj().query().await?;
assert_eq!(read, orders);
# Ok(())
# }
```

Writes append to the table, and `fake.rows::<T>(table)` reads back the rows written:

```rust
# use bigquery::*;
# use serde::{Deserialize, Serialize};
# #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
# struct Order { id: i64, customer: String, total: f64 }
# const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
# const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
# async fn save(db: &BigQueryDb, orders: &[Order]) -> BigQueryResult<BigQueryWriteSummary> {
#     db.fluent().insert().into(SHOP.table(ORDERS)).objects(orders).execute().await
# }
# #[tokio::main(flavor = "current_thread")]
# async fn main() -> BigQueryResult<()> {
use bigquery::testing::BigQueryFake;

let fake = BigQueryFake::start().await?;
fake.table(SHOP.table(ORDERS), |columns| columns.from_type::<Order>())
    .create()?;
let orders = vec![
    Order { id: 1, customer: "Alice".into(), total: 120.0 },
    Order { id: 2, customer: "Bob".into(), total: 80.5 },
];

let summary = save(fake.db(), &orders).await?;

assert_eq!(summary.rows_written, 2);
let written: Vec<Order> = fake.rows(SHOP.table(ORDERS))?;
assert_eq!(written, orders);
# Ok(())
# }
```

`rows` returns the visible rows, in the order they were acknowledged. Rows become visible as
BigQuery makes them visible for each write mode:

- the default and committed streams: when acknowledged;
- a buffered stream: up to the last flushed offset;
- a pending stream: at its commit, and never without one.

Rows are matched to the table's columns by name, ignoring case, and the fields of a RECORD column
the same way. Rows written before a column or a RECORD field was added, and flushed or committed
after, read it as NULL, as do the rows the table already holds.

A table read is served the table's rows, projected to the selected columns. The fake evaluates
no filter, so a read with a row restriction needs a rule,
`fake.when_read(table).row_restriction(..)`, with the rows it returns.
`fake.when_rows_rejected::<T>(table, reason, |row| ...)` fails each append request that has a row
the predicate accepts, with a row error per such row, and writes nothing of that request, as
BigQuery does. A row written from a serde type fails the same way when it has
a value beyond the precision of its `NUMERIC(p,s)` or `BIGNUMERIC(p,s)` column, and a value with
more decimal places than the scale is rounded.

A CDC writer's rows are recorded as changes and not applied to the table's rows.
`fake.changes::<T>(table)` returns them, each with its change type and sequence number.

Datasets and tables, from `fake.create_dataset(..)`, `fake.table(..)` or the code under test, are
the fake's state, and the admin calls are served from it.

## Faults and retries

A query rule's `fails` scripts the failure of a query. `fake.when_fault(rpc).fails(..)` does the
same for every other RPC, and `.on_table(..)` narrows it to the calls on one table. A fault answers
before every rule and before the tables, as one of:

- `BigQueryFakeFault::status(code, message)`, a gRPC status, which the client maps as it maps
  BigQuery's own: `NotFound` is a `DataNotFoundError`, `Unavailable` a retryable
  `DatabaseError`, etc.;
- `BigQueryFakeFault::ConnectionDropped`, a connection closed without an answer, which the
  client retries as a transport error;
- `BigQueryFakeFault::Hang`, a call that never answers. The client waits until its own timeout,
  if it has one.

Retries against the fake wait no backoff, so a test that scripts failures runs at full speed.
A failure a retry gets past, then an outage the client gives up on:

```rust
# use bigquery::*;
# use serde::{Deserialize, Serialize};
# #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
# struct Order { id: i64, customer: String, total: f64 }
# const ORDERS_OF: &str = "SELECT id, customer, total FROM shop.orders WHERE customer = @customer";
# async fn orders_of(db: &BigQueryDb, customer: &str) -> BigQueryResult<Vec<Order>> {
#     db.fluent().query(ORDERS_OF).param("customer", customer).obj().query().await
# }
# #[tokio::main(flavor = "current_thread")]
# async fn main() -> BigQueryResult<()> {
use bigquery::testing::{BigQueryFake, BigQueryFakeCode, BigQueryFakeFault};

let fake = BigQueryFake::start().await?;
let alice = vec![Order { id: 1, customer: "Alice".into(), total: 120.0 }];
let unavailable =
    || BigQueryFakeFault::status(BigQueryFakeCode::Unavailable, "backend went away");
let lost = fake.when_query_match(ORDERS_OF).times(1).fails(unavailable())?;
let answer = fake
    .when_query_match(ORDERS_OF)
    .param("customer", "Alice")
    .returns_rows(|columns| columns.from_type::<Order>(), &alice)?;
let outage = fake
    .when_query_match(ORDERS_OF)
    .param("customer", "Bob")
    .fails(unavailable())?;

assert_eq!(orders_of(fake.db(), "Alice").await?, alice);
assert_eq!((lost.calls(), answer.calls()), (1, 1));

let failed = orders_of(fake.db(), "Bob").await;
assert!(matches!(&failed, Err(err) if err.retry_possible()), "{failed:?}");
assert_eq!(outage.calls(), 4, "the first attempt and max_retries = 3 retries");
# Ok(())
# }
```

On `AppendRows`, `ConnectionDropped` writes the request and then drops the connection, as an
acknowledgement lost in transit. The client sends the request again, so on the default stream,
which is at least once, its rows are written twice. A write with `.exactly_once()` sends it with
its offset, and the table keeps one copy:

```rust
# use bigquery::*;
# use serde::{Deserialize, Serialize};
# #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
# struct Order { id: i64, customer: String, total: f64 }
# const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
# const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
# #[tokio::main(flavor = "current_thread")]
# async fn main() -> BigQueryResult<()> {
use bigquery::testing::{BigQueryFake, BigQueryFakeFault, BigQueryFakeRpc};

let fake = BigQueryFake::start().await?;
fake.table(SHOP.table(ORDERS), |columns| columns.from_type::<Order>())
    .create()?;
let lost = fake
    .when_fault(BigQueryFakeRpc::AppendRows)
    .times(1)
    .fails(BigQueryFakeFault::ConnectionDropped)?;
let orders = vec![Order { id: 1, customer: "Alice".into(), total: 120.0 }];

fake.db()
    .fluent()
    .insert()
    .into(SHOP.table(ORDERS))
    .objects(&orders)
    .exactly_once()
    .execute()
    .await?;

assert_eq!(fake.rows::<Order>(SHOP.table(ORDERS))?, orders);
assert_eq!(lost.calls(), 1);
# Ok(())
# }
```

## Unmatched calls

A call that nothing answers, such as a query no rule matches or an RPC the fake does not serve,
fails with `Unimplemented`, which the client does not retry. The fake records it, and
`fake.verify()` panics with every such call and the rules that were there to answer it. It also
panics for a rule limited with `.times(n)` that did not answer exactly `n` calls.

`verify` runs when the fake is dropped, so a test fails on an unscripted call even when the code
under test swallows its error:

```rust,should_panic
# use bigquery::*;
# use serde::{Deserialize, Serialize};
# #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
# struct Order { id: i64, customer: String, total: f64 }
# const ORDERS_OF: &str = "SELECT id, customer, total FROM shop.orders WHERE customer = @customer";
# async fn orders_of(db: &BigQueryDb, customer: &str) -> BigQueryResult<Vec<Order>> {
#     db.fluent().query(ORDERS_OF).param("customer", customer).obj().query().await
# }
# #[tokio::main(flavor = "current_thread")]
# async fn main() -> BigQueryResult<()> {
use bigquery::testing::BigQueryFake;

let fake = BigQueryFake::start().await?;
fake.when_query_match(ORDERS_OF)
    .param("customer", "Alice")
    .returns_rows(|columns| columns.from_type::<Order>(), Vec::<Order>::new())?;

let unmatched = orders_of(fake.db(), "Bob").await;
assert!(matches!(&unmatched, Err(err) if !err.retry_possible()), "{unmatched:?}");

drop(fake); // Panics: no rule answers the query for Bob
# Ok(())
# }
```

If the test is already panicking when the fake is dropped, the problems are logged with
`tracing::error!` instead, so the first panic is the one the test reports.

## Choosing the fake or BigQuery at startup

`fake.db()` is a real `BigQueryDb` connected to the fake's loopback server, so the fake needs no
trait shared with the real client. Code that takes a `BigQueryDb`, as an argument or a field of
the application state, runs unchanged against either. This also makes it possible to run the
whole application against the fake, with a factory that picks one at startup from the
application's own configuration, such as an environment variable. The library itself reads no
variable for this.

The factory returns an owner of the client. Dropping a `BigQueryFake` stops its server and runs
`verify()`, so the owner keeps the fake alive next to the client it hands out:

```rust,no_run
# use bigquery::*;
# use serde::{Deserialize, Serialize};
# #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
# struct Order { id: i64, customer: String, total: f64 }
# const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
# const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
# const ORDERS_OF: &str = "SELECT id, customer, total FROM shop.orders WHERE customer = @customer";
# async fn orders_of(db: &BigQueryDb, customer: &str) -> BigQueryResult<Vec<Order>> {
#     db.fluent().query(ORDERS_OF).param("customer", customer).obj().query().await
# }
pub struct AppBigQuery {
    db: BigQueryDb,
    _fake: Option<bigquery::testing::BigQueryFake>,
}

impl AppBigQuery {
    pub fn db(&self) -> &BigQueryDb {
        &self.db
    }
}

/// BigQuery in `project`, or a seeded fake when `APP_BIGQUERY=fake` is set.
pub async fn bigquery_db(project: &str) -> BigQueryResult<AppBigQuery> {
    if std::env::var("APP_BIGQUERY").as_deref() == Ok("fake") {
        let fake = bigquery::testing::BigQueryFake::start().await?;
        seed(&fake)?;
        return Ok(AppBigQuery {
            db: fake.db().clone(),
            _fake: Some(fake),
        });
    }
    Ok(AppBigQuery {
        db: BigQueryDb::new(project).await?,
        _fake: None,
    })
}

fn seed(fake: &bigquery::testing::BigQueryFake) -> BigQueryResult<()> {
    let alice = vec![Order { id: 1, customer: "Alice".into(), total: 120.0 }];
    fake.table(SHOP.table(ORDERS), |columns| columns.from_type::<Order>())
        .rows(&alice)
        .create()?;
    fake.when_query_match(ORDERS_OF)
        .returns_rows(|columns| columns.from_type::<Order>(), &alice)?;
    Ok(())
}

# #[tokio::main(flavor = "current_thread")]
# async fn main() -> BigQueryResult<()> {
// The rest of the application sees only the client
let bigquery = bigquery_db("my-gcp-project-id").await?;
let orders = orders_of(bigquery.db(), "Alice").await?;
# let _ = orders;
# Ok(())
# }
```

The fake answers a query only from a rule, so the fake branch seeds a rule for every query the
application sends, besides the tables it reads and writes. An application run against the fake
with `APP_BIGQUERY=fake cargo run` panics when the fake is dropped if a call went unanswered, as
a test does. An application that wants the fake out of its production builds can put the fake
branch behind a feature of its own.

## Limits

The fake binds its own port, so tests run in parallel, and a fake is `Send + Sync`, so a test can
share it with the tasks it spawns. A call sees the rules registered before it arrives.

Tokio's paused time, `#[tokio::test(start_paused = true)]`, is not supported. A paused runtime
jumps to its next timer whenever it is idle, and waiting on the fake's loopback socket counts as
idle.

The fake does not provide:

- SQL evaluation. The DDL that `.sync()` sends for a rename, drop or widen reaches the query rules
  and does not change the table;
- read filters. A row restriction needs a read rule, and `sample_percentage` and `snapshot_time`
  are served the current rows;
- failures in the middle of a read stream, and partial acknowledgements of an append;
- polling of running jobs, other than for `fails_job` and queries with a destination table;
- CDC changes applied to the table's rows;
- Arrow appends in Arrow types other than the table's own. A decimal is the exception: it is
  rescaled to the scale of its `NUMERIC(p,s)` or `BIGNUMERIC(p,s)` column and checked against its
  precision;
- the length checks of `STRING(n)` and `BYTES(n)`, default values, request limits and quotas;
- a read that selects a field of a STRUCT column, such as `.fields(["address.city"])`;
- `ListJobs`, `DeleteJob`, models, routines, row access policies, table snapshots and copies.

Full example available [here](https://github.com/abdolence/bigquery-rs/blob/master/examples/testing-fake.rs).