rivet-cli 0.19.0

<!-- GENERATED by dev/docgen/config_md.py from `rivet schema config` — DO NOT EDIT BY HAND. -->

# Config reference (generated — rivet-cli 0.19.0)

Rendered from the JSON Schema `rivet schema config` emits (schemars ← the Rust `Config` types). It cannot drift from the code. Hand-written guidance lives in the surrounding config guide; **this table set is generated** — edit the Rust structs, not this block.

### Top level (`rivet.yaml`)

| Field | Type | Required | Description |
|-------|------|:--------:|-------------|
| `source` | [SourceConfig](#sourceconfig) | **yes** |  |
| `exports` | array of [ExportConfig](#exportconfig) | **yes** |  |
| `notifications` | [NotificationsConfig](#notificationsconfig) |  |  |
| `parallel_exports` | `boolean` |  |  |
| `parallel_export_processes` | `boolean` |  |  |
| `load` | object |  | Reserved for a downstream, first-party warehouse loader: the load target lives here so ONE config can drive both the export and a downstream load. Rivet stops at "file in a bucket" and **does not interpret** this block — it is accepted and ignored, present only so `rivet check` / `run` / `apply` don't reject a config that carries it. |

### `source`

| Field | Type | Required | Description |
|-------|------|:--------:|-------------|
| `type` | `postgres` \| `mysql` \| `mssql` \| `mongo` | **yes** |  |
| `url` | `string` |  |  |
| `url_env` | `string` |  |  |
| `url_file` | `string` |  |  |
| `host` | `string` |  |  |
| `port` | `integer` |  |  |
| `user` | `string` |  |  |
| `password` | `string` |  |  |
| `password_env` | `string` |  |  |
| `database` | `string` |  |  |
| `environment` | `local` \| `replica` \| `production` |  | Operational profile of the source database.  Selects the **default** tuning profile when none is explicitly set in `source.tuning.profile` or `export.tuning.profile`:  | `environment`           | default profile | |-------------------------|------------------| | `production` (default)  | `balanced` (50 ms throttle, 10 k batch, retries) | | `replica`               | `balanced` | | `local`                 | `fast` (no throttle, 50 k batch — saves ~30% wall on localhost) |  Explicit `tuning.profile:` always wins over this hint. |
| `tuning` | [TuningConfig](#tuningconfig) |  |  |
| `tls` | [TlsConfig](#tlsconfig) |  | Transport security settings (ADR: SecOps). When absent, Rivet connects without TLS — a warning is emitted so operators are aware. See [`TlsConfig`]. |
| `mongo` | [MongoConfig](#mongoconfig) |  | MongoDB-specific read options (`source.mongo:`). Honoured only when `type: mongo`; ignored by the SQL engines. See [`MongoConfig`]. |

### `source.mongo` (MongoDB read options)

| Field | Type | Required | Description |
|-------|------|:--------:|-------------|
| `json` | `relaxed` \| `canonical` |  | JSON rendering of the `document` column. `relaxed` (default) keeps common scalars native (`42`, `"x"`); `canonical` wraps every number (`{"$numberLong":"…"}`) so Int64/Double round-trip losslessly through a JSON-number parser that would otherwise clamp values beyond 2^53. |
| `read_concern` | `server` \| `snapshot` |  | Read concern for the collection scan. `snapshot` gives a **point-in-time consistent** full export (no doc missed/double-read under concurrent writes) — requires MongoDB 5.0+ on a replica set; a standalone rejects it. Default (`server`) uses the server's default read concern. |
| `no_cursor_timeout` | `boolean` |  | Keep the scan cursor alive past the server's idle timeout (default 10 min) so a slow destination cannot let the server reap the cursor mid-scan and silently drop the tail of a large collection. Default: `true`. |
| `page_size` | `integer` |  | When set, read the collection with **keyset (seek) pagination** on `_id` instead of one long-held cursor: each page is a bounded `find({_id: {$gt: last}}).sort({_id: 1}).limit(page_size)` — an indexed range scan that becomes one output part file. Bounds longest-query time (no 35-minute cursor to hit a timeout / snapshot window) and is the base for parallel `_id`-range reads. Requires an **ObjectId** `_id` (the default); a non-ObjectId key errors with a clear message. Unset ⇒ the single-cursor full scan. |
| `resume` | `boolean` |  | With keyset paging (`page_size`), persist the last committed `_id` and **resume** from it next run — a crashed export continues where it left off, and a re-run captures only documents inserted since (ObjectId `_id` is time-ordered). Default `false` re-reads the whole collection each run (plain `mode: full` semantics). No effect without `page_size`. |

### `source.tls`

| Field | Type | Required | Description |
|-------|------|:--------:|-------------|
| `mode` | `disable` \| `require` \| `verify-ca` \| `verify-full` |  | Enforcement level. See [`TlsMode`]. |
| `ca_file` | `string` |  | PEM-encoded CA certificate to trust for server verification. Required for [`TlsMode::VerifyCa`] and [`TlsMode::VerifyFull`] against a private CA. |
| `accept_invalid_certs` | `boolean` |  | Accept certificates not chained to a trusted CA. Dangerous — disables server authentication — and only honored when explicitly `true`. |
| `accept_invalid_hostnames` | `boolean` |  | Accept certificates whose subjectAltName does not match the connection hostname. Dangerous — disables hostname verification. |

### `exports[]`

| Field | Type | Required | Description |
|-------|------|:--------:|-------------|
| `name` | `string` | **yes** |  |
| `query` | `string` |  |  |
| `query_file` | `string` |  |  |
| `table` | `string` |  | Shortcut for `query: "SELECT * FROM <schema>.<table>"`.  Accepts `table` or `schema.table` with ASCII-only identifiers (`[A-Za-z_][A-Za-z0-9_]*`). Generates an unquoted single-table query so the Postgres NUMERIC catalog-hint resolver recognises it and auto-types `numeric(p,s)` columns without manual overrides.  Mutually exclusive with `query` and `query_file`. |
| `tables` | array of `string` |  | CDC only: capture **several** tables through ONE change stream (one PostgreSQL slot / one MySQL binlog connection) instead of one export — and one slot — per table. Each table's parts land under `<destination>/<table>/` with their own `manifest.json` + `_SUCCESS`; the checkpoint (stream position) is shared. Mutually exclusive with `table:`. Not yet supported for SQL Server (capture instances are per-table). |
| `mode` | `full` \| `incremental` \| `chunked` \| `time_window` \| `cdc` |  |  |
| `cdc` | [CdcExportConfig](#cdcexportconfig) |  | Change-data-capture settings, required when `mode: cdc`. Reuses the export's `table`, `destination`, and `format`; carries only the CDC-specific knobs (resume checkpoint, per-engine stream params). |
| `cursor_column` | `string` |  |  |
| `cursor_fallback_column` | `string` |  | Secondary column for [`IncrementalCursorMode::Coalesce`] only (see ADR-0007). |
| `incremental_cursor_mode` | `single_column` \| `coalesce` |  | How primary (and optional fallback) columns drive incremental progression. |
| `chunk_column` | `string` |  |  |
| `chunk_dense` | `boolean` |  |  |
| `chunk_size` | `integer` |  |  |
| `chunk_size_memory_mb` | `integer` |  | Target memory budget per chunk in MB. When set, `chunk_size` is derived from this budget at plan-build time using a `pg_class` row-size estimate (`pg_relation_size / reltuples`), clamped to `[10_000, 5_000_000]` rows.  Mutually exclusive with an explicit non-default `chunk_size:`. Only applies to `mode: chunked` on a Postgres source using the `table:` shortcut (the row-size probe needs a known relation).  ```yaml exports:   - name: page_views     table: public.page_views     mode: chunked     chunk_size_memory_mb: 256 ``` |
| `chunk_count` | `integer` |  | Divide the column range into exactly this many equal chunks. Mutually exclusive with `chunk_dense` and `chunk_by_days`. When set, `chunk_size` is computed dynamically from min/max. |
| `chunk_by_days` | `integer` |  |  |
| `chunk_by_key` | `string` |  | Keyset (seek) pagination on this single index-backed unique key — the source-safe shape for tables without a single-integer PK (OPT-4). The column MUST be backed by a usable index (PK or unique); the planner refuses a non-indexed key rather than emit a full-scan + filesort query. |
| `parallel` | `integer` |  |  |
| `wave` | `integer` |  | Advisory execution wave (1 = highest priority, run first). Written by `rivet plan` from the source-aware prioritization score (see ADR-0006) and consumed by `rivet apply`, which runs exports wave-by-wave in ascending order. `None` = unscheduled (apply treats it as the last wave). Operators may hand-edit it; a later `rivet plan` refreshes it in place. |
| `parallel_safe` | `boolean` |  | Whether this export is cheap enough to run concurrently with its wave-mates under `rivet apply --parallel-export-processes`. Written by `rivet plan` (true when the source-aware cost class is `Low`, i.e. < ~100K rows); a heavier table already chunk-parallelizes internally, so two of them at once would overload the source. `None`/`false` → the export runs alone within its wave. Operators may hand-edit it; a later `rivet plan` refreshes it in place. |
| `time_column` | `string` |  |  |
| `time_column_type` | `timestamp` \| `unix` |  |  |
| `days_window` | `integer` |  |  |
| `partition_by` | `string` |  | Date/time output partitioning: split this export's rows into one destination sub-prefix per calendar bucket of this **DATE or TIMESTAMP** column, bucketed by [`partition_granularity`](Self::partition_granularity) (`day` / `month` / `year`), in a Hive-style `col=value/` layout (`created_at=2023-01-01/`, `created_at=2023-01/`, `created_at=2023/`). Requires a `{partition}` token in `destination.path` / `destination.prefix`.  This is **not** arbitrary value partitioning: the column's min/max is read and parsed as a date to generate contiguous calendar buckets, so a non-temporal column (e.g. `partition_by: status`) fails at run time with "could not parse partition min '<value>' from column '<col>' as a date". To split by a categorical column, write one export per value with a `WHERE` filter instead.  Orthogonal to `mode`: each partition runs the export's own mode, so `mode: chunked` chunks *within* a day. Rows whose partition column is NULL land in `col=__HIVE_DEFAULT_PARTITION__/` (Hive default partition) so no row is silently dropped. Not compatible with `mode: time_window`.  ```yaml exports:   - name: events     table: events     partition_by: created_at        # must be a DATE or TIMESTAMP column     partition_granularity: day     destination:       type: s3       bucket: my-bucket       prefix: "events/{partition}/"   # → events/created_at=2023-01-01/ ``` |
| `partition_granularity` | `day` \| `month` \| `year` |  | Calendar bucket width for [`partition_by`](Self::partition_by): `day` (default), `month`, or `year`. Determines how the partition column's date/timestamp range is split into contiguous Hive buckets (`col=2023-01-01/` / `col=2023-01/` / `col=2023/`). Has no effect unless `partition_by` is set. |
| `format` | `parquet` \| `csv` | **yes** |  |
| `compression` | `zstd` \| `snappy` \| `gzip` \| `lz4` \| `none` |  |  |
| `compression_level` | `integer` |  |  |
| `compression_profile` | `none` \| `fast` \| `balanced` \| `compact` |  |  |
| `skip_empty` | `boolean` |  |  |
| `destination` | [DestinationConfig](#destinationconfig) | **yes** |  |
| `verify` | `size` \| `content` |  | Integrity depth required of `--validate` for this export's parts. `size` (default) accepts size-only verification; `content` requires every part's content MD5 to be checked against the store's listing (no download) and **fails** validation for any part that could only be size-verified — e.g. a part too large to upload as a single PUT (raise `max_file_size` down so it fits), or a backend that exposes no checksum. |
| `meta_columns` | [MetaColumns](#metacolumns) |  |  |
| `quality` | [QualityConfig](#qualityconfig) |  |  |
| `max_file_size` | `string` |  | Rotate to a new part when the current file reaches this size. Accepts `B`/`KB`/`MB`/`GB` (case-insensitive) or a bare byte count; a fractional value is allowed (`1.5GB`). Units are binary (IEC-style): `KB` = 1024 bytes, `MB` = 1024 KB, `GB` = 1024 MB. Example: `256MB`. |
| `chunk_checkpoint` | `boolean` |  |  |
| `chunk_max_attempts` | `integer` |  |  |
| `tuning` | [TuningConfig](#tuningconfig) |  |  |
| `source_group` | `string` |  | Optional logical group for shared source capacity (replica, host). Advisory prioritization only. |
| `reconcile_required` | `boolean` |  | Hint (Epic C / ADR-0006) that this export should always be treated as reconcile-heavy by planning, independent of the `--reconcile` CLI flag. Advisory only. |
| `columns` | `object` |  | Per-column type overrides (roadmap §8). Keys are column names; values are short type strings such as `decimal(18,2)`, `timestamp_tz`, `json`.  ```yaml exports:   - name: payments     columns:       amount: decimal(18,2)       fee: decimal(18,6)       created_at: timestamp_tz ```  Overrides take priority over autodetection and are validated at plan time — an invalid type string fails before the export runs. |
| `target` | `string` |  | Downstream warehouse this export targets (`bigquery` / `bq`, `duckdb`). When set, `rivet check --type-report` resolves each column against it (native type, honest autoload type, recovery hint) without needing `--target` on the CLI — the CLI flag still wins when both are present. The Parquet interchange stays target-neutral (ADR-0014 T2); `target:` only drives guidance and the future load-schema artifact.  ```yaml exports:   - name: payments     target: bigquery ``` |
| `on_schema_drift` | `warn` \| `continue` \| `fail` |  | Policy applied when structural schema drift is detected (column added, removed, or retyped). Defaults to `warn`: log a warning and continue. |
| `shape_drift_warn_factor` | `number` |  | Growth-factor threshold for data shape drift warnings (Epic 8). When a string/binary column's max observed byte length in the current run exceeds `stored_max * shape_drift_warn_factor`, Rivet logs a warning. `None` uses the default of 2.0. Set to `0.0` to disable shape tracking. |
| `parquet` | [ParquetConfig](#parquetconfig) |  | Parquet row group tuning. Only meaningful when `format: parquet`. When absent, the parquet library default (1,048,576 rows/group) is used. |

### `exports[].cdc` (mode: cdc)

| Field | Type | Required | Description |
|-------|------|:--------:|-------------|
| `initial` | `snapshot` |  | First-run behaviour: `snapshot` = anchor → full snapshot → drain (see [`CdcInitialMode`]). Omitted ⇒ capture changes only (the default; the operator owns the initial load). |
| `checkpoint` | `string` |  | Persist/resume the source log position to this file. Omit to tail from the current position without checkpointing. |
| `until_current` | `boolean` |  | Catch up to the source's current end and exit (a bounded run), instead of streaming indefinitely — ideal for a scheduler. For MySQL this is a non-blocking binlog dump; PostgreSQL / SQL Server already drain-and-exit. |
| `max_events` | `integer` |  | Stop after N change events (default: until end of stream / interrupted). |
| `rollover` | `integer` |  | Rows per output part file (default 100000). A part also rolls at a transaction boundary, so it never splits a transaction. Larger ⇒ fewer, bigger files but more drain memory — the PostgreSQL peek reads a part's worth per batch, so drain RSS is O(rollover). Tune per workload: raise it to cut file count, lower it to cap memory on a small extractor. |
| `rollover_memory_mb` | `integer` |  | Roll a part once its buffered changes reach this many MB, whichever comes first with `rollover`. Caps the in-memory buffer and the part file size by bytes instead of a fixed row count — predictable for tables with wide (large JSON / blob) rows, mirroring the batch path's `batch_size_memory_mb`. |
| `server_id` | `integer` |  | MySQL replica server-id for the binlog connection (default 4271; must be distinct from the source's and any other replica). |
| `slot` | `string` |  | PostgreSQL logical replication slot name (default `rivet_slot`). |
| `capture_instance` | `string` |  | SQL Server CDC capture instance, e.g. `dbo_orders` — required for `sqlserver://` sources. |

### `exports[].tuning`

| Field | Type | Required | Description |
|-------|------|:--------:|-------------|
| `profile` | `fast` \| `balanced` \| `safe` |  |  |
| `batch_size` | `integer` |  |  |
| `batch_size_memory_mb` | `integer` |  | Target memory per batch in MB. Mutually exclusive with batch_size. |
| `throttle_ms` | `integer` |  |  |
| `statement_timeout_s` | `integer` |  |  |
| `max_retries` | `integer` |  |  |
| `retry_backoff_ms` | `integer` |  |  |
| `lock_timeout_s` | `integer` |  |  |
| `memory_threshold_mb` | `integer` |  |  |
| `max_batch_memory_mb` | `integer` |  | Hard cap on Arrow batch memory in MB. When a batch exceeds this limit, `on_batch_memory_exceeded` determines the response. |
| `on_batch_memory_exceeded` | `warn` \| `fail` \| `auto_shrink` |  | Policy applied when a batch exceeds `max_batch_memory_mb`. Default: `warn`. |
| `adaptive` | `boolean` |  | Enable real-time batch size adaptation based on DB pressure metrics. Postgres: samples `pg_stat_bgwriter`. MySQL: samples `Innodb_log_waits`. Also arms the OPT-2 concurrency governor when `parallel > 1`. |
| `min_parallel` | `integer` |  | Floor for the concurrency governor (lowest parallelism under pressure). Default 1. Ceiling is the export's `parallel`. |
| `max_value_mb` | `integer` |  | Hard per-value size ceiling in MB. A single text/JSON/blob cell larger than this aborts the run with `RIVET_VALUE_TOO_LARGE`. `0` disables the guard. Default: 256. |

### `exports[].destination`

| Field | Type | Required | Description |
|-------|------|:--------:|-------------|
| `type` | `local` \| `s3` \| `gcs` \| `azure` \| `stdout` | **yes** |  |
| `bucket` | `string` |  |  |
| `prefix` | `string` |  |  |
| `path` | `string` |  |  |
| `region` | `string` |  |  |
| `endpoint` | `string` |  |  |
| `credentials_file` | `string` |  |  |
| `access_key_env` | `string` |  |  |
| `secret_key_env` | `string` |  |  |
| `session_token_env` | `string` |  | Name of an env var holding an AWS STS session token, for use with short-lived credentials issued by AWS IAM Identity Center / SSO, `aws sts assume-role`, MFA-protected sessions, EKS IAM Roles for Service Accounts, etc.  Pair with `access_key_env` + `secret_key_env`. See `docs/cloud-auth.md` for the AWS auth-flow matrix. |
| `aws_profile` | `string` |  |  |
| `account_name` | `string` |  | Azure storage account name (the prefix in `<account>.blob.core.windows.net`). Plain string — not a secret. Pair with `account_key_env`. See `docs/cloud-auth.md` for the Azure auth-flow matrix. |
| `account_key_env` | `string` |  | Name of an env var holding the Azure Storage account key.  Treated as a credential and wiped from heap on drop — same SecOps treatment as `access_key_env`.  Pair with `account_name`.  Mutually exclusive with `sas_token_env`. |
| `sas_token_env` | `string` |  | Name of an env var holding an Azure Storage **SAS token** — typically a short-lived, scope-limited credential issued out-of-band (Azure portal / `az storage container generate-sas` / Azure SDK).  Use this instead of `account_key_env` when the operator does not have the long-lived account key or wants per-job scoped access.  Pair with `account_name`.  Mutually exclusive with `account_key_env`.  The token value is wiped from heap on drop via the same `Zeroizing<String>` wrapper as `account_key_env`.  Leading `?` is trimmed transparently so the operator can paste either the full `?sv=…&sig=…` query string or the raw token body. |
| `allow_anonymous` | `boolean` |  |  |

### `exports[].quality`

| Field | Type | Required | Description |
|-------|------|:--------:|-------------|
| `row_count_min` | `integer` |  |  |
| `row_count_max` | `integer` |  |  |
| `null_ratio_max` | `object` |  |  |
| `unique_columns` | array of `string` |  |  |
| `unique_max_entries` | `integer` |  | Cap on the number of distinct values tracked per column during uniqueness checks. When the limit is hit, a Warn issue is emitted and tracking stops for that column. Prevents unbounded HashSet growth on high-cardinality columns. |

### `exports[].parquet`

| Field | Type | Required | Description |
|-------|------|:--------:|-------------|
| `row_group_strategy` | `auto` \| `fixed_rows` \| `fixed_memory` |  | How to determine the row group size. Default: `auto`. |
| `row_group_rows` | `integer` |  | Exact number of rows per group (`fixed_rows` only). |
| `target_row_group_mb` | `integer` |  | Target Arrow buffer memory per row group in MB (`auto` and `fixed_memory`). Default: 128. |
| `max_row_group_mb` | `integer` |  | Hard upper bound on row group memory in MB. When set, further reduces computed row count. |