Skip to main content

Module usage

Module usage 

Source
Expand description

Per-run usage metering (#704): what a run moved and what it asked its backends to do, so the CLI can account for cost per pipeline / row / dataset and enforce run budgets (#703).

A UsageMeter is attached to a Pipeline with Pipeline::with_usage_meter. The observability decorators count every record and an estimate of its serialized size; the pre-labelled RoundtripRecorder each connector already receives tallies backend round trips by op and carries cost signals — backend-reported usage such as BigQuery’s bytes billed or a streaming insert’s payload size — into the same meter. A library caller that attaches no meter pays nothing: counting is skipped entirely.

Byte counts are estimates of the JSON serialization (no allocation, one walk over each value) — good enough to attribute cost and enforce a max_bytes budget, never a wire-accurate figure.

Structs§

CostSignal
A backend-reported usage figure a connector learned during the run — BigQuery’s totalBytesBilled, the payload size of a streaming insert, a warehouse’s credits — attributed to the connector that reported it.
UsageMeter
The live, thread-safe counters of one run. Cheap to share (Arc).
UsageSnapshot
The counters a run accumulated, frozen for reporting and storage.

Enums§

UsageSide
Which side of the pipeline a round trip or signal came from.

Functions§

estimate_json_bytes
Approximate serialized JSON size of v, in bytes, without allocating.
estimate_page_bytes
Estimated serialized size of a page of records.