Crate faucet_stream 

faucet-stream

A declarative, config-driven data pipeline with pluggable source and sink connectors.

📖 Guide, tutorials & cookbook: https://pawansikawat.github.io/faucet-stream/

Feature flags

Feature	Description
source-rest (default)	REST API source with pagination, auth, transforms
source-graphql	GraphQL API source with cursor pagination
source-xml	XML/SOAP API source with XML-to-JSON conversion
source-grpc	gRPC source with dynamic protobuf messages
source-postgres	PostgreSQL query source
source-postgres-cdc	PostgreSQL CDC source (logical replication)
source-mysql	MySQL query source
source-mssql	Microsoft SQL Server query source
source-sqlite	SQLite query source
source-duckdb	DuckDB query source
source-sqs	AWS SQS source
source-nats	NATS source
source-sftp	SFTP source
source-s3	AWS S3 file source
source-mongodb	MongoDB query source
source-mongodb-cdc	MongoDB CDC source (Change Streams)
source-mysql-cdc	MySQL CDC source (binlog replication)
source-redis	Redis source (streams, lists, keys)
source-webhook	Webhook HTTP receiver source
source-websocket	WebSocket streaming source
source-csv	CSV file source
source-elasticsearch	Elasticsearch search/scroll source
source-kafka	Apache Kafka consumer source
source-kinesis	AWS Kinesis Data Streams source
source-spanner	Google Cloud Spanner query source
source-parquet	Apache Parquet file source (local, glob, S3)
source-delta	Apache Delta Lake source (local FS or S3/Azure/GCS, time travel)
source-databricks	Databricks SQL query source (Statement Execution API)
sink-bigquery	Google BigQuery streaming insert sink
sink-iceberg	Apache Iceberg sink (append-only, REST/Glue/SQL/HMS catalogs)
sink-postgres	PostgreSQL sink (jsonb or auto-mapped columns)
sink-jsonl	JSON Lines file sink
sink-snowflake	Snowflake SQL REST API sink
sink-mysql	MySQL sink
sink-mssql	Microsoft SQL Server sink
sink-sqlite	SQLite sink
sink-duckdb	DuckDB sink
sink-sqs	AWS SQS sink
sink-nats	NATS sink
sink-sftp	SFTP sink
sink-s3	AWS S3 file sink
sink-mongodb	MongoDB insert sink
sink-redis	Redis sink (streams, lists, key-value)
sink-csv	CSV file sink
sink-elasticsearch	Elasticsearch bulk index sink
sink-http	HTTP POST sink
sink-kafka	Apache Kafka producer sink
sink-kinesis	AWS Kinesis Data Streams sink
sink-spanner	Google Cloud Spanner mutation sink
sink-parquet	Apache Parquet file sink (local, S3)
encryption	AES-256-GCM at-rest sealing for file state-store bookmarks and per-line JSONL/DLQ output
sink-delta	Apache Delta Lake sink (append-only; local FS or S3/Azure/GCS)
kafka-schema-registry	Schema Registry support for Kafka connectors
source	All source connectors
sink	All sink connectors
full	Every connector

Re-exports

pub use faucet_lineage as lineage;lineage

pub use faucet_transform_sql as transform_sql;transform-sql

pub use faucet_transform_wasm as transform_wasm;transform-wasm

Modules

adaptive

Adaptive batch sizing — an AIMD controller that auto-tunes the effective write batch size per pipeline row from observed sink latency + error rate. Pure logic (no I/O); run_stream feeds it observations and emits metrics. See docs/superpowers/specs/2026-05-31-adaptive-batch-sizing-design.md.

async_stream

Asynchronous stream of elements.

authauth

Single-flight OAuth2 / token-endpoint auth providers (enable the auth feature). Share one across connectors via with_auth_provider or the CLI auth: { ref } catalog.

check

Preflight check types for faucet doctor (#126).

columnararrow

Opt-in Apache Arrow columnar record path (feature arrow, RFC 0002 / #375).

common_azuresink-azure-blob or source-azure-blob

Shared Azure Blob / ADLS Gen2 types (credentials enum, object-store builder), re-exported when either Azure Blob connector is enabled.

common_clickhousesink-clickhouse or source-clickhouse

Shared ClickHouse types (connection block, HTTP client builder), re-exported when either ClickHouse connector is enabled.

common_gcssink-gcs or source-gcs

common_kafkasink-kafka or source-kafka

common_kinesissink-kinesis or source-kinesis

Shared AWS Kinesis types (credentials enum, client builder), re-exported for library callers when either Kinesis connector is enabled.

common_pubsubsink-pubsub or source-pubsub

Shared Google Cloud Pub/Sub types (credentials enum, connection block, client builder), re-exported when either Pub/Sub connector is enabled.

common_redshiftsink-redshift or source-redshift

Shared Amazon Redshift types (credentials enum, connection block, pool builder), re-exported when either Redshift connector is enabled.

common_spannersink-spanner or source-spanner

Shared Cloud Spanner types (credentials enum, connection block, value conversion), re-exported for library callers when either Spanner connector is enabled.

compressioncompression

Transparent gzip / zstd compression wrappers for file-shaped connectors.

config

Configuration loading utilities.

contractcontract

Data contracts (issue #204): a declarative, versioned schema + semantic contract for a pipeline’s output — required fields, types, nullability, enum sets, regex patterns, numeric/length bounds — enforced per page after transforms and quality checks and before the sink write. Breaches fail, quarantine (via the DLQ), or warn per the contract-level policy.

discover

Live source introspection (faucet discover, #211).

dlq

Dead-letter queue (DLQ) wiring shared by the pipeline runner.

drift

Schema-drift detection + policy types (issue #194).

encryptionencryption

Encryption at rest for faucet-managed local files (#207).

error

Error types for faucet-stream.

futures_core

Core traits and types for asynchronous operations in Rust.

idempotency

Exactly-once / idempotent delivery primitives.

join

Pure hash-join logic for the topology join node (issue #72).

maskingmasking

PII detection + column-level masking policies (issue #206): a declarative, destination-scoped policy that classifies sensitive fields — by field-name pattern, by value detector (email / card-with-Luhn / SSN / phone / IPv4), or by explicit field list — and rewrites them per action (redact / hash / tokenize / partial).

observability

Pipeline-internal observability: tracing spans and metrics counters/ histograms wired automatically around every source, sink, transform, and state-store operation. See docs/superpowers/specs/2026-05-23-observability-otel-prometheus-design.md.

pipeline

Source-to-sink pipeline orchestration.

qualityquality

Data-quality checks: declarative per-record and per-batch assertions that quarantine violating records to the DLQ or abort the run. Pure evaluation; the pipeline wires the DLQ routing in run_stream.

replication

Incremental replication support.

resilience

Unified resilience policy: retry, backoff classification, circuit breaker, and poison-pill row handling. See docs/superpowers/specs/2026-06-17-resilience-policy-design.md.

retry

Shared exponential-backoff retry executor for HTTP-style sources.

schema

JSON Schema inference from record samples.

schemars

Schemars

shard

Source sharding for clustered (Mode B) execution.

sink

sourcesource-rest

stage

Pipeline-level transform stages. A TransformStage wraps one of six shapes:

state

topology

Multi-edge pipeline topology — fan-out (tee), fan-in (merge), and hash-join over an explicit node graph (issues #71 and #72).

traits

Shared traits for faucet sources and sinks.

transform

Record transformation pipeline.

transforming_source

Wrap any Source with a fixed list of TransformStages applied to every emitted record. The canonical way for library callers to attach stages (transforms wrapped via TransformStage::Map, plus Filter / Explode / Custom); the CLI uses this same type internally.

util

Shared utilities used across faucet source and sink crates.

verify

Read-time integrity verification for streaming object bodies.

write_mode

Unified write-mode types + planner shared by every upsert-capable sink.

Macros

json

Construct a serde_json::Value from a JSON literal.

schema_for

Generates a Schema for the given type using default settings. The default settings currently conform to JSON Schema 2020-12, but this is liable to change in a future version of Schemars if support for other JSON Schema versions is added.

Structs

AdaptiveBatchConfig

Configuration for the adaptive batch-size controller. Lives under execution.adaptive_batch_size. Default enabled = false (opt-in); when disabled the pipeline writes each page exactly as before.

Adjustment

A size change the controller decided on.

AimdController

Additive-increase / multiplicative-decrease controller. Pure + deterministic.

AuthReference

A { ref: <name> } pointer to a named provider in the top-level auth: catalog. The only permitted key is ref.

CancellationToken

A token which can be used to signal a cancellation request to one or more tasks.

CdcUnwrapSpectransform-cdc-unwrap

Spec for TransformStage::CdcUnwrap. Normalizes a CDC change-event envelope ({op, before, after, …}) into a flat row plus a marker field, so a downstream upsert sink never needs to understand CDC. A 1→0|1 stage: DDL/truncate events (and non-delete events with no after image) are dropped.

CheckContext

Inputs a probe may need. The doctor command enforces timeout on the whole check() call; connectors may also use it to bound their own client calls.

CheckReport

A connector’s full preflight report.

CheckTally

Per-check counters + elapsed time, keyed by check name. Emitted as metrics by the observability wrapper.

CircuitBreaker

Tracks consecutive failures; opens when threshold is reached.

CircuitBreakerConfig

Circuit-breaker tuning. Counts consecutive page-level write failures.

ColumnChange

One column’s drift, expressed in JSON-Schema type-fragment terms (e.g. {"type":"integer"} or {"type":["string","null"]}).

ColumnarPage

A page of records in columnar (Arrow) form, the columnar analogue of StreamPage.

CompiledContract

A fully compiled contract. Built once via CompiledContract::compile.

CompiledEncryption

A compiled, validated encryption policy: the derived write key plus every derived read key (write key first).

CompiledMasking

A masking policy compiled and ready to apply to pages.

CompiledQuality

A fully compiled quality spec. Built once via CompiledQuality::compile.

ContractOutcome

Result of applying the contract pass to one page.

ContractSpec

The contract: config block: a declarative, versioned schema + semantic contract for a pipeline’s output, enforced per page after transforms and quality checks and before the sink write.

ContractViolation

One observed contract breach.

DatasetDescriptor

One dataset discovered behind a source’s connection.

DeleteMarker

Identifies a record as a delete (vs. an upsert) by a marker field’s value. e.g. { field: "__op", values: ["d", "delete"] }.

DlqConfig

Pipeline-level DLQ wiring.

DlqStats

Counters returned alongside PipelineResult when a DLQ is wired.

DurationGuard

On Drop, records the elapsed time since construction into the named histogram with the supplied labels. Recording on drop guarantees a sample even if the surrounding future is cancelled or panics.

Edge

A directed edge from one node’s output to another’s input.

EncryptionSpec

The user-facing encryption: config block (state store / jsonl sink).

ExplodeSpectransform-explode

Spec for TransformStage::Explode. Fans one record out into N records based on the array at path. Compiled into a CompiledExplode at pipeline-build time; per-record work is one path resolve + N clones + N merges.

FieldContract

One promised top-level field: its type plus optional semantic constraints.

FileStateStore

File-backed StateStore. Each key maps to a JSON file at {root}/{safe_filename(key)}.json, written via atomic rename. The filename stem percent-encodes : as %3A so keys using the pipeline:rest:issues convention are valid on Windows.

FilterSpectransform-filter

Predicate spec for TransformStage::Filter. Compiled into a CompiledFilter at pipeline-build time; per-record work is a single path resolve + comparison.

GuaranteeInputs

Inputs to derive_delivery_guarantee — the facts about a concrete source × sink × config combination the derivation keys off.

HashJoin

A hash join: build the index, then probe it.

InstallReport

Report from install_observability so callers can log what actually happened (recorder installed vs. already-installed vs. disabled).

InstrumentedSink

Wraps a &dyn Sink (or any &S: Sink) and emits spans + metrics around write_batch and flush. Constructed by Pipeline::run.

InstrumentedSource

Wraps a &dyn Source (or any &S: Source) and emits spans + metrics around every call. Constructed by Pipeline::run and never exposed to end users; the wrapped source remains the user-facing object.

InstrumentedStateStore

Wraps an Arc<dyn StateStore> and emits faucet.state.* spans + metrics around every operation.

JoinConfig

Compiled join configuration. Dotted key paths are pre-split at construction.

JoinNode

A join node: the pure JoinConfig plus the labels of its two incoming edges identifying which upstream is the build (right) side and which is the probe (left) side.

JoinStats

Running counters for a join, mirroring the faucet_join_* metrics.

KeyTuple

Ordered key column → value pairs, in key declaration order.

Labels

Common labels carried by every span and metric.

LengthCheck

Built-in IntegrityCheck that fails when the number of bytes read does not match the length advertised by the object store (Content-Length for S3, object size for GCS). Catches a cleanly-truncated transfer that would otherwise be accepted as a complete object.

MaskHit

One field-masking event, for metrics/observability.

MaskRule

One masking rule: a matcher + the action to apply to matching fields, optionally scoped to a subset of destination sinks.

MaskingOutcome

Result of masking one page.

MaskingSpec

The masking: config block: a declarative, destination-scoped policy that classifies sensitive fields (by field-name pattern, by value detector, or by explicit field list) and applies a masking action per page. The masking pass runs first — before the quality, contract, and schema-drift passes and before every sink write — so PII never reaches a sink, the DLQ, or a lineage sample unmasked.

MatchSpec

How a rule classifies a field as sensitive. Any combination is allowed; a field matches the rule if any configured criterion matches (name pattern OR value detector OR explicit field name).

MemoryStateStore

In-memory StateStore for tests and ephemeral pipelines.

Node

A node in the topology: a stable id plus its typed kind.

ObservabilityConfig

Configuration for install_observability. Either or both sections may be None; unset sections install nothing.

Observation

One observed sub-batch write outcome fed to the controller.

OtelConfig

Pure-data OTLP export configuration. Contains no opentelemetry types.

Pipeline

A pipeline that moves data from a Source to a Sink.

PipelineResult

Result of a pipeline run.

PoisonPolicy

Poison-pill (per-row) policy.

Probe

One named probe within a CheckReport (e.g. "read", "auth", "network", "permissions", "schema", "io", "sentinel").

Projection

A single field projection: copy from (a dotted path into the build record) onto the probe record under the name as_.

PrometheusConfig

QualityOutcome

Result of applying the quality pass to one page.

QualitySpec

The quality: config block. Per-record checks run first (partitioning the page into survivors + quarantined); per-batch checks then run over the survivors.

QuarantinedRecord

A record removed from the page by a quality check, destined for the DLQ.

ResiliencePolicy

The umbrella policy attached to a run.

ResponseValidatorsource-rest

Optional callback to decide whether the token endpoint response is successful.

RestStreamsource-rest

A configured REST API stream that handles pagination, auth, and extraction.

RestStreamConfigsource-rest

Configuration for a RestStream.

RetryClassSet

The set of classes the user has opted into retrying. Backed by a fixed-size bool array (only four variants), so it is Copy and allocation-free.

RetryMetrics

Per-op labels for the metered runner. Identifies which pipeline / matrix row / I/O operation a retry belongs to so the resilience metrics (faucet_resilience_retries_total / _retry_sleep_seconds / _giveup_total) carry the spec’s {pipeline, row, op} label set. op is a closed set (sink_write / flush / state_put / source).

RetryPolicy

Retry configuration shared by the pipeline loop and source connectors.

RunStreamOptions

SchemaDiff

Result of diffing a page’s inferred shape against the destination schema.

SchemaDriftPolicy

Compiled, ready-to-run drift policy. Cheap to clone.

SchemaDriftSpec

User-facing schema: config block (pipeline level).

SchemaEvolution

The applyable subset of a SchemaDiff handed to crate::Sink::evolve_schema. Never carries incompatible columns — those are routed by the policy.

ShardSpec

One independently-processable slice of a shardable source.

StreamPage

One page emitted by Source::stream_pages.

Topology

A directed acyclic graph of typed nodes.

TopologyBuilder

Fluent builder for a Topology.

TopologyOptions

Per-run options for Topology::run.

TopologyResult

Outcome of a topology run.

TracingConfig

TransformingSource

Source decorator that applies a fixed list of compiled stages to every record. Emits faucet_transform_* metrics per page via instrumented_apply_stages.

UnwrappedEnvelope

A DLQ envelope parsed back into its original payload plus the metadata needed to inspect and replay it. Produced by unwrap_envelope.

VerifyingReader

An AsyncRead adapter that observes every byte read from inner and runs the configured IntegrityChecks at EOF. Wrap the raw network reader before any BufReader / decompression layer.

ViolatingRecord

A record removed from the page by a quarantine breach, destined for the DLQ.

WritePlan

The partition of a page by write mode. Infallible to build — per-row failures (missing/null key) land in failed with their original page index so the caller can route them to a DLQ or abort.

WriteSpec

Shared write-mode config, embedded in each upsert-capable sink config via #[serde(flatten)] so write_mode / key / delete_marker appear at the sink-config top level.

Enums

AdjustDirection

Direction of a batch-size adjustment (metric label).

AdjustReason

Why an adjustment happened (metric label).

Authsource-rest

Supported authentication methods.

AuthSpec

A connector’s auth: field: either an inline auth definition A (the { type, config } shape), or a { ref: <name> } reference to a shared provider defined in the top-level auth: catalog.

BackoffKind

How the inter-attempt delay grows.

BatchCheck

A per-batch check, evaluated per page over the survivors of the per-record pass.

CastOnErrortransform-cast

Failure policy for RecordTransform::Cast. Default: Error.

CastTypetransform-cast

Target type for RecordTransform::Cast.

CompareOp

Ordering / equality operator for the compare check.

Compression

Internal post-resolution codec. No Auto variant — call CompressionConfig::resolve.

CompressionConfig

User-facing compression config. Defaults to Auto.

ContractFieldType

Declared type of a contract field.

Credential

A resolved credential produced by an AuthProvider or built from inline auth config. Connectors map this onto their wire protocol (HTTP header, gRPC metadata, …).

DeliveryGuarantee

The end-to-end guarantee a pipeline provides for a given source × sink × config combination.

DeliveryMode

Delivery guarantee for a pipeline run.

Detector

Built-in value detectors. All are conservative (anchored full-string matches; the card detector additionally requires a valid Luhn checksum) so false positives stay rare — masking silently rewrites data, so a false positive is a data-quality bug, not just noise.

DlqReason

Reason a page produced DLQ traffic. Used as a metric label and span attribute; closed-set enum so cardinality stays bounded.

EffectivelyOnceMechanism

The mechanism through which a pipeline achieves effectively-once delivery.

EncryptionAlgorithm

Supported AEAD algorithms. A single variant today; the format byte in the on-disk header leaves room for more without breaking old files.

EnvelopeError

Error returned by unwrap_envelope when a value is not a usable DLQ envelope. Only the payload is mandatory — every other field is optional so envelopes written by older versions still replay.

FaucetError

All possible errors returned by faucet-stream.

FilterOptransform-filter

Comparison operator for FilterSpec.

HashAlgorithmtransform-hash

Hash algorithm for RecordTransform::Hash.

HashEncodingtransform-hash

Output encoding for RecordTransform::Hash digests.

InstallError

JoinMode

Join mode — how probe records with no build match are handled.

JsonParseOnErrortransform-json-parse

Failure policy for RecordTransform::JsonParse. Default: Keep.

JsonType

Expected JSON type for the type_is check.

KeyCaseModetransform-keys-case

Output convention for RecordTransform::KeysCase.

KeyNormalize

How to normalize keys before comparison.

MaskAction

What to do with a value once a rule matches it.

NodeKind

A typed topology node.

OnBatchError

Policy applied when a sink reports an outer failure (the whole batch failed, no per-row info).

OnBreach

What to do when a record breaches the contract. Contract-level (not per-field): a contract is a single versioned promise, so one policy governs every rule in it.

OnCollision

What to do when a projected as name collides with an existing field on the probe record.

OnDrift

What to do when drift is detected.

OnDuplicate

What to do when one probe key matches more than one build record.

OnFailure

What to do when a check fails. The allowed subset is validated per check at compile time (see compile.rs).

OnIncompatible

evolve-only: what to do with a narrowing/incompatible change that can’t be auto-applied.

OnMissingtransform-explode

Behaviour when an ExplodeSpec’s path doesn’t yield a non-empty array. The default is Passthrough because silently dropping records is the worst failure mode for ETL pipelines — surfacing the record unchanged lets downstream stages decide.

OtelProtocol

OTLP transport protocol.

OtelSignal

A telemetry signal that can be exported over OTLP.

PaginationStylesource-rest

Supported pagination strategies.

PoisonAction

What to do with a row that keeps failing after max_row_attempts.

ProbeStatus

Outcome of a single probe.

RecordCheck

A per-record check. Addressed field accepts the filter/explode path subset (bare key, dot.path, $['bracketed']).

RecordTransform

A transformation applied to every record fetched by a source (e.g. the REST source’s RestStream).

ReplayGuarantee

How faithfully a Source replays its record stream when resumed from a bookmark.

ReplicationMethod

Determines how records are replicated from the source.

RetryClass

The transient-failure class an error belongs to. Closed set so it can be a bounded metric label and a retry_on config value.

SinkGuarantee

The strongest delivery guarantee a Sink can uphold.

SqlBaseType

Backend-neutral base column type inferred from a JSON-Schema fragment. Each SQL sink maps these to its concrete keyword.

TopologyOnError

What to do when a node fails.

TransformStage

One stage in a transform pipeline.

Value

Represents any valid JSON value.

ValueCaseModetransform-value-case

String-value casing mode for RecordTransform::ValueCase.

WriteMode

Write semantics for a sink. Serialized snake_case. Default Append.

Constants

DEFAULT_BATCH_SIZE

Default page size used when a caller does not specify one.

DEFAULT_EXPIRY_RATIOsource-rest

Default fraction of expires_in after which the token is refreshed.

DEFAULT_TOKEN_ENDPOINT_EXPIRY_RATIOsource-rest

Default fraction of expires_in after which the token is refreshed.

MAX_BATCH_SIZE

Hard upper bound on batch_size. Values above this (other than the special 0 “no batching” sentinel) are rejected at config validation time to prevent accidental O(total) buffering in the default implementation of Source::stream_pages.

Traits

AuthProvider

A live, shareable source of credentials.

IntegrityCheck

A check fed the raw bytes of an object body as they stream past, and asked to validate once the underlying reader reaches EOF.

JsonSchema

A type which can be described as a JSON Schema document.

Sink

A sink writes records to an external system.

Source

A source fetches records from an external system.

StateStore

Persistent key/value store for replication bookmarks and pipeline checkpoints.

Stream

A stream of values produced asynchronously.

Functions

adds_null

True when to permits null but from does not — the column needs its NOT NULL constraint relaxed.

apply_contract

Apply the contract to one page. Pure: no metrics, no DLQ I/O.

apply_masking

Apply the masking policy to one page. Pure: no metrics, no I/O, infallible.

apply_quality

Apply the full per-page quality pass. Pure: no metrics, no DLQ I/O.

base_widened

True when the non-null base type family changes from from to to (after the integer→number collapse) — i.e. the column needs an ALTER TYPE.

build_envelope

Build a single DLQ envelope.

classify

Map a FaucetError to its RetryClass, or None if it is not a transient failure. The single source of truth for retriability — both the policy runner and FaucetError::is_retriable defer to this.

columns_to_schema

Assemble an infer_schema-shaped object schema from (column_name, type_fragment) pairs, preserving input order semantics (serde_json map ordering applies).

compile_stage

Compile a TransformStage into its CompiledStage form.

compress_buf

One-shot in-memory compression. Used by S3 and GCS sinks that build a full Vec<u8> body before upload.

derive_delivery_guarantee

Derive the end-to-end DeliveryGuarantee a pipeline actually provides.

execute_with_policy

Execute op under policy. Retries retriable errors (per RetryPolicy::is_retriable) up to max_attempts total attempts, sleeping the backoff delay between attempts. If cancel is provided, a cancellation during a backoff sleep stops waiting and returns the last error immediately (the caller observes the token and flushes). Ok returns at once.

execute_with_policy_metered

Like execute_with_policy, but emits the resilience metrics (faucet_resilience_retries_total{op,class}, faucet_resilience_retry_sleep_seconds{op}, and faucet_resilience_giveup_total{op} when retries are exhausted) using the labels in metrics. Behavior (retry/give-up/cancel outcomes) is identical to execute_with_policy — only the metric emission differs.

execute_with_retry

Execute operation with up to max_retries retries on retriable errors, using exponential backoff (base_backoff * 2^attempt) with random jitter. Non-retriable errors return immediately; Ok returns immediately.

fetch_oauth2_tokensource-rest

Fetch an OAuth2 token using the client credentials grant.

fetch_token_from_endpointsource-rest

Fetch a token from the given endpoint and extract it using JSONPath.

format_token

Render a page sequence as a fixed-width, lexicographically-ordered token.

format_token_with_bookmark

Render a commit token that carries the page’s resume bookmark alongside the sequence: "{seq:020}#{bookmark-json}".

infer_arrow_schema

Infer an Arrow Schema from a slice of JSON records (each a JSON object).

install_observabilityNon-observability-install

Non-observability-install stub. Returns an empty report, never panics.

instrumented_apply_contract

Apply the contract pass and emit metrics. Returns the same outcome as apply_contract; on a fail-policy breach, increments faucet_contract_aborts_total before propagating the error.

instrumented_apply_masking

Apply the masking pass and emit metrics. Infallible — masking rewrites matching fields in place and never fails a run. Increments faucet_masking_fields_total{pipeline,row,rule,action,detector} once per masked field (detector is "" for name-based matches).

instrumented_apply_quality

Apply the quality pass and emit metrics. Returns the same outcome as apply_quality; on abort, increments faucet_quality_aborts_total before propagating the error.

instrumented_apply_stages

Apply a sequence of compiled stages to records (a full page). Per-record stages flat-map over each record; PageFn stages transform the whole page at once. Emits one faucet.transform.apply span and counters faucet_transform_records_in_total / _records_out_total per call (per page).

json_gt

Total-order “greater than” over JSON values, using the same comparison filter_incremental applies to replication keys (numbers numerically, strings lexicographically — so RFC3339 timestamps order correctly). Public so callers bounding a replay window (e.g. faucet backfill --to-bookmark) compare exactly like the incremental filter does.

json_schema_base_type

Map a top-level JSON-Schema type fragment to a base SQL type, or None for a pure-null fragment (caller falls back to TEXT/NVARCHAR for an added column whose only observed value was null).

key_to_doc_id

Render a key tuple into a single document id (Elasticsearch _id).

key_to_filter

Build a Mongo/ES filter document { col: value, … } from a key tuple.

nullable_type

Wrap a type fragment as nullable ({"type": ["T", "null"]}), matching the nullable shape infer_schema emits.

parse_token

Parse the numeric sequence from a token produced by format_token or format_token_with_bookmark. Returns None on garbage.

parse_token_parts

Parse a stored commit token into (seq, embedded_bookmark).

plan_writes

Partition page into upserts + deletes per spec. The single place all six sinks share. WriteMode::Append should never reach here (callers route append separately); if it does, every row is treated as an upsert.

record_batch_to_values

Decode a RecordBatch into JSON objects (one per row).

redact_uri_credentials

Strip credentials from a connection string so it can be used as a lineage dataset URI without leaking secrets. Handles two shapes, best-effort:

register_build_info

Register the faucet_build_info{version} gauge (set to 1) under the currently-installed metrics recorder. Safe to call from any code path that wants to ensure the gauge is set; install_observability invokes this automatically. Gauges are naturally idempotent under the metrics model — repeat calls just re-set the same value.

run_stream

Run a streaming pipeline, writing each StreamPage to the sink as it arrives and persisting bookmarks per page.

shutdown_otelNon-otel

No-op shutdown_otel when the otel feature is disabled, so CLI call sites compile in every build.

sql_type_to_json_schema

Map a SQL catalog type name (as reported by information_schema.columns or an equivalent) to a JSON-Schema type fragment matching the shape infer_schema produces.

to_json_schema

Render the contract as a standalone JSON Schema (draft 2020-12) document.

to_openlineage_facet

Render the contract as an OpenLineage SchemaDatasetFacet — the same facet shape faucet-lineage attaches to dataset events, so downstream OpenLineage consumers can ingest the contract as a schema promise.

unwrap_envelope

Parse a DLQ envelope produced by build_envelope back into its original payload plus metadata.

unwrap_state

Unwrap a stored state value into (bookmark, seq).

update_bookmark_lag

Update the bookmark-lag gauges if the bookmark is a parseable RFC3339 timestamp. Returns true if the gauges were updated, false otherwise.

validate_batch_size

Validate a batch_size value against the global constraints.

values_to_record_batch

Encode a slice of JSON records into a single RecordBatch against schema.

values_to_record_batch_inferred

Convenience: infer the schema from records and encode them into a batch.

warn_mismatch

Log a one-shot warning when the explicit codec disagrees with the filename’s detected codec. Deduplicates per (path, declared) pair across the whole process so a million-object scan does not flood logs.

wrap_state

Wrap a bookmark + sequence into the exactly-once state value.

Type Aliases

RowOutcome

Per-row outcome from Sink::write_batch_partial.

SharedAuthProvider

A shared AuthProvider handle. Cloning it shares the one live provider (and its single token cache) across connectors.

Attribute Macros

async_trait

Derive Macros

JsonSchema

Derive macro for JsonSchema trait.