Crate faucet_stream 

faucet-stream

A declarative, config-driven data pipeline with pluggable source and sink connectors.

📖 Guide, tutorials & cookbook: https://pawansikawat.github.io/faucet-stream/

Feature flags

Feature	Description
source-rest (default)	REST API source with pagination, auth, transforms
source-graphql	GraphQL API source with cursor pagination
source-xml	XML/SOAP API source with XML-to-JSON conversion
source-grpc	gRPC source with dynamic protobuf messages
source-postgres	PostgreSQL query source
source-postgres-cdc	PostgreSQL CDC source (logical replication)
source-mysql	MySQL query source
source-mssql	Microsoft SQL Server query source
source-sqlite	SQLite query source
source-s3	AWS S3 file source
source-mongodb	MongoDB query source
source-mongodb-cdc	MongoDB CDC source (Change Streams)
source-mysql-cdc	MySQL CDC source (binlog replication)
source-redis	Redis source (streams, lists, keys)
source-webhook	Webhook HTTP receiver source
source-websocket	WebSocket streaming source
source-csv	CSV file source
source-elasticsearch	Elasticsearch search/scroll source
source-kafka	Apache Kafka consumer source
source-kinesis	AWS Kinesis Data Streams source
source-spanner	Google Cloud Spanner query source
source-parquet	Apache Parquet file source (local, glob, S3)
source-delta	Apache Delta Lake source (local FS or S3/Azure/GCS, time travel)
source-databricks	Databricks SQL query source (Statement Execution API)
sink-bigquery	Google BigQuery streaming insert sink
sink-iceberg	Apache Iceberg sink (append-only, REST/Glue/SQL/HMS catalogs)
sink-postgres	PostgreSQL sink (jsonb or auto-mapped columns)
sink-jsonl	JSON Lines file sink
sink-snowflake	Snowflake SQL REST API sink
sink-mysql	MySQL sink
sink-mssql	Microsoft SQL Server sink
sink-sqlite	SQLite sink
sink-s3	AWS S3 file sink
sink-mongodb	MongoDB insert sink
sink-redis	Redis sink (streams, lists, key-value)
sink-csv	CSV file sink
sink-elasticsearch	Elasticsearch bulk index sink
sink-http	HTTP POST sink
sink-kafka	Apache Kafka producer sink
sink-kinesis	AWS Kinesis Data Streams sink
sink-spanner	Google Cloud Spanner mutation sink
sink-parquet	Apache Parquet file sink (local, S3)
encryption	AES-256-GCM at-rest sealing for file state-store bookmarks and per-line JSONL/DLQ output
sink-delta	Apache Delta Lake sink (append-only; local FS or S3/Azure/GCS)
kafka-schema-registry	Schema Registry support for Kafka connectors
source	All source connectors
sink	All sink connectors
full	Every connector

Re-exports

pub use faucet_lineage as lineage;lineage

pub use faucet_transform_sql as transform_sql;transform-sql

Modules

adaptive

Adaptive batch sizing — an AIMD controller that auto-tunes the effective write batch size per pipeline row from observed sink latency + error rate. Pure logic (no I/O); run_stream feeds it observations and emits metrics. See docs/superpowers/specs/2026-05-31-adaptive-batch-sizing-design.md.

async_stream

Asynchronous stream of elements.

authauth

Single-flight OAuth2 / token-endpoint auth providers (enable the auth feature). Share one across connectors via with_auth_provider or the CLI auth: { ref } catalog.

check

Preflight check types for faucet doctor (#126).

common_gcssink-gcs or source-gcs

common_kafkasink-kafka or source-kafka

common_kinesissink-kinesis or source-kinesis

Shared AWS Kinesis types (credentials enum, client builder), re-exported for library callers when either Kinesis connector is enabled.

common_spannersink-spanner or source-spanner

Shared Cloud Spanner types (credentials enum, connection block, value conversion), re-exported for library callers when either Spanner connector is enabled.

compressioncompression

Transparent gzip / zstd compression wrappers for file-shaped connectors.

config

Configuration loading utilities.

contractcontract

Data contracts (issue #204): a declarative, versioned schema + semantic contract for a pipeline’s output — required fields, types, nullability, enum sets, regex patterns, numeric/length bounds — enforced per page after transforms and quality checks and before the sink write. Breaches fail, quarantine (via the DLQ), or warn per the contract-level policy.

discover

Live source introspection (faucet discover, #211).

dlq

Dead-letter queue (DLQ) wiring shared by the pipeline runner.

drift

Schema-drift detection + policy types (issue #194).

encryptionencryption

Encryption at rest for faucet-managed local files (#207).

error

Error types for faucet-stream.

futures_core

Core traits and types for asynchronous operations in Rust.

idempotency

Exactly-once / idempotent delivery primitives.

maskingmasking

PII detection + column-level masking policies (issue #206): a declarative, destination-scoped policy that classifies sensitive fields — by field-name pattern, by value detector (email / card-with-Luhn / SSN / phone / IPv4), or by explicit field list — and rewrites them per action (redact / hash / tokenize / partial).

observability

Pipeline-internal observability: tracing spans and metrics counters/ histograms wired automatically around every source, sink, transform, and state-store operation. See docs/superpowers/specs/2026-05-23-observability-otel-prometheus-design.md.

pipeline

Source-to-sink pipeline orchestration.

qualityquality

Data-quality checks: declarative per-record and per-batch assertions that quarantine violating records to the DLQ or abort the run. Pure evaluation; the pipeline wires the DLQ routing in run_stream.

replication

Incremental replication support.

resilience

Unified resilience policy: retry, backoff classification, circuit breaker, and poison-pill row handling. See docs/superpowers/specs/2026-06-17-resilience-policy-design.md.

retry

Shared exponential-backoff retry executor for HTTP-style sources.

schema

JSON Schema inference from record samples.

schemars

Schemars

shard

Source sharding for clustered (Mode B) execution.

sink

sourcesource-rest

stage

Pipeline-level transform stages. A TransformStage wraps one of six shapes:

state

traits

Shared traits for faucet sources and sinks.

transform

Record transformation pipeline.

transforming_source

Wrap any Source with a fixed list of TransformStages applied to every emitted record. The canonical way for library callers to attach stages (transforms wrapped via TransformStage::Map, plus Filter / Explode / Custom); the CLI uses this same type internally.

util

Shared utilities used across faucet source and sink crates.

verify

Read-time integrity verification for streaming object bodies.

write_mode

Unified write-mode types + planner shared by every upsert-capable sink.

Macros

json

Construct a serde_json::Value from a JSON literal.

schema_for

Generates a Schema for the given type using default settings. The default settings currently conform to JSON Schema 2020-12, but this is liable to change in a future version of Schemars if support for other JSON Schema versions is added.

Structs

AdaptiveBatchConfig

Configuration for the adaptive batch-size controller. Lives under execution.adaptive_batch_size. Default enabled = false (opt-in); when disabled the pipeline writes each page exactly as before.

Adjustment

A size change the controller decided on.

AimdController

Additive-increase / multiplicative-decrease controller. Pure + deterministic.

AuthReference

A { ref: <name> } pointer to a named provider in the top-level auth: catalog. The only permitted key is ref.

CancellationToken

A token which can be used to signal a cancellation request to one or more tasks.

CdcUnwrapSpectransform-cdc-unwrap

Spec for TransformStage::CdcUnwrap. Normalizes a CDC change-event envelope ({op, before, after, …}) into a flat row plus a marker field, so a downstream upsert sink never needs to understand CDC. A 1→0|1 stage: DDL/truncate events (and non-delete events with no after image) are dropped.

CheckContext

Inputs a probe may need. The doctor command enforces timeout on the whole check() call; connectors may also use it to bound their own client calls.

CheckReport

A connector’s full preflight report.

CheckTally

Per-check counters + elapsed time, keyed by check name. Emitted as metrics by the observability wrapper.

CircuitBreaker

Tracks consecutive failures; opens when threshold is reached.

CircuitBreakerConfig

Circuit-breaker tuning. Counts consecutive page-level write failures.

ColumnChange

One column’s drift, expressed in JSON-Schema type-fragment terms (e.g. {"type":"integer"} or {"type":["string","null"]}).

CompiledContract

A fully compiled contract. Built once via CompiledContract::compile.

CompiledEncryption

A compiled, validated encryption policy: the derived write key plus every derived read key (write key first).

CompiledMasking

A masking policy compiled and ready to apply to pages.

CompiledQuality

A fully compiled quality spec. Built once via CompiledQuality::compile.

ContractOutcome

Result of applying the contract pass to one page.

ContractSpec

The contract: config block: a declarative, versioned schema + semantic contract for a pipeline’s output, enforced per page after transforms and quality checks and before the sink write.

ContractViolation

One observed contract breach.

DatasetDescriptor

One dataset discovered behind a source’s connection.

DeleteMarker

Identifies a record as a delete (vs. an upsert) by a marker field’s value. e.g. { field: "__op", values: ["d", "delete"] }.

DlqConfig

Pipeline-level DLQ wiring.

DlqStats

Counters returned alongside PipelineResult when a DLQ is wired.

DurationGuard

On Drop, records the elapsed time since construction into the named histogram with the supplied labels. Recording on drop guarantees a sample even if the surrounding future is cancelled or panics.

EncryptionSpec

The user-facing encryption: config block (state store / jsonl sink).

ExplodeSpectransform-explode

Spec for TransformStage::Explode. Fans one record out into N records based on the array at path. Compiled into a CompiledExplode at pipeline-build time; per-record work is one path resolve + N clones + N merges.

FieldContract

One promised top-level field: its type plus optional semantic constraints.

FileStateStore

File-backed StateStore. Each key maps to a JSON file at {root}/{safe_filename(key)}.json, written via atomic rename. The filename stem percent-encodes : as %3A so keys using the pipeline:rest:issues convention are valid on Windows.

FilterSpectransform-filter

Predicate spec for TransformStage::Filter. Compiled into a CompiledFilter at pipeline-build time; per-record work is a single path resolve + comparison.

GuaranteeInputs

Inputs to derive_delivery_guarantee — the facts about a concrete source × sink × config combination the derivation keys off.

InstallReport

Report from install_observability so callers can log what actually happened (recorder installed vs. already-installed vs. disabled).

InstrumentedSink

Wraps a &dyn Sink (or any &S: Sink) and emits spans + metrics around write_batch and flush. Constructed by Pipeline::run.

InstrumentedSource

Wraps a &dyn Source (or any &S: Source) and emits spans + metrics around every call. Constructed by Pipeline::run and never exposed to end users; the wrapped source remains the user-facing object.

InstrumentedStateStore

Wraps an Arc<dyn StateStore> and emits faucet.state.* spans + metrics around every operation.

KeyTuple

Ordered key column → value pairs, in key declaration order.

Labels

Common labels carried by every span and metric.

LengthCheck

Built-in IntegrityCheck that fails when the number of bytes read does not match the length advertised by the object store (Content-Length for S3, object size for GCS). Catches a cleanly-truncated transfer that would otherwise be accepted as a complete object.

MaskHit

One field-masking event, for metrics/observability.

MaskRule

One masking rule: a matcher + the action to apply to matching fields, optionally scoped to a subset of destination sinks.

MaskingOutcome

Result of masking one page.

MaskingSpec

The masking: config block: a declarative, destination-scoped policy that classifies sensitive fields (by field-name pattern, by value detector, or by explicit field list) and applies a masking action per page. The masking pass runs first — before the quality, contract, and schema-drift passes and before every sink write — so PII never reaches a sink, the DLQ, or a lineage sample unmasked.

MatchSpec

How a rule classifies a field as sensitive. Any combination is allowed; a field matches the rule if any configured criterion matches (name pattern OR value detector OR explicit field name).

MemoryStateStore

In-memory StateStore for tests and ephemeral pipelines.

ObservabilityConfig

Configuration for install_observability. Either or both sections may be None; unset sections install nothing.

Observation

One observed sub-batch write outcome fed to the controller.

OtelConfig

Pure-data OTLP export configuration. Contains no opentelemetry types.

Pipeline

A pipeline that moves data from a Source to a Sink.

PipelineResult

Result of a pipeline run.

PoisonPolicy

Poison-pill (per-row) policy.

Probe

One named probe within a CheckReport (e.g. "read", "auth", "network", "permissions", "schema", "io", "sentinel").

PrometheusConfig

QualityOutcome

Result of applying the quality pass to one page.

QualitySpec

The quality: config block. Per-record checks run first (partitioning the page into survivors + quarantined); per-batch checks then run over the survivors.

QuarantinedRecord

A record removed from the page by a quality check, destined for the DLQ.

ResiliencePolicy

The umbrella policy attached to a run.

ResponseValidatorsource-rest

Optional callback to decide whether the token endpoint response is successful.

RestStreamsource-rest

A configured REST API stream that handles pagination, auth, and extraction.

RestStreamConfigsource-rest

Configuration for a RestStream.

RetryClassSet

The set of classes the user has opted into retrying. Backed by a fixed-size bool array (only four variants), so it is Copy and allocation-free.

RetryMetrics

Per-op labels for the metered runner. Identifies which pipeline / matrix row / I/O operation a retry belongs to so the resilience metrics (faucet_resilience_retries_total / _retry_sleep_seconds / _giveup_total) carry the spec’s {pipeline, row, op} label set. op is a closed set (sink_write / flush / state_put / source).

RetryPolicy

Retry configuration shared by the pipeline loop and source connectors.

RunStreamOptions

SchemaDiff

Result of diffing a page’s inferred shape against the destination schema.

SchemaDriftPolicy

Compiled, ready-to-run drift policy. Cheap to clone.

SchemaDriftSpec

User-facing schema: config block (pipeline level).

SchemaEvolution

The applyable subset of a SchemaDiff handed to crate::Sink::evolve_schema. Never carries incompatible columns — those are routed by the policy.

ShardSpec

One independently-processable slice of a shardable source.

StreamPage

One page emitted by Source::stream_pages.

TracingConfig

TransformingSource

Source decorator that applies a fixed list of compiled stages to every record. Emits faucet_transform_* metrics per page via instrumented_apply_stages.

UnwrappedEnvelope

A DLQ envelope parsed back into its original payload plus the metadata needed to inspect and replay it. Produced by unwrap_envelope.

VerifyingReader

An AsyncRead adapter that observes every byte read from inner and runs the configured IntegrityChecks at EOF. Wrap the raw network reader before any BufReader / decompression layer.

ViolatingRecord

A record removed from the page by a quarantine breach, destined for the DLQ.

WritePlan

The partition of a page by write mode. Infallible to build — per-row failures (missing/null key) land in failed with their original page index so the caller can route them to a DLQ or abort.

WriteSpec

Shared write-mode config, embedded in each upsert-capable sink config via #[serde(flatten)] so write_mode / key / delete_marker appear at the sink-config top level.

Enums

AdjustDirection

Direction of a batch-size adjustment (metric label).

AdjustReason

Why an adjustment happened (metric label).

Authsource-rest

Supported authentication methods.

AuthSpec

A connector’s auth: field: either an inline auth definition A (the { type, config } shape), or a { ref: <name> } reference to a shared provider defined in the top-level auth: catalog.

BackoffKind

How the inter-attempt delay grows.

BatchCheck

A per-batch check, evaluated per page over the survivors of the per-record pass.

CastOnErrortransform-cast

Failure policy for RecordTransform::Cast. Default: Error.

CastTypetransform-cast

Target type for RecordTransform::Cast.

CompareOp

Ordering / equality operator for the compare check.

Compression

Internal post-resolution codec. No Auto variant — call CompressionConfig::resolve.

CompressionConfig

User-facing compression config. Defaults to Auto.

ContractFieldType

Declared type of a contract field.

Credential

A resolved credential produced by an AuthProvider or built from inline auth config. Connectors map this onto their wire protocol (HTTP header, gRPC metadata, …).

DeliveryGuarantee

The end-to-end guarantee a pipeline provides for a given source × sink × config combination.

DeliveryMode

Delivery guarantee for a pipeline run.

Detector

Built-in value detectors. All are conservative (anchored full-string matches; the card detector additionally requires a valid Luhn checksum) so false positives stay rare — masking silently rewrites data, so a false positive is a data-quality bug, not just noise.

DlqReason

Reason a page produced DLQ traffic. Used as a metric label and span attribute; closed-set enum so cardinality stays bounded.

EffectivelyOnceMechanism

The mechanism through which a pipeline achieves effectively-once delivery.

EncryptionAlgorithm

Supported AEAD algorithms. A single variant today; the format byte in the on-disk header leaves room for more without breaking old files.

EnvelopeError

Error returned by unwrap_envelope when a value is not a usable DLQ envelope. Only the payload is mandatory — every other field is optional so envelopes written by older versions still replay.

FaucetError

All possible errors returned by faucet-stream.

FilterOptransform-filter

Comparison operator for FilterSpec.

InstallError

JsonType

Expected JSON type for the type_is check.

KeyCaseModetransform-keys-case

Output convention for RecordTransform::KeysCase.

MaskAction

What to do with a value once a rule matches it.

OnBatchError

Policy applied when a sink reports an outer failure (the whole batch failed, no per-row info).

OnBreach

What to do when a record breaches the contract. Contract-level (not per-field): a contract is a single versioned promise, so one policy governs every rule in it.

OnDrift

What to do when drift is detected.

OnFailure

What to do when a check fails. The allowed subset is validated per check at compile time (see compile.rs).

OnIncompatible

evolve-only: what to do with a narrowing/incompatible change that can’t be auto-applied.

OnMissingtransform-explode

Behaviour when an ExplodeSpec’s path doesn’t yield a non-empty array. The default is Passthrough because silently dropping records is the worst failure mode for ETL pipelines — surfacing the record unchanged lets downstream stages decide.

OtelProtocol

OTLP transport protocol.

OtelSignal

A telemetry signal that can be exported over OTLP.

PaginationStylesource-rest

Supported pagination strategies.

PoisonAction

What to do with a row that keeps failing after max_row_attempts.

ProbeStatus

Outcome of a single probe.

RecordCheck

A per-record check. Addressed field accepts the filter/explode path subset (bare key, dot.path, $['bracketed']).

RecordTransform

A transformation applied to every record fetched by a source (e.g. the REST source’s RestStream).

ReplayGuarantee

How faithfully a Source replays its record stream when resumed from a bookmark.

ReplicationMethod

Determines how records are replicated from the source.

RetryClass

The transient-failure class an error belongs to. Closed set so it can be a bounded metric label and a retry_on config value.

SinkGuarantee

The strongest delivery guarantee a Sink can uphold.

SqlBaseType

Backend-neutral base column type inferred from a JSON-Schema fragment. Each SQL sink maps these to its concrete keyword.

TransformStage

One stage in a transform pipeline.

Value

Represents any valid JSON value.

ValueCaseModetransform-value-case

String-value casing mode for RecordTransform::ValueCase.

WriteMode

Write semantics for a sink. Serialized snake_case. Default Append.

Constants

DEFAULT_BATCH_SIZE

Default page size used when a caller does not specify one.

DEFAULT_EXPIRY_RATIOsource-rest

Default fraction of expires_in after which the token is refreshed.

DEFAULT_TOKEN_ENDPOINT_EXPIRY_RATIOsource-rest

Default fraction of expires_in after which the token is refreshed.

MAX_BATCH_SIZE

Hard upper bound on batch_size. Values above this (other than the special 0 “no batching” sentinel) are rejected at config validation time to prevent accidental O(total) buffering in the default implementation of Source::stream_pages.

Traits

AuthProvider

A live, shareable source of credentials.

IntegrityCheck

A check fed the raw bytes of an object body as they stream past, and asked to validate once the underlying reader reaches EOF.

JsonSchema

A type which can be described as a JSON Schema document.

Sink

A sink writes records to an external system.

Source

A source fetches records from an external system.

StateStore

Persistent key/value store for replication bookmarks and pipeline checkpoints.

Stream

A stream of values produced asynchronously.

Functions

adds_null

True when to permits null but from does not — the column needs its NOT NULL constraint relaxed.

apply_contract

Apply the contract to one page. Pure: no metrics, no DLQ I/O.

apply_masking

Apply the masking policy to one page. Pure: no metrics, no I/O, infallible.

apply_quality

Apply the full per-page quality pass. Pure: no metrics, no DLQ I/O.

base_widened

True when the non-null base type family changes from from to to (after the integer→number collapse) — i.e. the column needs an ALTER TYPE.

build_envelope

Build a single DLQ envelope.

classify

Map a FaucetError to its RetryClass, or None if it is not a transient failure. The single source of truth for retriability — both the policy runner and FaucetError::is_retriable defer to this.

columns_to_schema

Assemble an infer_schema-shaped object schema from (column_name, type_fragment) pairs, preserving input order semantics (serde_json map ordering applies).

compile_stage

Compile a TransformStage into its CompiledStage form.

compress_buf

One-shot in-memory compression. Used by S3 and GCS sinks that build a full Vec<u8> body before upload.

derive_delivery_guarantee

Derive the end-to-end DeliveryGuarantee a pipeline actually provides.

execute_with_policy

Execute op under policy. Retries retriable errors (per RetryPolicy::is_retriable) up to max_attempts total attempts, sleeping the backoff delay between attempts. If cancel is provided, a cancellation during a backoff sleep stops waiting and returns the last error immediately (the caller observes the token and flushes). Ok returns at once.

execute_with_policy_metered

Like execute_with_policy, but emits the resilience metrics (faucet_resilience_retries_total{op,class}, faucet_resilience_retry_sleep_seconds{op}, and faucet_resilience_giveup_total{op} when retries are exhausted) using the labels in metrics. Behavior (retry/give-up/cancel outcomes) is identical to execute_with_policy — only the metric emission differs.

execute_with_retry

Execute operation with up to max_retries retries on retriable errors, using exponential backoff (base_backoff * 2^attempt) with random jitter. Non-retriable errors return immediately; Ok returns immediately.

fetch_oauth2_tokensource-rest

Fetch an OAuth2 token using the client credentials grant.

fetch_token_from_endpointsource-rest

Fetch a token from the given endpoint and extract it using JSONPath.

format_token

Render a page sequence as a fixed-width, lexicographically-ordered token.

format_token_with_bookmark

Render a commit token that carries the page’s resume bookmark alongside the sequence: "{seq:020}#{bookmark-json}".

install_observabilityNon-observability-install

Non-observability-install stub. Returns an empty report, never panics.

instrumented_apply_contract

Apply the contract pass and emit metrics. Returns the same outcome as apply_contract; on a fail-policy breach, increments faucet_contract_aborts_total before propagating the error.

instrumented_apply_masking

Apply the masking pass and emit metrics. Infallible — masking rewrites matching fields in place and never fails a run. Increments faucet_masking_fields_total{pipeline,row,rule,action,detector} once per masked field (detector is "" for name-based matches).

instrumented_apply_quality

Apply the quality pass and emit metrics. Returns the same outcome as apply_quality; on abort, increments faucet_quality_aborts_total before propagating the error.

instrumented_apply_stages

Apply a sequence of compiled stages to records (a full page). Per-record stages flat-map over each record; PageFn stages transform the whole page at once. Emits one faucet.transform.apply span and counters faucet_transform_records_in_total / _records_out_total per call (per page).

json_gt

Total-order “greater than” over JSON values, using the same comparison filter_incremental applies to replication keys (numbers numerically, strings lexicographically — so RFC3339 timestamps order correctly). Public so callers bounding a replay window (e.g. faucet backfill --to-bookmark) compare exactly like the incremental filter does.

json_schema_base_type

Map a top-level JSON-Schema type fragment to a base SQL type, or None for a pure-null fragment (caller falls back to TEXT/NVARCHAR for an added column whose only observed value was null).

key_to_doc_id

Render a key tuple into a single document id (Elasticsearch _id).

key_to_filter

Build a Mongo/ES filter document { col: value, … } from a key tuple.

nullable_type

Wrap a type fragment as nullable ({"type": ["T", "null"]}), matching the nullable shape infer_schema emits.

parse_token

Parse the numeric sequence from a token produced by format_token or format_token_with_bookmark. Returns None on garbage.

parse_token_parts

Parse a stored commit token into (seq, embedded_bookmark).

plan_writes

Partition page into upserts + deletes per spec. The single place all six sinks share. WriteMode::Append should never reach here (callers route append separately); if it does, every row is treated as an upsert.

redact_uri_credentials

Strip credentials from a connection string so it can be used as a lineage dataset URI without leaking secrets. Handles two shapes, best-effort:

register_build_info

Register the faucet_build_info{version} gauge (set to 1) under the currently-installed metrics recorder. Safe to call from any code path that wants to ensure the gauge is set; install_observability invokes this automatically. Gauges are naturally idempotent under the metrics model — repeat calls just re-set the same value.

run_stream

Run a streaming pipeline, writing each StreamPage to the sink as it arrives and persisting bookmarks per page.

shutdown_otelNon-otel

No-op shutdown_otel when the otel feature is disabled, so CLI call sites compile in every build.

sql_type_to_json_schema

Map a SQL catalog type name (as reported by information_schema.columns or an equivalent) to a JSON-Schema type fragment matching the shape infer_schema produces.

to_json_schema

Render the contract as a standalone JSON Schema (draft 2020-12) document.

to_openlineage_facet

Render the contract as an OpenLineage SchemaDatasetFacet — the same facet shape faucet-lineage attaches to dataset events, so downstream OpenLineage consumers can ingest the contract as a schema promise.

unwrap_envelope

Parse a DLQ envelope produced by build_envelope back into its original payload plus metadata.

unwrap_state

Unwrap a stored state value into (bookmark, seq).

update_bookmark_lag

Update the bookmark-lag gauges if the bookmark is a parseable RFC3339 timestamp. Returns true if the gauges were updated, false otherwise.

validate_batch_size

Validate a batch_size value against the global constraints.

warn_mismatch

Log a one-shot warning when the explicit codec disagrees with the filename’s detected codec. Deduplicates per (path, declared) pair across the whole process so a million-object scan does not flood logs.

wrap_state

Wrap a bookmark + sequence into the exactly-once state value.

Type Aliases

RowOutcome

Per-row outcome from Sink::write_batch_partial.

SharedAuthProvider

A shared AuthProvider handle. Cloning it shares the one live provider (and its single token cache) across connectors.

Attribute Macros

async_trait

Derive Macros

JsonSchema

Derive macro for JsonSchema trait.