Skip to main content

Crate faucet_core

Crate faucet_core 

Source
Expand description

§faucet-core

Shared types, traits, and utilities for the faucet-stream ecosystem.

This crate provides the common foundation used by all faucet source and sink connectors:

Re-exports§

pub use adaptive::AdaptiveBatchConfig;
pub use adaptive::AdjustDirection;
pub use adaptive::AdjustReason;
pub use adaptive::Adjustment;
pub use adaptive::AimdController;
pub use adaptive::Observation;
pub use auth::AuthProvider;
pub use auth::AuthReference;
pub use auth::AuthSpec;
pub use auth::Credential;
pub use auth::SharedAuthProvider;
pub use check::CheckContext;
pub use check::CheckReport;
pub use check::Probe;
pub use check::ProbeStatus;
pub use dlq::DlqConfig;
pub use dlq::DlqReason;
pub use dlq::DlqStats;
pub use dlq::OnBatchError;
pub use dlq::build_envelope;
pub use error::FaucetError;
pub use observability::instrumented_apply_quality;quality
pub use observability::DurationGuard;
pub use observability::InstallError;
pub use observability::InstallReport;
pub use observability::InstrumentedSink;
pub use observability::InstrumentedSource;
pub use observability::InstrumentedStateStore;
pub use observability::Labels;
pub use observability::ObservabilityConfig;
pub use observability::PrometheusConfig;
pub use observability::RunStreamOptions;
pub use observability::TracingConfig;
pub use observability::install_observability;
pub use observability::instrumented_apply_stages;
pub use observability::register_build_info;
pub use observability::update_bookmark_lag;
pub use pipeline::DEFAULT_BATCH_SIZE;
pub use pipeline::MAX_BATCH_SIZE;
pub use pipeline::Pipeline;
pub use pipeline::PipelineResult;
pub use pipeline::StreamPage;
pub use pipeline::run_stream;
pub use pipeline::validate_batch_size;
pub use replication::ReplicationMethod;
pub use retry::execute_with_retry;
pub use stage::ExplodeSpec;transform-explode
pub use stage::OnMissing;transform-explode
pub use stage::FilterOp;transform-filter
pub use stage::FilterSpec;transform-filter
pub use stage::TransformStage;
pub use stage::compile_stage;
pub use state::FileStateStore;
pub use state::MemoryStateStore;
pub use state::StateStore;
pub use traits::RowOutcome;
pub use traits::Sink;
pub use traits::Source;
pub use transform::KeyCaseMode;transform-keys-case
pub use transform::RecordTransform;
pub use transform::ValueCaseMode;transform-value-case
pub use transform::CastOnError;transform-cast
pub use transform::CastType;transform-cast
pub use transforming_source::TransformingSource;
pub use compression::Compression;compression
pub use compression::CompressionConfig;compression
pub use compression::compress_buf;compression
pub use compression::warn_mismatch;compression
pub use quality::BatchCheck;quality
pub use quality::CheckTally;quality
pub use quality::CompareOp;quality
pub use quality::CompiledQuality;quality
pub use quality::JsonType;quality
pub use quality::OnFailure;quality
pub use quality::QualityOutcome;quality
pub use quality::QualitySpec;quality
pub use quality::QuarantinedRecord;quality
pub use quality::RecordCheck;quality
pub use quality::apply_quality;quality
pub use async_stream;
pub use futures_core;
pub use schemars;
pub use serde_json;

Modules§

adaptive
Adaptive batch sizing — an AIMD controller that auto-tunes the effective write batch size per pipeline row from observed sink latency + error rate. Pure logic (no I/O); run_stream feeds it observations and emits metrics. See docs/superpowers/specs/2026-05-31-adaptive-batch-sizing-design.md.
auth
Shared, connector-agnostic authentication abstraction.
check
Preflight check types for faucet doctor (#126).
compressioncompression
Transparent gzip / zstd compression wrappers for file-shaped connectors.
config
Configuration loading utilities.
dlq
Dead-letter queue (DLQ) wiring shared by the pipeline runner.
error
Error types for faucet-stream.
observability
Pipeline-internal observability: tracing spans and metrics counters/ histograms wired automatically around every source, sink, transform, and state-store operation. See docs/superpowers/specs/2026-05-23-observability-otel-prometheus-design.md.
pipeline
Source-to-sink pipeline orchestration.
qualityquality
Data-quality checks: declarative per-record and per-batch assertions that quarantine violating records to the DLQ or abort the run. Pure evaluation; the pipeline wires the DLQ routing in run_stream.
replication
Incremental replication support.
retry
Shared exponential-backoff retry executor for HTTP-style sources.
schema
JSON Schema inference from record samples.
stage
Pipeline-level transform stages. A TransformStage wraps one of four shapes:
state
Pluggable state store for incremental replication bookmarks.
traits
Shared traits for faucet sources and sinks.
transform
Record transformation pipeline.
transforming_source
Wrap any Source with a fixed list of TransformStages applied to every emitted record. The canonical way for library callers to attach stages (transforms wrapped via TransformStage::Map, plus Filter / Explode / Custom); the CLI uses this same type internally.
util
Shared utilities used across faucet source and sink crates.

Macros§

json
Construct a serde_json::Value from a JSON literal.
schema_for
Generates a Schema for the given type using default settings. The default settings currently conform to JSON Schema 2020-12, but this is liable to change in a future version of Schemars if support for other JSON Schema versions is added.

Structs§

CancellationToken
Re-exported so callers of Pipeline::with_cancel / RunStreamOptions::with_cancel can name the token type without adding tokio-util themselves. A token which can be used to signal a cancellation request to one or more tasks.

Enums§

Value
Represents any valid JSON value.

Traits§

JsonSchema
A type which can be described as a JSON Schema document.
Stream
A stream of values produced asynchronously.

Attribute Macros§

async_trait

Derive Macros§

JsonSchema
Derive macro for JsonSchema trait.