faucet-stream

The umbrella crate for the faucet-stream ecosystem — a modular, config-driven Rust data-pipeline toolkit. One dependency gives you feature-gated access to every source and sink connector, the state-store backends, shared auth providers, data-quality checks, OpenLineage emission, and the embedded SQL transform — all built on the lightweight faucet-core traits.

This crate is for Rust library users who want to build pipelines in code. If you'd rather drive pipelines declaratively from YAML/JSON with zero Rust, install the faucet-cli binary instead — it wraps these same connectors behind a faucet run pipeline.yaml command.

Why the umbrella vs. individual crates?

• One dependency, one version line. faucet-stream pins every connector to a compatible version so you never hand-align faucet-source-rest, faucet-sink-postgres, etc. Add the features you want and the right crates come along.
• Pay only for what you enable. Every connector is behind an opt-in Cargo feature. The default build pulls in just the REST source, so an unused connector never touches your compile time or binary size.
• Uniform re-exports. Connector types surface under one faucet_stream::* namespace, alongside the full faucet-core API (Source, Sink, Pipeline, run_stream, FaucetError, transforms).

Prefer depending on the individual crates (faucet-core + faucet-source-rest + faucet-sink-postgres, …) when you want the absolute minimum dependency closure or are publishing your own connector crate — see Using individual crates below.

Installation

# Default: REST source only (smallest build)
cargo add faucet-stream

# Pick exactly the connectors you need
cargo add faucet-stream --features source-rest,source-s3,sink-postgres,sink-jsonl

# All sources / all sinks
cargo add faucet-stream --features source
cargo add faucet-stream --features sink

# Everything — every connector, state backend, auth, quality, lineage, SQL transform
cargo add faucet-stream --features full

The default feature set is ["source-rest"]. Every other connector and capability below is opt-in.

Feature-flag matrix

Source connectors

Sink connectors

Iceberg catalog add-ons

State-store backends

The memory and file state stores are always available via faucet-core. These add durable backends for replication bookmarks / exactly-once tokens:

Capabilities

compression uses optional-dependency forwarding (faucet-source-csv?/compression, …): it turns on gzip/zstd only for the file-shaped connectors you have already enabled (source-csv, source-s3, source-gcs, sink-jsonl, sink-csv, sink-s3, sink-gcs). It does not pull connectors you haven't requested.

Transforms

Transforms reshape records mid-pipeline. Enabling source-rest (the default) already turns on transform-flatten, transform-rename-keys, and transform-keys-case.

Aggregate features

Pick your connectors

Common combinations — enable only what your pipeline touches:

# REST API → PostgreSQL ELT
cargo add faucet-stream --features source-rest,sink-postgres

# Data-lake landing: REST → S3 with gzip
cargo add faucet-stream --features source-rest,sink-s3,compression

# S3 → BigQuery warehouse load
cargo add faucet-stream --features source-s3,sink-bigquery

# Postgres CDC → Postgres mirror with exactly-once + durable state
cargo add faucet-stream --features source-postgres-cdc,sink-postgres,state-postgres,transform-cdc-unwrap

# Kafka with Schema Registry → Parquet on S3
cargo add faucet-stream --features source-kafka,sink-parquet,kafka-schema-registry

# Everything, for prototyping
cargo add faucet-stream --features full

Library usage

The whole faucet-core API is re-exported unconditionally; connector types appear when their feature is enabled.

use faucet_stream::{Pipeline, run_stream};
use faucet_source_rest::{RestStream, RestStreamConfig, Auth, PaginationStyle};
use faucet_sink_jsonl::{JsonlSink, JsonlSinkConfig};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Build a source: paginated REST API with bearer auth
    let source = RestStream::new(
        RestStreamConfig::new("https://api.example.com", "/v1/users")
            .auth(Auth::Bearer { token: "my-token".into() })
            .records_path("$.data[*]")
            .pagination(PaginationStyle::Cursor {
                next_token_path: "$.meta.next_cursor".into(),
                param_name: "cursor".into(),
            }),
    )?;

    // Build a sink
    let sink = JsonlSink::new(JsonlSinkConfig::new("users.jsonl"))?;

    // Batch mode: fetch all, then write
    let result = Pipeline::new(&source, &sink).run().await?;
    println!("Wrote {} records", result.records_written);

    // Streaming mode: write each page as it arrives (memory bounded at O(batch_size))
    let _ = run_stream(source.stream_pages(), &sink).await?;
    Ok(())
}

What's re-exported

Always available (from faucet-core):

• Source, Sink traits; Pipeline, PipelineResult, run_stream
• FaucetError; RecordTransform, ReplicationMethod
• config::load_json, config::load_env, config::load_env_file
• the connector-author re-exports — async_trait, serde_json, Value, json!, JsonSchema, schema_for!

Plus the config + connector types from each enabled feature (e.g. RestStream / RestStreamConfig / Auth / PaginationStyle under source-rest). For connector-specific config fields, auth variants, and capability details, see each connector crate's own README and the docs site.

Examples

Runnable examples live in examples/; each declares its required features at the top of the file. The rest_to_jsonl and rest_streaming examples run end-to-end against a public test API; the rest are compile-only source→sink pairings that need real infrastructure to run.

cargo run -p faucet-stream --example rest_to_jsonl --features "source-rest sink-jsonl"
cargo run -p faucet-stream --example postgres_to_bigquery --features "source-postgres sink-bigquery"

Using individual crates

You can depend on the connector crates directly instead of the umbrella for the tightest dependency closure or finer compile-time control:

cargo add faucet-core
cargo add faucet-source-rest
cargo add faucet-sink-postgres

If you are building your own connector, depend only on faucet-core — it is the single required dependency and re-exports everything a faucet-source-* / faucet-sink-* crate needs.

Troubleshooting / FAQ

See also

• Getting started and the library tutorial
• Connector reference & capability matrix
• Choosing connectors
• faucet-core — the traits + pipeline engine (the only dependency connector authors need)
• faucet-cli — drive these connectors from YAML/JSON with no Rust code

License

Licensed under either of MIT or Apache-2.0 at your option.