faucet-stream

Umbrella crate for the faucet-stream ecosystem. Provides feature-gated access to all source and sink connectors through a single dependency.

Installation

[dependencies]
# Default: REST source only
faucet-stream = "1.0"

# All sources
faucet-stream = { version = "1.0", features = ["source"] }

# All sinks
faucet-stream = { version = "1.0", features = ["sink"] }

# Everything
faucet-stream = { version = "1.0", features = ["full"] }

# Pick what you need
faucet-stream = { version = "1.0", features = ["source-rest", "source-s3", "sink-postgres", "sink-jsonl"] }

Feature Flags

Source Connectors

Sink Connectors

Aggregate Features

Transform Features

Quick Start

use faucet_stream::{
    RestStream, RestStreamConfig, Auth, PaginationStyle,
    Pipeline,
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Configure a source
    let source = RestStream::new(
        RestStreamConfig::new("https://api.example.com", "/v1/users")
            .auth(Auth::Bearer {
                token: "my-token".into(),
            })
            .records_path("$.data[*]")
            .pagination(PaginationStyle::Cursor {
                next_token_path: "$.meta.next_cursor".into(),
                param_name: "cursor".into(),
            }),
    )?;

    let records = source.fetch_all().await?;
    println!("Fetched {} records", records.len());
    Ok(())
}

Pipeline: Source to Sink

use faucet_stream::{Pipeline, run_stream};

// Batch mode: fetch all, then write
let result = Pipeline::new(&source, &sink).run().await?;
println!("Wrote {} records", result.records_written);

// Streaming mode: write page-by-page (bounded memory)
let result = run_stream(source.stream_pages(), &sink).await?;

Examples

Runnable examples live in examples/. Each one declares its required Cargo features so cargo check --all-targets skips examples whose connectors aren't enabled.

Pipeline shape (these run end-to-end against jsonplaceholder.typicode.com)

Connector matrix (compile-only without external infra)

The remaining examples illustrate popular source→sink pairings. They compile under their feature gates, but actually running them needs the listed infrastructure (DB, S3 bucket, GCP credentials, etc.).

Run any example with the feature flags it documents at the top of the file, e.g.:

cargo run -p faucet-stream --example postgres_to_bigquery \
    --features "source-postgres sink-bigquery"

What's Re-exported

This crate re-exports everything from faucet-core unconditionally:

• Source, Sink traits
• Pipeline, PipelineResult, run_stream
• FaucetError
• RecordTransform, ReplicationMethod
• config::load_json, config::load_env, config::load_env_file
• async_trait, serde_json, Value, json!, JsonSchema, schema_for!

Plus all types from enabled connector features (e.g. RestStream, RestStreamConfig, Auth, PaginationStyle when source-rest is enabled).

Using Individual Crates

You can also depend on connector crates directly instead of using the umbrella:

[dependencies]
faucet-core = "1.0"
faucet-source-rest = "1.0"
faucet-sink-postgres = "1.0"

This gives finer control over dependencies and compile times.

License

Licensed under either of MIT or Apache-2.0 at your option.