fraiseql-wire

Streaming JSON queries for Postgres 17, built for FraiseQL

fraiseql-wire is a minimal, async Rust query engine that streams JSON data from Postgres with low latency and bounded memory usage.

It is not a general-purpose Postgres driver. It is a focused, purpose-built transport for JSON queries of the form:

SELECT data
FROM {source}
[WHERE predicate]
[ORDER BY expression [COLLATE collation] [ASC|DESC]]
[LIMIT N] [OFFSET M]

Where {source} is a JSON-shaped relation (v_{entity} views or tv_{entity} tables).

The primary goal is to enable efficient, backpressure-aware streaming of JSON from Postgres into Rust, with support for hybrid filtering (SQL + Rust predicates), adaptive chunking, pause/resume flow control, and comprehensive metrics.

Why fraiseql-wire?

Traditional database drivers are optimized for flexibility and completeness. FraiseQL-Wire is optimized for:

🚀 Low latency (process rows as soon as they arrive)
🧠 Low memory usage (no full result buffering)
🔁 Streaming-first APIs (Stream<Item = Result<Value, _>>)
🧩 Hybrid filtering (SQL + Rust predicates)
🔍 JSON-native workloads

If your application primarily:

Reads JSON (json / jsonb)
Uses views as an abstraction layer
Needs to process large result sets incrementally

…then fraiseql-wire is a good fit.

Non-goals

fraiseql-wire intentionally does not support:

Writes (INSERT, UPDATE, DELETE)
Transactions
Prepared statements
Arbitrary SQL
Multi-column result sets
Full Postgres type decoding

If you need those features, use tokio-postgres or sqlx.

Supported Query Shape

All queries must conform to:

SELECT data
FROM {source}
[WHERE <predicate>]
[ORDER BY <expression> [COLLATE <collation>] [ASC|DESC]]
[LIMIT <count>]
[OFFSET <count>]

Query Components

Component	Support	Notes
SELECT	`SELECT data` only	Result column must be named `data` and type `json`/`jsonb`
FROM	`v_{entity}` / `tv_{entity}`	Views and tables with JSON column
WHERE	SQL predicates	Optional; use `where_sql()` in builder
ORDER BY	Server-side sorting	With optional COLLATE; server-executed, no client buffering
LIMIT/OFFSET	Pagination	For result set reduction
Filtering	SQL + Rust predicates	Hybrid: SQL reduces wire traffic, Rust refines streamed data

Hard Constraints

Exactly one column in result set (named data)
Column type must be json or jsonb
Results streamed in-order (server-side ordering for ORDER BY)
One active query per connection
No client-side reordering or aggregation

Example

Streaming JSON results

use futures::StreamExt;

let client = FraiseClient::connect("postgres:///example").await?;

let mut stream = client
    .query("user")
    .where_sql("data->>'status' = 'active'")
    .chunk_size(256)
    .execute()
    .await?;

while let Some(item) = stream.next().await {
    let json = item?;
    println!("{json}");
}

Collecting (optional)

let users: Vec<serde_json::Value> =
    stream.collect::<Result<_, _>>()?;

Hybrid Predicates (SQL + Rust)

Not all predicates belong in SQL. FraiseQL-Wire supports hybrid filtering:

let stream = client
    .query("user")
    .where_sql("data->>'type' = 'customer'")
    .where_rust(|json| expensive_check(json))
    .execute()
    .await?;

SQL predicates reduce data sent over the wire
Rust predicates allow expressive, application-level filtering
Filtering happens while streaming

Streaming Model

Under the hood:

Results are read incrementally from the Postgres socket
Rows are batched into small chunks
Chunks are sent through a bounded async channel
Consumers apply backpressure naturally via .await

This ensures:

Bounded memory usage
CPU and I/O overlap
Fast time-to-first-row

Cancellation & Drop Semantics

If the stream is dropped early:

The in-flight query is cancelled
The connection is closed
Background tasks are terminated

This prevents runaway queries and resource leaks.

Postgres 17 & Chunked Rows Mode

fraiseql-wire is designed to take advantage of Postgres 17 streaming behavior, and can optionally leverage chunked rows mode via a libpq-based backend.

The public API remains the same regardless of backend; chunking is an internal optimization.

Quick Start

Installation

Add to Cargo.toml:

[dependencies]
fraiseql-wire = "0.1"
tokio = { version = "1", features = ["full"] }
futures = "0.3"
serde_json = "1"

Basic Usage

use fraiseql_wire::client::FraiseClient;
use futures::stream::StreamExt;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Connect to Postgres
    let client = FraiseClient::connect("postgres://localhost/mydb").await?;

    // Stream results
    let mut stream = client.query("users").execute().await?;

    while let Some(item) = stream.next().await {
        let json = item?;
        println!("{}", json);
    }

    Ok(())
}

Running Examples

See examples/ directory:

# Start Postgres with test data
docker-compose up -d

# Run examples
cargo run --example basic_query
cargo run --example filtering
cargo run --example ordering
cargo run --example streaming
cargo run --example error_handling

Error Handling

Errors are surfaced as part of the stream:

Stream<Item = Result<serde_json::Value, FraiseError>>

Possible error sources include:

Connection or authentication failures
SQL execution errors
Protocol violations
Invalid result schema
JSON decoding failures
Query cancellation

Fatal errors terminate the stream.

For detailed error diagnosis, see TROUBLESHOOTING.md.

Performance Characteristics

📉 Memory usage scales with chunk_size, not result size
⏱ First rows are available immediately
🔄 Server I/O and client processing overlap
📦 JSON decoding is incremental

Benchmarked Performance (v0.1.0)

Memory Efficiency: The key advantage

Scenario	fraiseql-wire	tokio-postgres	Difference
10K rows	1.3 KB	2.6 MB	2000x
100K rows	1.3 KB	26 MB	20,000x
1M rows	1.3 KB	260 MB	200,000x

fraiseql-wire uses O(chunk_size) memory while traditional drivers use O(result_size).

Latency & Throughput: Comparable to tokio-postgres

Metric	fraiseql-wire	tokio-postgres
Connection setup	~250 ns (CPU)	~250 ns (CPU)
Query parsing	~5-30 µs	~5-30 µs
Throughput	100K-500K rows/sec	100K-500K rows/sec
Time-to-first-row	2-5 ms	2-5 ms

For detailed performance analysis, see PERFORMANCE_TUNING.md and benches/COMPARISON_GUIDE.md.

When to Use fraiseql-wire

Use this crate if you:

Stream large JSON result sets
Want predictable memory usage
Use Postgres views as an API boundary
Prefer async streams over materialized results
Are building FraiseQL or similar query layers

When Not to Use It

Avoid this crate if you need:

Writes or transactions
Arbitrary SQL
Strong typing across many Postgres types
Multi-query sessions
Compatibility with existing ORMs

Advanced Features

Type-Safe Deserialization

Stream results as custom structs instead of raw JSON:

#[derive(Deserialize)]
struct Project {
    id: String,
    name: String,
    status: String,
}

let stream = client.query::<Project>("projects").execute().await?;
while let Some(project) = stream.next().await {
    let p: Project = project?;
    println!("{}: {}", p.id, p.name);
}

Type T affects only deserialization; SQL, filtering, and ordering are identical regardless of T.

Stream Control (Pause/Resume)

Pause and resume streams for advanced flow control:

let mut stream = client.query("entities").execute().await?;

// Process some rows
while let Some(item) = stream.next().await {
    println!("{item?}");
    break;  // Stop after one
}

// Pause to do other work
stream.pause().await?;
// ... perform other operations ...
stream.resume().await?;  // Continue from where we left off

Adaptive Chunking

Automatic chunk size optimization based on channel occupancy:

let stream = client
    .query("large_table")
    .adaptive_chunking(true)    // Enabled by default
    .adaptive_min_size(16)      // Don't go below 16
    .adaptive_max_size(1024)    // Don't exceed 1024
    .execute()
    .await?;

SQL Field Projection

Reduce payload size via database-level field filtering:

let stream = client
    .query("users")
    .select_projection("jsonb_build_object('id', data->>'id', 'name', data->>'name')")
    .execute()
    .await?;
// Returns only id and name fields, reducing network overhead

Metrics & Tracing

Built-in metrics via the metrics crate:

fraiseql_stream_rows_yielded – Total rows yielded from streams
fraiseql_stream_rows_filtered – Rows filtered by predicates
fraiseql_query_duration_ms – Query execution time
fraiseql_memory_usage_bytes – Estimated memory consumption

Enable tracing with:

RUST_LOG=fraiseql_wire=debug cargo run

Project Status

✅ Production Ready

API is stable and well-tested
166+ unit tests, comprehensive integration tests
Zero clippy warnings (strict -D warnings)
Fully optimized streaming engine with proven performance characteristics
Ready for production use

All core features implemented with comprehensive CI validation:

✅ Async JSON streaming (integration tests across PostgreSQL 15-18)
✅ Hybrid SQL + Rust predicates (25+ WHERE operators with full test coverage)
✅ Type-safe deserialization (generic streaming API with custom struct support)
✅ Stream pause/resume (backpressure-aware flow control)
✅ Adaptive chunking (automatic memory-aware chunk optimization)
✅ SQL field projection (SELECT clause optimization for reduced payload)
✅ Server-side ordering (ORDER BY with COLLATE support, no client buffering)
✅ Pagination (LIMIT/OFFSET for result set reduction)
✅ Metrics & tracing (comprehensive observability via metrics crate)
✅ Error handling (detailed error types and recovery patterns)
✅ Connection pooling support (documented integration patterns)
✅ TLS/SCRAM authentication (PostgreSQL 17+ security features)

Roadmap

Connection pooling integration guide (CONNECTION_POOLING.md)
Advanced filtering patterns (ADVANCED_FILTERING.md)
PostgreSQL 15-18 compatibility (POSTGRES_COMPATIBILITY.md)
SCRAM/TLS end-to-end integration tests in CI
Comprehensive metrics and tracing
Server-side ordering (ORDER BY with COLLATE)
Pagination support (LIMIT/OFFSET)
SQL field projection for payload optimization
Extended metric examples and dashboards
Performance tuning guide for large datasets
PostgreSQL 19+ compatibility tracking
Binary protocol optimization (extended query protocol)

Documentation & Guides

QUICK_START.md – Installation and first steps
TESTING_GUIDE.md – How to run unit, integration, and load tests
TROUBLESHOOTING.md – Error diagnosis and common issues
CI_CD_GUIDE.md – GitHub Actions, local development, releases
PERFORMANCE_TUNING.md – Benchmarking and optimization
CONTRIBUTING.md – Development workflows and architecture
PRD.md – Product requirements and design
.github/PUBLISHING.md – Automatic crates.io publishing setup and workflow

Examples

examples/basic_query.rs – Simple streaming usage
examples/filtering.rs – SQL and Rust predicates
examples/ordering.rs – ORDER BY with collation
examples/streaming.rs – Large result handling and chunk tuning
examples/error_handling.rs – Error handling patterns

Philosophy

This is not a Postgres driver. It is a JSON query pipe.

By narrowing scope, fraiseql-wire delivers performance and clarity that general-purpose drivers cannot.

Credits

Author:

Lionel Hamayon (@evoludigit)

Part of: FraiseQL — Compiled GraphQL for deterministic Postgres execution

License

MIT OR Apache-2.0

fraiseql-wire 2.0.0-beta.2