Wingfoil

Wingfoil is a blazingly fast, highly scalable stream processing framework designed for latency-critical use cases such as electronic trading and real-time AI systems.

It ships with a growing library of production-ready I/O adapters covering tick stores, message buses, market protocols, and observability backends — so you can plug graphs into real data sources and sinks with a single line.

Wingfoil simplifies receiving, processing, distributing and monitoring streaming data across your entire stack.

Features

Fast: Ultra low latency and high throughput with an efficient DAG based execution engine.
Backtesting: Replay historical data to backtest and optimise strategies.
Simple and obvious to use: Define your graph of calculations; Wingfoil manages its execution.
I/O Adapters: production-ready integrations for iceoryx2, KDB+, Kafka, Fluvio, FIX, ZeroMQ, etcd, Prometheus, OpenTelemetry, CSV, and more.
Multi-language: currently available as a Rust crate, python package and a TypeScript/JavaScript client.
Graph dynamism: rewire your graph in response to incoming data.
Async/Tokio: seamless integration, allows you to leverage async at your graph edges.
Multi-threading: distribute graph execution across cores.

Quick Start

In this example we build a simple, linear pipeline with all nodes ticking in lock-step.

use wingfoil::*;
use std::time::Duration;
fn main() {
    let period = Duration::from_secs(1);
    ticker(period)
        .count()
        .map(|i| format!("hello, world {:}", i))
        .print()
        .run(RunMode::RealTime, RunFor::Duration(period*3)
    );
}

This output is produced:

hello, world 1
hello, world 2
hello, world 3

Order Book Example

Wingfoil lets you easily wire up complex business logic, splitting and recombining streams, and modulating the frequency of data. I/O adapters make it easy to plug in real data sources and sinks. In this example we load a CSV of AAPL limit orders, maintain an order book using the lobster crate, derive trades and two-way prices, and export back to CSV — all in a few lines:

let book = RefCell::new(lobster::OrderBook::default());
let get_time = |msg: &Message| NanoTime::new((msg.seconds * 1e9) as u64);
let (fills, prices) = csv_read("aapl.csv", get_time, true)
    .map(move |chunk| process_orders(chunk, &book))
    .split();
let prices_export = prices
    .filter_value(|price: &Option<TwoWayPrice>| !price.is_none())
    .map(|price| price.unwrap())
    .distinct()
    .csv_write("prices.csv");
let fills_export = fills.csv_write("fills.csv");
Graph::new(vec![prices_export, fills_export], RunMode::HistoricalFrom(NanoTime::ZERO), RunFor::Forever)
    .print()
    .run()
    .unwrap();

This output is produced:

Full example.

More Examples

Short code snippets for each adapter live in the examples README. The examples below are all runnable — see each one's README.md for setup and commands.

Core concepts

Example	Description
`order_book`	Load NASDAQ AAPL limit orders from CSV, maintain an order book, derive trades and two-way prices, export to CSV.
`breadth_first`	Why wingfoil's BFS execution avoids the O(2^N) node explosion of naive depth-first DAGs.
`run_mode`	Swap `RunMode::RealTime` and `RunMode::HistoricalFrom` with the same graph wiring for backtesting.
`async`	Integrate Tokio async/await at graph edges (I/O adapters) while keeping the core graph synchronous.
`threading`	Distribute graph execution across worker threads with `producer()` / `mapper()`.
`dynamic`	Add and remove nodes at runtime. Includes `demux`, `dynamic-group`, and `dynamic-manual` variants.
`tracing`	Instrumentation modes (log, tracing, instruments) for event and span handling.
`latency`	Per-hop latency stamping with `Traced<T, L>` and `LatencyReport`, transported over iceoryx2.

I/O adapters

Example	Description
`kdb`	KDB+ integration: time-sliced reads, cached reads (LRU file cache), and round-trip write/read/validate.
`kafka`	Kafka / Redpanda adapter — subscribe, transform, publish pipeline via `rdkafka`.
`fluvio`	Fluvio distributed streaming — subscribe, transform, publish pipeline.
`fix`	FIX 4.4 protocol: self-contained loopback, client, echo server, and live LMAX market data over TLS.
`zmq`	ZeroMQ pub/sub with direct addressing or etcd-based service discovery.
`etcd`	etcd key-value store adapter for sub/pub with transformation.
`iceoryx2`	Zero-copy IPC over shared memory (spin, threaded, signaled polling modes).
`aeron`	Low-latency Aeron UDP/IPC transport — publish and subscribe to `i64` values with spin and threaded polling modes.
`web`	WebSocket adapter streaming synthetic prices and receiving UI events.
`telemetry`	Metrics export via Prometheus scraping (pull) and OpenTelemetry OTLP (push).

Get Involved!

We want to hear from you! Especially if you:

are interested in contributing
know of a project that wingfoil would be well-suited for
would like to request a feature or report a bug
have any feedback

Please do get in touch:

ping us on discord
email us at hello@wingfoil.io
submit an issue
get involved in the discussion

wingfoil 6.0.5