Crate lmflow 

A dataflow-graph compute engine.

Computation is described as a directed graph: nodes are kernels, and timestamped packets flow along the edges. This crate is the engine itself — scheduling, worker threads, edge queues, topology validation and YAML parsing.

Two interfaces

• Rust API — the public items of this crate (Graph, Packet, Kernel, …).
• C ABI — the ffi module, declared in include/lmflow/flow.h. This is the only stable interface, and it is what C, C++, Python and mobile hosts use.

Kernels are decoupled from the engine

A kernel can be written in Rust (Kernel + register_kernel), in C++ (the header-only flow.hpp sugar layer), or in Python (pybind11). All three register into one registry through the same function-pointer vtable, so the engine neither knows nor cares which language a kernel came from — and kernels written in different languages can be mixed freely in a single graph.

This crate is a pure-Rust engine by default: it neither compiles nor bundles any C++, so cargo add lmflow needs no C++ toolchain. Kernels are always supplied explicitly by the host or by a linked kernel component; the engine itself only schedules them and never interprets payloads.

Running a graph

use lmflow::{register_kernel, Graph, Kernel, KernelCtx, Packet, Timestamp};

#[derive(Default)]
struct Relay;

impl Kernel for Relay {
    fn process(&mut self, context: &mut KernelCtx) -> lmflow::Result<()> {
        context.forward(0, 0)
    }
}

register_kernel::<Relay>("Relay")?;
let graph = Graph::from_yaml(
    r#"
nodes:
  - { name: relay, kernel: Relay, input_ports: [in], output_ports: [out] }
input_ports: [in]
output_ports: [out]
"#,
)?;

let out = graph.add_poller("out")?;
graph.start()?;
graph.input("in")?.send(Packet::from_i64(21).at(Timestamp(0)))?;

assert_eq!(out.next().and_then(|p| p.as_i64()), Some(21));

graph.close_all_inputs();
graph.wait_done()?;

Writing a kernel

use lmflow::{register_kernel, Kernel, KernelCtx, Packet};

#[derive(Default)]
struct Double;

impl Kernel for Double {
    fn process(&mut self, cc: &mut KernelCtx) -> lmflow::Result<()> {
        let v = cc.input(0).and_then(|p| p.as_i64()).unwrap_or(0);
        cc.emit(0, Packet::from_i64(v * 2))
    }
}

register_kernel::<Double>("Double")?; // once, before Graph::from_yaml

Beyond this crate

The lm-flow repository also carries 18 optional C++ kernels (../cpp/kernels/). They are built separately by CMake as lmflow::kernels; this crate never compiles or bundles C++.

Full documentation, including the C/C++ and Python interfaces: https://laomou.github.io/lm-flow/. The authoritative design document — scheduling model, timestamp and termination semantics, lock ordering rules and the decision log — is docs/design.md (written in Chinese).

Re-exports

pub use graph::DotView;

pub use graph::Graph;

pub use graph::Input;

pub use graph::OutputEvent;

pub use graph::Poller;

pub use graph::PollerBackpressureStatsSnapshot;

pub use graph::PollerOptions;

pub use graph::PollerOverflow;

pub use graph::State;

pub use graph::WatermarkBackpressureStatsSnapshot;

pub use kernel_api::register_kernel;

pub use kernel_api::Kernel;

pub use kernel_api::KernelContract;

pub use kernel_api::KernelCtx;

pub use kernel_runner::KernelRunner;

pub use metadata::Metadata;

pub use metadata::MetadataPredicate;

pub use metadata::MetadataValue;

pub use packet::BufferData;

pub use packet::Builtin;

pub use packet::InteropType;

pub use packet::Packet;

pub use status::Error;

pub use status::Result;

pub use timestamp::Timestamp;

Modules

config

图配置的 YAML 表示。

context

算子上下文:本次调用的输入、输出暂存、参数、side packet、自我信息。

executor

执行器:把「就绪节点」派给某个执行器执行。

ffi

The C ABI layer: the implementation behind include/lmflow/flow.h.

graph

The graph: topology, validation, execution and stream shutdown.

kernel

算子:vtable 抽象、全局注册表、端口表与契约。

kernel_api

Sugar for writing kernels in Rust: Kernel + register_kernel, plus the safe KernelCtx / KernelContract wrappers.

kernel_runner

Single-kernel test harness shared by Rust, C++ and Python hosts.

metadata

packet

数据包:引用计数共享的不可变 payload + 时间戳。

runtime

运行期共享设施:日志接收器、图级共享状态、C 字符串驻留。

status

错误类型与 C ABI 状态码。

timestamp

Timestamps: the ordering and lifetime markers carried by every packet in the stream.

trace

逐次调用的执行 trace:有界事件环 + Chrome Trace 导出。