Struct Engine 

pub struct Engine { /* private fields */ }

High-performance async workflow engine for message processing.

Architecture

The engine is designed for async-first operation with Tokio:

• Separation of Concerns: Distinct executors for workflows and tasks
• Shared DataLogic: Single DataLogic instance with Arc for thread-safe sharing
• Arc: Pre-compiled logic shared across all async tasks
• Async Functions: Native async support for I/O-bound operations

Performance Characteristics

• Zero Runtime Compilation: All logic compiled during initialization
• Zero-Copy Sharing: Arc-wrapped compiled logic shared without cloning
• Optimal for Mixed Workloads: Async I/O with blocking CPU evaluation
• Thread-Safe by Design: All components safe to share across Tokio tasks

Implementations

impl Engine

pub fn new( workflows: Vec<Workflow>, custom_functions: Option<HashMap<String, Box<dyn AsyncFunctionHandler + Send + Sync>>>, ) -> Self

Creates a new Engine instance with configurable parameters.

Arguments

• workflows - The workflows to use for processing messages
• custom_functions - Optional custom async function handlers

Example

use dataflow_rs::{Engine, Workflow};

let workflows = vec![Workflow::from_json(r#"{"id": "test", "name": "Test", "priority": 0, "tasks": [{"id": "task1", "name": "Task 1", "function": {"name": "map", "input": {"mappings": []}}}]}"#).unwrap()];

// Simple usage with defaults
let engine = Engine::new(workflows, None);

pub async fn process_message(&self, message: &mut Message) -> Result<()>

Processes a message through workflows that match their conditions.

This async method:

1. Iterates through workflows sequentially in deterministic order (sorted by ID)
2. Delegates workflow execution to the WorkflowExecutor
3. Updates message metadata

Arguments

• message - The message to process through workflows

Returns

• Result<()> - Ok(()) if processing succeeded, Err if a fatal error occurred

pub fn workflows(&self) -> &Arc<HashMap<String, Workflow>>

Get a reference to the workflows

pub fn datalogic(&self) -> &Arc<DataLogic>

Get a reference to the DataLogic instance

pub fn logic_cache(&self) -> &Vec<Arc<CompiledLogic>>

Get a reference to the compiled logic cache