Expand description
Remote worker for distributed pipeline execution.
Receives execution plans from a coordinator, runs them locally,
and reports results. Each worker manages isolated Python environments
(EnvManager) and communicates via WebSocket (protocol).
Python filters execute in a child subprocess — the GIL is completely isolated from Rust/Tokio threads. No PyO3, no segfaults.
Re-exports§
pub use detect::ResourceLimits;pub use env_manager::EnvManager;pub use python_process::PythonProcess;pub use python_process::SubprocessFilter;pub use server::serve_worker;pub use server::serve_worker_authenticated;pub use server::worker_router;pub use server::worker_router_authenticated;pub use worker::Worker;pub use ws_transport::WsTransport;pub use protocol::*;
Modules§
- detect
- Auto-detection of hardware capabilities and resource limiting.
- env_
manager - Python environment manager: creates and maintains isolated venvs/conda envs per pipeline, with incremental dependency updates.
- protocol
- Wire protocol for coordinator ↔ worker communication.
- python_
process - Python subprocess — persistent daemon for filter execution.
- server
- Axum HTTP/WebSocket server for the worker process.
- worker
- Worker — receives and executes plans from a coordinator.
- ws_
transport - WebSocket-based Transport implementation.