ModuRL
ModuRL is a deep reinforcement learning framework for Rust, built on Candle.
It provides training algorithms, environments, models, policy distributions, storage, schedules, and logging. Tensor shapes, devices, networks, and optimizer configuration remain explicit.
[!WARNING] ModuRL is early in development. Public APIs may change between revisions.
Quick start
Install Git and a Rust toolchain that supports edition 2024. Then clone the repository and train PPO on eight CartPole environments:
git clone https://github.com/ModuRL/ModuRL.git
cd ModuRL
cargo run --release -p examples --example ppo_cartpole
The same Gym and Space traits used by training agents also support direct environment interaction:
use Device;
use *;
use CartPoleV1;
use EnvironmentError;
The example uses the CPU by default. Additional examples cover other algorithms and environments.
End-to-end training throughput
ModuRL completed the matched PPO, DQN, and SAC workloads faster than all three Python runners in this CPU benchmark.
[!NOTE] These benchmarks use small 2x64 networks on CartPole (PPO and DQN) and Pendulum (SAC). ModuRL's speed advantage is expected to be smaller with larger networks, where neural-network computation dominates runtime.
| Algorithm | ModuRL median | vs. SB3 runner | vs. CleanRL runner | vs. Tianshou runner |
|---|---|---|---|---|
| PPO | 2.102 s | 4.54x | 4.47x | 6.19x |
| DQN | 0.242 s | 6.49x | 3.97x | 15.50x |
| SAC | 1.396 s | 2.63x | 2.29x | 4.47x |
These CPU results were measured on a 12th Gen Intel Core i7-1255U with one compute thread. Each value is the median of five fresh-process samples.
The CleanRL runners adapt upstream reference scripts. The DQN Stable-Baselines3 and Tianshou runners also use documented adaptations so all frameworks perform the matched workload.
Compare frameworks within one algorithm. The algorithms perform different amounts of optimizer work per transition. These measurements do not cover sample efficiency, reward, or large-model performance. CUDA and Metal were not measured.
See the benchmark workloads, fairness controls, and accelerator commands, plus the raw PPO, DQN, and SAC samples.
What ModuRL includes
- Training algorithms: PPO, A2C, SAC, DDPG, TD3, DQN, and Double DQN.
- Models and policies: MLPs, shared or separate actor-critic networks, categorical and Gaussian policies, transformed distributions, and user-supplied Candle modules.
- Experience: rollout buffers, replay buffers, configurable replay devices, and parameter schedules.
- Environments: single and vectorized interfaces, classic control, Box2D, MuJoCo, and Atari integrations.
- Wrappers: time limits, observation and reward normalization, reward clipping, frame stacking, max-and-skip, and raw reward recording.
- Logging: algorithm-specific hooks, terminal graphs, and TensorBoard event output.
Architecture and choices
ModuRL keeps training decisions at the call site. Typed builders require each algorithm's components and provide defaults for tuning options. Agent builders also validate algorithm-specific settings in build().
After creating the Candle modules and optimizers, a separate-network PPO agent is assembled like this:
let policy =
new;
let networks = Separate;
let mut agent = builder
.action_space
.network_info
.batch_size
.mini_batch_size
.clip_range
.training_horizon
.device
.logging_info
.build?;
The public boundaries make each part replaceable:
| Decision | Available choices |
|---|---|
| Environment execution | Implement Gym for one environment or MultiGym for a batch. Collect Gym implementations with VectorizedGymWrapper::from(envs), or enable multithreading and use MultithreadedVectorizedGymWrapper. |
| Network topology | Use PPONetworkInfo::Separate for independent actor and critic modules, or PPONetworkInfo::Shared for one trunk with policy and value heads. |
| Policy representation | Wrap a Candle module in ProbabilisticPolicyModel<D>. Choose a categorical, Gaussian, transformed, or application-defined Distribution. |
| Training state and devices | On-policy agents use rollout storage. Replay agents use ReplayStorageConfig to select observation dtype and either one-device or hybrid storage. |
| Schedules and logging | Use constant, linear, exponential, or application-defined ParameterSchedule values. Pass a logger implementation to the agent builder, or omit logging. |
Agents train against these contracts instead of concrete environment, network, or logger types. See Core Concepts for how the parts interact.
Workspace crates
| Crate | Purpose | Pure Rust? | License |
|---|---|---|---|
modurl |
Agents, models, distributions, buffers, spaces, schedules, and environment traits | Yes, with default features | MIT |
modurl_gym |
Classic-control and Box2D environments, rendering, and Gym utilities | Yes, with default features | MIT |
modurl_mujoco |
Gymnasium-compatible MuJoCo v5 environments | No; uses the native MuJoCo library | MIT; Gymnasium asset license |
modurl_ale |
Arcade Learning Environment integration and Atari wrappers | No; builds bundled C++ | GPL-2.0-only; third-party notices |
modurl_logger |
Terminal graphs and TensorBoard event logging | Yes | MIT |
examples |
Runnable training programs; not published as a crate | Mixed; depends on the example | MIT |
modurl-benches |
Reproducible cross-framework throughput benchmarks; not published as a crate | Mixed; includes Python runners | MIT |
The core modurl crate has no default Cargo features. Enable its
multithreading feature for multithreaded vectorized and stacked environments.
Enable tracing to emit performance spans for agent learning, rollout
collection, and optimization. All spans use the modurl::performance target;
install a tracing subscriber in the application to record or display them.
Workspace crates define additional features for rendering and environment
support.
Documentation
The ModuRL Guide covers the training algorithms, environment interfaces, models, policies, distributions, and device configuration.
Build the core API documentation locally:
cargo doc -p modurl --no-deps --open
Development
Run the main checks used by workspace CI:
cargo fmt --all --check
cargo test --locked -p modurl -p modurl_gym
cargo test --locked --all-targets -p examples
MuJoCo and Atari checks have additional native requirements. See the modurl_mujoco README and modurl_ale README before working on those crates.
Issues and pull requests are welcome. Open an issue before starting a large API change.