reinforcex 0.0.4

Deep Reinforcement Learning Framework
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129

# About ReinforceX

ReinforceX (ReX) is a deep reinforcement learning framework built entirely from scratch in Rust.
We plan to implement various reinforcement learning algorithms, including value-based, policy-based, and actor-critic methods.

Advantages of Rust:
Efficient memory management prevents memory leaks that often trouble data scientists when using Python.
Enables thread-safe execution for parallel training.
Offers overall faster training speeds compared to Python-based frameworks.

# Package

crates.io: https://crates.io/crates/reinforcex
```
cargo add reinforcex
```

# Algorithms

DQN
PPO

# API

Instantiate the agent.
```Rust
use reinforcex::agents::{BaseAgent, DQN};
use reinforcex::explorers::EpsilonGreedy;
use reinforcex::models::FCQNetwork;
use reinforcex::memory::ReplayBuffer;
use std::sync::Arc;

let device = Device::cuda_if_available();
let vs = nn::VarStore::new(device);
let n_input_channels = 4;
let action_size = 2;
let n_hidden_layers = 2;
let n_hidden_channels = 128;

let optimizer = nn::Adam::default().build(&vs, 3e-4).unwrap();
let model = Box::new(FCQNetwork::new(
    vs,
    n_input_channels,
    action_size,
    n_hidden_layers,
    n_hidden_channels,
));

let gamma = 0.97;
let n_steps = 3;
let batchsize = 16;
let update_interval = 8;
let target_update_interval = 100;
let replay_buffer_capacity = 2000

let explorer = EpsilonGreedy::new(0.5, 0.1, 50000);
let transition_buffer = Arc::new(ReplayBuffer::new(replay_buffer_capacity, n_steps));

let mut agent = DQN::new(
    model,
    optimizer,
    transition_buffer,
    action_size as usize,
    batchsize,
    update_interval,
    target_update_interval,
    Box::new(explorer),
    gamma,
);
```

Methods of agent.
```Rust
fn act(&self, obs: &Tensor) -> Tensor;
fn act_and_train(&mut self, obs: &Tensor, reward: f64) -> Tensor;
fn stop_episode_and_train(&mut self, obs: &Tensor, reward: f64);
```

Pseudo code for training.
```Rust
for episode in 0..max_episode {
  for step in 0..max_step {
    action = agent.act_and_train(&mut self, obs: &Tensor, reward: f64)
    obs, reward = env.step(action)
  }
  agent.stop_episode_and_train(&mut self, obs: &Tensor, reward: f64)
}
```

This is a pseudo code for parallel learning.
ReplayBuffer is shared by all agents.
```Rust
use rayon::prelude::*;

let buffer = Arc::new(ReplayBuffer::new(1000, 1));

(0..n_threads).into_par_iter().for_each(|_| {
    let mut dqn = DQN::new(
        transition_buffer: Arc::clone(&buffer),
        ...(other params)...
    );

    for episode in 0..max_episode {
        for step in 0..max_step {
            action = agent.act_and_train(&mut self, obs: &Tensor, reward: f64)
            obs, reward = env.step(action)
        }
        agent.stop_episode_and_train(&mut self, obs: &Tensor, reward: f64)
    }
});
```


# Sample experiments

The experimental environment is built using OpenAI Gym. Since Gym is a Python framework, set up a Python environment and run the following pip command:
```
pip install gymnasium==0.26.3
```
We use Gym as the environment by calling Python from Rust.

```
cargo run --features cpu -- --env cartpole --algo dqn
```

<img width="597" alt="image" src="https://github.com/user-attachments/assets/b8c0606b-ec11-4b5a-b7fc-3070ad327d72" />

# Unit test

```
cargo test
```

# License

MIT License (https://github.com/kakky-hacker/reinforcex/blob/master/LICENSE)