modurl 0.1.0

Modular reinforcement learning algorithms and utilities built on Candle
Documentation
# Value-Based Training


Use DQN or DDQN when your environment has a discrete action space and you want
the agent to learn an action value for every possible action.

This chapter explains the pieces shared by `DQNAgent` and `DDQNAgent`. Read
[DQN](./dqn.md) for a complete CartPole program. Read [Double DQN](./ddqn.md)
when you want DDQN's target calculation instead.

## Choose DQN or DDQN


Both agents train an online Q-network and periodically copy its parameters to a
target Q-network. They differ only in how they form the next-state training
target:

| Agent | Chooses the next action | Evaluates that action |
| --- | --- | --- |
| `DQNAgent` | The target Q-network's largest value | The same target Q-network |
| `DDQNAgent` | The online Q-network's largest value | The target Q-network |

Start with DQN when you need the standard DQN update. Choose DDQN when you want
to reduce overestimation of action values. In DQN, the target network both
chooses the largest next-state value and uses that value in the training target.
Taking a maximum tends to favor values that are accidentally too high. DDQN
uses the online network to choose the action and the target network to evaluate
it, which reduces that optimistic bias. The builder fields and training loop are
otherwise the same.

## Terms


A *Q-network* maps one observation to one value per discrete action. Its output
width must equal the number of actions. An *online Q-network* is the network the
optimizer updates. A *target Q-network* has the same architecture and variable
names, but the agent only refreshes its parameters by copying the online network
at a fixed interval.

*Epsilon-greedy exploration* chooses a random valid action with probability
epsilon and otherwise chooses the online network's highest-valued action.
`epsilon_schedule` controls epsilon over the configured training horizon.

## Use a Dueling Q-Network


`DuelingMLP` separates its final representation into a scalar state value and
one advantage per action. It mean-centers the advantages and combines the two
streams into the same `[batch, action_count]` output expected from any
Q-network:

```rust
let online_q_network = DuelingMLP::builder()
    .input_size(observation_space.shape()[0])
    .output_size(2)
    .vb(online_vb)
    .hidden_layer_sizes(vec![64, 64])
    .value_hidden_layer_sizes(vec![64])
    .advantage_hidden_layer_sizes(vec![64])
    .build()?;
```

`hidden_layer_sizes` configures the shared trunk. The value and advantage
hidden-layer fields configure the two independent streams after that trunk.
Leave either stream's list empty when you want its output head to connect
directly to the shared features.

Pass identically configured online and target `DuelingMLP` instances to either
`DQNAgent` or `DDQNAgent`. Dueling changes the network architecture; DDQN
independently changes the next-state target calculation, so the two techniques
can be used together.

## Where to Go Next


Build the complete [DQN CartPole program](./dqn.md). It shows the two
Q-networks, replay configuration, and training call in one place. Then use the
small, documented change in [Double DQN](./ddqn.md) to change its target
calculation.