rlevo-core

Core abstractions for the rlevo Deep Reinforcement Learning with Evolutionary Optimization library.

This crate defines the foundational trait hierarchy and concrete types shared across all other rlevo crates. It deliberately contains no learning algorithms or environment implementations — only the interfaces those depend on.

Overview

rlevo-core models a reinforcement learning problem as a typed interaction loop:

Environment ──reset/step──► Snapshot ──observation──► Agent ──action──► Environment
                                   └──reward──────────────────────────────────────┘

All dimensions (state, observation, action) are enforced at compile time via const generics, so shape mismatches become type errors rather than runtime panics.

Modules

base — Core Traits

The primitive building blocks of the RL abstraction.

TensorConvertible is the integration point with Burn: implement it on your state/action types to enable neural-network-based agents.

action — Action Spaces

Three orthogonal action abstractions cover the standard taxonomy:

state — Advanced State Abstractions

Extensions to State<D> for partial observability, recurrence, and hierarchy:

environment — Environment Protocol

reward — Concrete Reward Types

ScalarReward(f32) is a thin newtype implementing Reward. It covers the vast majority of environments. Custom reward types can implement Reward directly.

memory — Experience Replay Buffers

Prioritized Experience Replay follows the algorithm described in:

Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2016). Prioritized Experience Replay. ICLR 2016. arXiv:1511.05952

experience — Trajectory Storage

evolution — Evolutionary Optimization

Minimal trait surface that rlevo-evolution builds on:

metrics — Performance Tracking

agent — Agent Traits (reserved)

Placeholder module reserved for a future unified agent trait hierarchy (act / learn / checkpoint). Empty in v0.1.0 — concrete algorithms currently live in rlevo-reinforcement-learning and rlevo-evolution and will migrate behind these traits once the API stabilizes.

render — Rendering Abstractions

Design Principles

Const generics for shapes. Every trait is parameterised by a const D (or SD, AD) so that dimension mismatches fail at compile time, not at runtime.

Separation of state and observation. State<D> represents the true MDP state; Observation<D> is what the agent receives. This separation makes POMDPs first-class citizens rather than workarounds.

Terminated vs. Truncated. EpisodeStatus distinguishes a natural terminal state from a step-limit cutoff. Algorithms that bootstrap values (e.g., TD-learning) need this distinction to avoid incorrect target computation.

Trait-based composition. No inheritance. New action spaces, state types, and genome kinds are added by implementing the appropriate trait.

Zero-cost abstractions. NullRenderer, Copy reward/state types, and const-generic shapes incur no runtime cost.

Usage

Add to your Cargo.toml:

[dependencies]
rlevo-core = { path = "../rlevo-core" }  # or version once published

Minimal Environment

use rlevo_core::{
    base::{Action, Observation, Reward, State},
    environment::{Environment, EnvironmentError, SnapshotBase},
    reward::ScalarReward,
};

struct MyEnv { /* ... */ }

impl Environment<1, 1, 1> for MyEnv {
    type StateType = MyState;
    type ObservationType = MyObservation;
    type ActionType = MyAction;
    type RewardType = ScalarReward;
    type SnapshotType = SnapshotBase<1, MyObservation, ScalarReward>;

    fn new(render: bool) -> Self { MyEnv { /* ... */ } }

    fn reset(&mut self) -> Result<Self::SnapshotType, EnvironmentError> {
        // reset internal state, return initial snapshot
        Ok(SnapshotBase::running(MyObservation::default(), ScalarReward::new(0.0)))
    }

    fn step(&mut self, action: MyAction) -> Result<Self::SnapshotType, EnvironmentError> {
        // apply action, compute reward
        Ok(SnapshotBase::terminated(MyObservation::default(), ScalarReward::new(1.0)))
    }
}

Replay Buffer

The prioritized replay buffer lives in rlevo-rl, not rlevo-core (per ADR 0003 — replay/experience/metrics moved to where they are consumed):

use rlevo_rl::memory::PrioritizedExperienceReplayBuilder;

let mut buffer = PrioritizedExperienceReplayBuilder::default()
    .with_capacity(100_000)
    .with_alpha(0.6)
    .build();

// buffer.add(obs, action, reward, next_obs, is_done, priority);
// let batch = buffer.sample_batch::<2, 2, MyBackend>(32, &device)?;

Examples

# Egocentric grid agent: State, Observation, TensorConvertible,
# DiscreteAction, and MultiDiscreteAction (move + interact)
cargo run -p rlevo-core --example grid_agent

# Constrained continuous state (2D robot workspace with orientation bounds)
cargo run -p rlevo-core --example continuous_state_with_constraints

Academic References

The following papers directly informed the algorithms and concepts in this crate:

• Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2016). Prioritized Experience Replay. International Conference on Learning Representations (ICLR). arXiv:1511.05952 — basis for PrioritizedExperienceReplay and the priority-weighted sampling scheme.

• Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.). MIT Press. — foundational framework for the Environment / Snapshot / Reward trait design, Bellman targets in ExperienceTuple, and the Terminated vs. Truncated episode status distinction.

• Puterman, M. L. (1994). Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley acm digital library. — theoretical grounding for MarkovState, BeliefState, and TransitionDynamics.

License

Licensed under either of Apache License, Version 2.0 or MIT License at your option.