Struct relearn::envs::meta::MetaEnv

pub struct MetaEnv<E> {
    pub env_distribution: E,
}

A meta reinforcement learning environment that treats RL itself as an environment.

An episode in this meta environment is called a “Trial” and consists of several episodes from the inner environment. A new inner environment with a different structure seed is sampled for each Trial. A meta episode ends when a fixed number of inner episodes have been completed.

The step metadata from the inner environment are embedded as observations.

Observations

A MetaObservation. Consists of the inner observation, the previous step action and feedback, and whether the inner episode is done.

Actions

The action space is the same as the action space of the inner environments. Actions are forwarded to the inner environment except when the current state is the last state of the inner episode (episode_done == true). In that case, the provided action is ignored and the next state will be the start of a new inner episode.

Feedback

The inner environment feedback must implement MetaFeedback (and MetaFeedbackSpace for the space). This feedback is decomposed into separate inner and outer feedback. In the case of Reward feedback, the same value is used as both the inner and outer feedback.

States

The state (MetaState) consists of an inner environment instance, an inner environment state, an episode index within the trial, and details of the most recent inner step within the episode.

Reference

This meta environment design is roughly consistent with the structure used in the paper “RL^2: Fast Reinforcement Learning via Slow Reinforcement Learning” by Duan et al.

Fields

env_distribution: E

Environment distribution from which each trial’s episode is sampled.

Implementations

impl<E> MetaEnv<E>

pub const fn new(env_distribution: E) -> Self

impl<E> MetaEnv<E> where
    E: EnvStructure, 

pub const fn inner_structure(&self) -> InnerEnvStructure<&Self>

View the structure of the inner environment.

Trait Implementations

impl<EC> BuildEnv for MetaEnv<EC> where
    EC: BuildEnvDist,
    EC::Action: Copy,
    EC::FeedbackSpace: MetaFeedbackSpace<Element = EC::Feedback>,
    <EC::FeedbackSpace as Space>::Element: MetaFeedback,
    EC::Feedback: MetaFeedback, 

type Observation = <<MetaEnv<EC> as BuildEnv>::Environment as Environment>::Observation

Environment observation type.

type Action = <<MetaEnv<EC> as BuildEnv>::Environment as Environment>::Action

Environment action type.

type Feedback = <<MetaEnv<EC> as BuildEnv>::Environment as Environment>::Feedback

Environment feedback type.

type ObservationSpace = <<MetaEnv<EC> as BuildEnv>::Environment as EnvStructure>::ObservationSpace

Environment observation space type.

type ActionSpace = <<MetaEnv<EC> as BuildEnv>::Environment as EnvStructure>::ActionSpace

Environment action space type.

type FeedbackSpace = <<MetaEnv<EC> as BuildEnv>::Environment as EnvStructure>::FeedbackSpace

Environment feedback space type.

type Environment = MetaEnv<<EC as BuildEnvDist>::EnvDistribution>

Type of environment to build

fn build_env(&self, _: &mut Prng) -> Result<Self::Environment, BuildEnvError>

Build an environment instance.

impl<E: Clone> Clone for MetaEnv<E>

fn clone(&self) -> MetaEnv<E>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<E: Debug> Debug for MetaEnv<E>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<E: Default> Default for MetaEnv<E>

fn default() -> Self

Returns the “default value” for a type.

impl<'de, E> Deserialize<'de> for MetaEnv<E> where
    E: Deserialize<'de>, 

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
    __D: Deserializer<'de>, 

Deserialize this value from the given Serde deserializer.

impl<E> EnvStructure for MetaEnv<E> where
    E: EnvStructure,
    E::FeedbackSpace: MetaFeedbackSpace, 

type ObservationSpace = MetaObservationSpace<<E as EnvStructure>::ObservationSpace, <E as EnvStructure>::ActionSpace, <<E as EnvStructure>::FeedbackSpace as MetaFeedbackSpace>::InnerSpace>

type ActionSpace = <E as EnvStructure>::ActionSpace

type FeedbackSpace = <<E as EnvStructure>::FeedbackSpace as MetaFeedbackSpace>::OuterSpace

fn observation_space(&self) -> Self::ObservationSpace

Space containing all possible observations.

fn action_space(&self) -> Self::ActionSpace

The space of all possible actions.

fn feedback_space(&self) -> Self::FeedbackSpace

The space of all possible feedback.

fn discount_factor(&self) -> f64

A discount factor applied to future feedback.

impl<E> Environment for MetaEnv<E> where
    E: EnvDistribution,
    E::Action: Clone,
    E::Observation: Clone,
    E::Feedback: MetaFeedback, 

type State = MetaState<<E as EnvDistribution>::Environment>

Environment state type. Not necessarily observable by the agent.

type Observation = MetaObservation<<E as EnvDistribution>::Observation, <E as EnvDistribution>::Action, <<E as EnvDistribution>::Feedback as MetaFeedback>::Inner>

Observation of the state provided to the agent.

type Action = <E as EnvDistribution>::Action

Action selected by the agent.

type Feedback = <<E as EnvDistribution>::Feedback as MetaFeedback>::Outer

Feedback provided to a learning agent as the result of each step. Reward, for example.

fn initial_state(&self, rng: &mut Prng) -> Self::State

Sample a state for the start of a new episode.

fn observe(&self, state: &Self::State, rng: &mut Prng) -> Self::Observation

Generate an observation for a given state.

fn step(
    &self,
    state: Self::State,
    action: &Self::Action,
    rng: &mut Prng,
    logger: &mut dyn StatsLogger
) -> (Successor<Self::State>, Self::Feedback)

Perform a state transition in reponse to an action.

fn run<T, L>(self, actor: T, seed: SimSeed, logger: L) -> Steps<Self, T, Prng, L> where
    T: Actor<Self::Observation, Self::Action>,
    L: StatsLogger,
    Self: Sized, 

Run this environment with the given actor.

impl<E: Hash> Hash for MetaEnv<E>

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H) where
    H: Hasher, 

Feeds a slice of this type into the given Hasher.

impl<E: PartialEq> PartialEq<MetaEnv<E>> for MetaEnv<E>

fn eq(&self, other: &MetaEnv<E>) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &MetaEnv<E>) -> bool

This method tests for !=.

impl<E> Serialize for MetaEnv<E> where
    E: Serialize, 

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error> where
    __S: Serializer, 

Serialize this value into the given Serde serializer.

impl<E: Copy> Copy for MetaEnv<E>

impl<E: Eq> Eq for MetaEnv<E>

impl<E> StructuralEq for MetaEnv<E>

impl<E> StructuralPartialEq for MetaEnv<E>