fancy_gym/README.md

## ALR Robotics Control Environments

This project offers a large variety of reinforcement learning environments under the unifying interface of [OpenAI gym](https://gym.openai.com/).
We provide support (under the OpenAI interface) for the benchmark suites
[DeepMind Control](https://deepmind.com/research/publications/2020/dm-control-Software-and-Tasks-for-Continuous-Control)
(DMC) and [Metaworld](https://meta-world.github.io/). 
Custom (Mujoco) gym environments can be created according
to [this guide](https://www.gymlibrary.ml/content/environment_creation/). 
Unlike existing libraries, we additionally support to control agents with movement primitives, such as 
Dynamic Movement Primitives (DMPs) and Probabilistic Movement Primitives (ProMP, we only consider the mean usually).

## Movement Primitive Environments (Episode-Based/Black-Box Environments)

Unlike step-based environments, movement primitive (MP) environments are closer related to stochastic search, black-box
optimization, and methods that are often used in traditional robotics and control. 
MP environments are episode-based and always execute a full trajectory, which is generated by a trajectory generator, 
such as a Dynamic Movement Primitive (DMP) or a Probabilistic Movement Primitive (ProMP). 
The generated trajectory is translated into individual step-wise actions by a trajectory tracking controller. 
The exact choice of controller is, however, dependent on the type of environment. 
We currently support position, velocity, and PD-Controllers for position, velocity, and torque control, respectively 
as well as a special controller for the MetaWorld control suite.  
The goal of all MP environments is still to learn a optimal policy. Yet, an action
represents the parametrization of the motion primitives to generate a suitable trajectory. Additionally, in this
framework we support all of this also for the contextual setting, i.e. we expose a subset of the observation space 
as a single context in the beginning of the episode. This requires to predict a new action/MP parametrization for each
context. 
All environments provide next to the cumulative episode reward all collected information from each
step as part of the info dictionary. This information is, however, mainly meant for debugging as well as logging 
and not for training. 

|Key| Description|
|---|---|
`trajectory`| Generated trajectory from MP
`step_actions`| Step-wise executed action based on controller output
`step_observations`| Step-wise intermediate observations
`step_rewards`| Step-wise rewards
`trajectory_length`| Total number of environment interactions
`other`| All other information from the underlying environment are returned as a list with length `trajectory_length` maintaining the original key. In case some information are not provided every time step, the missing values are filled with `None`.

## Installation

1. Clone the repository

```bash 
git clone git@github.com:ALRhub/alr_envs.git
```

2. Go to the folder

```bash 
cd alr_envs
```

3. Install with

```bash 
pip install -e . 
```

## Using the framework

We prepared [multiple examples](alr_envs/examples/), please have a look there for more specific examples.

### Step-wise environments

```python
import alr_envs

env = alr_envs.make('HoleReacher-v0', seed=1)
state = env.reset()

for i in range(1000):
    state, reward, done, info = env.step(env.action_space.sample())
    if i % 5 == 0:
        env.render()

    if done:
        state = env.reset()
``` 

For Deepmind control tasks we expect the `env_id` to be specified as `domain_name-task_name` or for manipulation tasks
as `manipulation-environment_name`. All other environments can be created based on their original name.

Existing MP tasks can be created the same way as above. Just keep in mind, calling `step()` always executs a full
trajectory.

```python
import alr_envs

env = alr_envs.make('HoleReacherProMP-v0', seed=1)
# render() can be called once in the beginning with all necessary arguments. To turn it of again just call render(None). 
env.render()

state = env.reset()

for i in range(5):
    state, reward, done, info = env.step(env.action_space.sample())

    # Not really necessary as the environments resets itself after each trajectory anyway.
    state = env.reset()
```

To show all available environments, we provide some additional convenience. Each value will return a dictionary with two
keys `DMP` and `ProMP` that store a list of available environment names.

```python
import alr_envs

print("Custom MP tasks:")
print(alr_envs.ALL_ALR_MOTION_PRIMITIVE_ENVIRONMENTS)

print("OpenAI Gym MP tasks:")
print(alr_envs.ALL_GYM_MOTION_PRIMITIVE_ENVIRONMENTS)

print("Deepmind Control MP tasks:")
print(alr_envs.ALL_DMC_MOVEMENT_PRIMITIVE_ENVIRONMENTS)

print("MetaWorld MP tasks:")
print(alr_envs.ALL_METAWORLD_MOTION_PRIMITIVE_ENVIRONMENTS)
```

### How to create a new MP task

In case a required task is not supported yet in the MP framework, it can be created relatively easy. For the task at
hand, the following interface needs to be implemented.

```python
import numpy as np
from mp_env_api import MPEnvWrapper


class MPWrapper(MPEnvWrapper):

    @property
    def active_obs(self):
        """
            Returns boolean mask for each substate in the full observation.
            It determines whether the observation is returned for the contextual case or not.
            This effectively allows to filter unwanted or unnecessary observations from the full step-based case.
            E.g. Velocities starting at 0 are only changing after the first action. Given we only receive the first  
            observation, the velocities are not necessary in the observation for the MP task.
        """
        return np.ones(self.observation_space.shape, dtype=bool)

    @property
    def current_vel(self):
        """
            Returns the current velocity of the action/control dimension. 
            The dimensionality has to match the action/control dimension.
            This is not required when exclusively using position control, 
            it should, however, be implemented regardless.
            E.g. The joint velocities that are directly or indirectly controlled by the action.
        """
        raise NotImplementedError()

    @property
    def current_pos(self):
        """
            Returns the current position of the action/control dimension. 
            The dimensionality has to match the action/control dimension.
            This is not required when exclusively using velocity control, 
            it should, however, be implemented regardless.
            E.g. The joint positions that are directly or indirectly controlled by the action.
        """
        raise NotImplementedError()

    @property
    def goal_pos(self):
        """
            Returns a predefined final position of the action/control dimension.
            This is only required for the DMP and is most of the time learned instead.
        """
        raise NotImplementedError()

    @property
    def dt(self):
        """
            Returns the time between two simulated steps of the environment
        """
        raise NotImplementedError()

```

If you created a new task wrapper, feel free to open a PR, so we can integrate it for others to use as well. 
Without the integration the task can still be used. A rough outline can be shown here, for more details we recommend 
having a look at the [examples](alr_envs/examples/).

```python
import alr_envs

# Base environment name, according to structure of above example
base_env_id = "ball_in_cup-catch"

# Replace this wrapper with the custom wrapper for your environment by inheriting from the MPEnvWrapper.
# You can also add other gym.Wrappers in case they are needed, 
# e.g. gym.wrappers.FlattenObservation for dict observations
wrappers = [alr_envs.dmc.suite.ball_in_cup.MPWrapper]
mp_kwargs = {...}
kwargs = {...}
env = alr_envs.make_dmp_env(base_env_id, wrappers=wrappers, seed=1, mp_kwargs=mp_kwargs, **kwargs)
# OR for a deterministic ProMP (other traj_gen_kwargs are required):
# env = alr_envs.make_promp_env(base_env, wrappers=wrappers, seed=seed, traj_gen_kwargs=mp_args)

rewards = 0
obs = env.reset()

# number of samples/full trajectories (multiple environment steps)
for i in range(5):
    ac = env.action_space.sample()
    obs, reward, done, info = env.step(ac)
    rewards += reward

    if done:
        print(base_env_id, rewards)
        rewards = 0
        obs = env.reset()
```