# Inverted Double Pendulum¶

This environment is part of the Mujoco environments which contains general information about the environment.

Action Space

Box(-1.0, 1.0, (1,), float32)

Observation Space

Box(-inf, inf, (9,), float64)

import

gymnasium.make("InvertedDoublePendulum-v5")

## Description¶

This environment originates from control theory and builds on the cartpole environment based on the work of Barto, Sutton, and Anderson in “Neuronlike adaptive elements that can solve difficult learning control problems”, powered by the Mujoco physics simulator - allowing for more complex experiments (such as varying the effects of gravity or constraints). This environment involves a cart that can be moved linearly, with one pole attached to it and a second pole attached to the other end of the first pole (leaving the second pole as the only one with a free end). The cart can be pushed left or right, and the goal is to balance the second pole on top of the first pole, which is in turn on top of the cart, by applying continuous forces to the cart.

## Action Space¶

The agent take a 1-element vector for actions. The action space is a continuous (action) in [-1, 1], where action represents the numerical force applied to the cart (with magnitude representing the amount of force and sign representing the direction)

Num

Action

Control Min

Control Max

Name (in corresponding XML file)

Joint

Type (Unit)

0

Force applied on the cart

-1

1

slider

slide

Force (N)

## Observation Space¶

The observation space consists of the following parts (in order):

• qpos (1 element): Position values of the robot’s cart.

• sin(qpos) (2 elements): The sine of the angles of poles.

• cos(qpos) (2 elements): The cosine of the angles of poles.

• qvel (3 elements): The velocities of these individual body parts (their derivatives).

• qfrc_constraint (1 element): Constraint force of the cart. There is one constraint force for contacts for each degree of freedom (3). The approach and handling of constraints by MuJoCo is unique to the simulator and is based on their research. More information can be found in their documentation or in their paper “Analytically-invertible dynamics with contacts and constraints: Theory and implementation in MuJoCo”.

The observation space is a Box(-Inf, Inf, (9,), float64) where the elements are as follows:

Num

Observation

Min

Max

Name (in corresponding XML file)

Joint

Type (Unit)

0

position of the cart along the linear surface

-Inf

Inf

slider

slide

position (m)

1

sine of the angle between the cart and the first pole

-Inf

Inf

sin(hinge)

hinge

unitless

2

sine of the angle between the two poles

-Inf

Inf

sin(hinge2)

hinge

unitless

3

cosine of the angle between the cart and the first pole

-Inf

Inf

cos(hinge)

hinge

unitless

4

cosine of the angle between the two poles

-Inf

Inf

cos(hinge2)

hinge

unitless

5

velocity of the cart

-Inf

Inf

slider

slide

velocity (m/s)

6

angular velocity of the angle between the cart and the first pole

-Inf

Inf

hinge

hinge

7

angular velocity of the angle between the two poles

-Inf

Inf

hinge2

hinge

8

constraint force - x

-Inf

Inf

slider

slide

Force (N)

excluded

constraint force - y

-Inf

Inf

slider

slide

Force (N)

excluded

constraint force - z

-Inf

Inf

slider

slide

Force (N)

## Rewards¶

The total reward is: reward = alive_bonus - distance_penalty - velocity_penalty.

• alive_bonus: Every timestep that the Inverted Pendulum is healthy (see definition in section “Episode End”), it gets a reward of fixed value healthy_reward (default is $$10$$).

• distance_penalty: This reward is a measure of how far the tip of the second pendulum (the only free end) moves, and it is calculated as $$0.01 x_{pole2-tip}^2 + (y_{pole2-tip}-2)^2$$, where $$x_{pole2-tip}, y_{pole2-tip}$$ are the xy-coordinatesof the tip of the second pole.

• velocity_penalty: A negative reward to penalize the agent for moving too fast. $$10^{-3} \omega_1 + 5 \times 10^{-3} \omega_2$$, where $$\omega_1, \omega_2$$ are the angular velocities of the hinges.

info contains the individual reward terms.

## Starting State¶

The initial position state is $$\mathcal{U}_{[-reset\_noise\_scale \times I_{3}, reset\_noise\_scale \times I_{3}]}$$. The initial velocity state is $$\mathcal{N}(0_{3}, reset\_noise\_scale^2 \times I_{3})$$.

where $$\mathcal{N}$$ is the multivariate normal distribution and $$\mathcal{U}$$ is the multivariate uniform continuous distribution.

## Episode End¶

### Termination¶

The environment terminates when the Inverted Double Pendulum is unhealthy. The Inverted Double Pendulum is unhealthy if any of the following happens:

1.Termination: The y_coordinate of the tip of the second pole $$\leq 1$$.

Note: The maximum standing height of the system is 1.2 m when all the parts are perpendicularly vertical on top of each other.

### Truncation¶

The default duration of an episode is 1000 timesteps.

## Arguments¶

InvertedDoublePendulum provides a range of parameters to modify the observation space, reward function, initial state, and termination condition. These parameters can be applied during gymnasium.make in the following way:

import gymnasium as gym
env = gym.make('InvertedDoublePendulum-v5', healthy_reward=10, ...)


Parameter

Type

Default

Description

xml_file

str

"inverted_double_pendulum.xml"

Path to a MuJoCo model

healthy_reward

float

10

Constant reward given if the pendulum is healthy (upright) (see Rewards section)

reset_noise_scale

float

0.1

Scale of random perturbations of initial position and velocity (see Starting State section)

## Version History¶

• v5:

• Minimum mujoco version is now 2.3.3.

• Added default_camera_config argument, a dictionary for setting the mj_camera properties, mainly useful for custom environments.

• Added frame_skip argument, used to configure the dt (duration of step()), default varies by environment check environment documentation pages.

• Fixed bug: healthy_reward was given on every step (even if the Pendulum is unhealthy), now it is only given if the DoublePendulum is healthy (not terminated)(related GitHub issue).

• Excluded the qfrc_constraint (“constraint force”) of the hinges from the observation space (as it was always 0, thus providing no useful information to the agent, resulting in slightly faster training) (related GitHub issue).

• Added xml_file argument.

• Added reset_noise_scale argument to set the range of initial states.

• Added healthy_reward argument to configure the reward function (defaults are effectively the same as in v4).

• Added individual reward terms in info ( info["reward_survive"], info["distance_penalty"], info["velocity_penalty"]).

• v4: All MuJoCo environments now use the MuJoCo bindings in mujoco >= 2.1.3.

• v3: This environment does not have a v3 release.

• v2: All continuous control environments now use mujoco-py >= 1.50.

• v1: max_time_steps raised to 1000 for robot based tasks (including inverted pendulum).

• v0: Initial versions release.