Training Agents¶
The most common application of Gymnasium is for training RL agents. Therefore, these tutorials aim to show a range of example implementations for different environments.
Each tutorial trains a small agent from scratch with a compact, readable algorithm, so the full loop (collecting experience, updating the policy or value function, and evaluating the result) stays visible instead of being hidden behind a training library.
What the tutorials cover¶
Solving Blackjack with Tabular Q-Learning - Q-learning with an epsilon-greedy policy on
Blackjack-v1, including a heatmap of the learned state-action values.Solving FrozenLake with Tabular Q-Learning - the same algorithm on
FrozenLake-v1, compared across map sizes.Action Masking in the Taxi Environment - tabular Q-learning on
Taxi-v4with and without theaction_maskthat the environment returns ininfo.Training using REINFORCE for Mujoco - a PyTorch implementation of REINFORCE, a Monte-Carlo policy gradient method, on
InvertedPendulum-v5.Speeding up A2C Training with Vector Envs - an A2C implementation trained with
SyncVectorEnvandAsyncVectorEnvonLunarLander-v3, showing how running environments in parallel increases throughput.
Requirements¶
The tutorials require Python 3.11 or newer, gymnasium with the toy-text extra (for Blackjack,
FrozenLake and Taxi), and the plotting and progress-bar packages used by the training scripts:
pip install "gymnasium[toy-text]" matplotlib pandas seaborn tqdm
The two deep reinforcement learning tutorials also need PyTorch and the extra for the environment they train on:
pip install "gymnasium[mujoco,box2d]" torch
Training using REINFORCE for Mujoco trains on InvertedPendulum-v5 from the mujoco extra and
Speeding up A2C Training with Vector Envs trains on LunarLander-v3 from the box2d extra.
Running a tutorial locally¶
Every tutorial page ends with a Download Python source code link. Save the file and run it from a terminal:
python blackjack_q_learning.py
Training takes time: Blackjack trains for 100,000 episodes, FrozenLake for 2,000 episodes per run across several map sizes, Taxi for 5,000 episodes in each of 12 runs, REINFORCE for 5,000 episodes and A2C for 1,000 updates in several parallel environments. Lower the episode, update and timestep counts while experimenting, and raise them again for a final run.
The vector environment tutorial explains how to wrap the training script in
if __name__ == "__main__":, which AsyncVectorEnv needs on platforms that start processes with
spawn (macOS and Windows).
Where to go next¶
Gymnasium Basics covers the
reset/steploop and wrappers if you are new to Gymnasium.Vector environments documents the vector API used by the A2C tutorial.
