cat README.md
# Action Persistence in Deep RL
What happens when an agent **repeats each action for k steps** instead of
deciding at every step? A framework and study comparing fixed and learned
action persistence for **PPO, SAC and DQN**. Team project for the
Reinforcement Learning course at UZH.
## Why
Most RL agents act at whatever rate the simulator runs, which is rarely the
rate a task needs. Holding an action for k steps lowers the control frequency:
fewer decisions, longer-lasting exploration, smoother commands — at the price
of slower reactions. We wanted to see where that trade pays off, and whether an
agent can pick its own k.
## How
- **Persistence wrapper** — an environment wrapper that repeats every action
k = 1, 2, 4, 8, 16 times, so any algorithm can be trained at any k
- **Adaptive agents** — PPO, SAC and DQN variants that choose the persistence
together with the action, alongside a FiGAR-style PPO and an option-critic
baseline
- **Environments** — classic control, Atari, MiniGrid, and the MuJoCo
locomotion tasks HalfCheetah and Walker2d
- **Analysis** — reward per decision step and per environment step, action
oscillation and flip rate, policies evaluated at a different k than they were
trained with, and robustness to an artificial action delay
- **Interface** — a Gradio app to configure runs and compare results
## Results
- Counted in **decisions**, PPO with k = 2 keeps up with k = 1 on both MuJoCo
tasks; counted in **environment steps**, k = 1 learns fastest and larger k
lags behind
- Persistence makes control **smoother**: PPO's action oscillation falls from
0.34 to 0.18 on HalfCheetah and from 0.31 to 0.09 on Walker2d between k = 1
and k = 8
- Persistence makes policies **robust to delay**: policies trained with k = 1
lose most of their reward once actions arrive a few steps late, while k = 4
and k = 8 stay roughly flat
- The adaptive agents land between the fixed settings; they did not beat the
best fixed k
## Stack
- **RL** — Stable-Baselines3 (PPO, SAC, DQN), PyTorch, Gymnasium, MuJoCo
- **Analysis** — pandas, seaborn, matplotlib
- **Interface** — Gradio
ls figures/
ls -l
Python, PyTorch, Stable-Baselines3, Gymnasium, MuJoCo, Gradio