cat README.md
# Action Persistence in Deep RL What happens when an agent **repeats each action for k steps** instead of deciding at every step? A framework and study comparing fixed and learned action persistence for **PPO, SAC and DQN**. Team project for the Reinforcement Learning course at UZH. ## Why Most RL agents act at whatever rate the simulator runs, which is rarely the rate a task needs. Holding an action for k steps lowers the control frequency: fewer decisions, longer-lasting exploration, smoother commands — at the price of slower reactions. We wanted to see where that trade pays off, and whether an agent can pick its own k. ## How - **Persistence wrapper** — an environment wrapper that repeats every action k = 1, 2, 4, 8, 16 times, so any algorithm can be trained at any k - **Adaptive agents** — PPO, SAC and DQN variants that choose the persistence together with the action, alongside a FiGAR-style PPO and an option-critic baseline - **Environments** — classic control, Atari, MiniGrid, and the MuJoCo locomotion tasks HalfCheetah and Walker2d - **Analysis** — reward per decision step and per environment step, action oscillation and flip rate, policies evaluated at a different k than they were trained with, and robustness to an artificial action delay - **Interface** — a Gradio app to configure runs and compare results ## Results - Counted in **decisions**, PPO with k = 2 keeps up with k = 1 on both MuJoCo tasks; counted in **environment steps**, k = 1 learns fastest and larger k lags behind - Persistence makes control **smoother**: PPO's action oscillation falls from 0.34 to 0.18 on HalfCheetah and from 0.31 to 0.09 on Walker2d between k = 1 and k = 8 - Persistence makes policies **robust to delay**: policies trained with k = 1 lose most of their reward once actions arrive a few steps late, while k = 4 and k = 8 stay roughly flat - The adaptive agents land between the fixed settings; they did not beat the best fixed k ## Stack - **RL** — Stable-Baselines3 (PPO, SAC, DQN), PyTorch, Gymnasium, MuJoCo - **Analysis** — pandas, seaborn, matplotlib - **Interface** — Gradio

ls figures/
Four line plots of PPO reward on HalfCheetah and Walker2d for k = 1, 2, 4, 8 and adaptive persistence, plotted against decision steps and against environment steps Four line plots of PPO reward on HalfCheetah and Walker2d for k = 1, 2, 4, 8 and adaptive persistence, plotted against decision steps and against environment steps
PPO on HalfCheetah and Walker2d. Left: reward per decision the agent makes. Right: the same runs per environment step, where each decision at persistence k costs k steps.
Bar charts of action oscillation and flip rate for PPO and SAC at persistence 1, 2, 4, 8 and adaptive, on HalfCheetah and Walker2d Bar charts of action oscillation and flip rate for PPO and SAC at persistence 1, 2, 4, 8 and adaptive, on HalfCheetah and Walker2d
Control smoothness of the trained policies: action oscillation and flip rate by persistence, lower is smoother.
Line plots of mean reward against an artificial action delay of 0 to 8 steps for PPO and SAC at several persistence values, on HalfCheetah and Walker2d Line plots of mean reward against an artificial action delay of 0 to 8 steps for PPO and SAC at several persistence values, on HalfCheetah and Walker2d
Robustness to action delay: mean reward when every action arrives 0–8 steps late.

ls -l
tech Python, PyTorch, Stable-Baselines3, Gymnasium, MuJoCo, Gradio