Proximal Policy Optimization Algorithms

Reading guide

Problem and motivation

Standard policy gradient methods perform a single stochastic gradient descent step per sampled trajectory, leading to poor sample efficiency. While Trust Region Policy Optimization (TRPO) resolves instability by enforcing a constraint on the Kullback-Leibler (KL) divergence between the old and new policy distributions, it requires computing second-order derivatives (Fisher Information Matrix via conjugate gradient algorithms). This makes TRPO computationally expensive, hard to implement, and difficult to integrate with architectures that share parameters between the policy and value networks or use recurrent neural networks (RNNs).

Key ideas

PPO retains the stability benefits of TRPO while relying solely on first-order optimization. It achieves this by modifying the objective function rather than enforcing a hard or computationally heavy constraint:

• Probability Ratio: Defines the importance weight $r_t(\theta )=\frac{{\pi }_{\theta }(a_t|s_t)}{{\pi }_{{\theta }_{\text{old}}}(a_t|s_t)}$, where $r_t({\theta }_{\text{old}})=1$.
• Clipped Surrogate Objective: Prevents $r_t(\theta )$ from moving far away from 1 by clipping the ratio to $[1-ϵ,1+ϵ]$ and taking the minimum of the clipped and unclipped objectives. This acts as a pessimistic bound, discouraging overly large policy updates.
• Adaptive KL Penalty Alternative: Explores an alternative approach that adaptively adjusts a KL penalty coefficient $\beta$ based on the measured KL divergence relative to a target value $d_{\text{targ}}$.

Method (with the key equations)

Let $r_t(\theta )=\frac{{\pi }_{\theta }(a_t|s_t)}{{\pi }_{{\theta }_{\text{old}}}(a_t|s_t)}$ and ${\hat{A}}_t$ be an estimator of the advantage function at timestep $t$. The core clipped surrogate objective is:

$$L^{\text{CLIP}}(\theta )={\hat{\mathbb{E}}}_t\left[\min \left(r_t(\theta ){\hat{A}}_t,\text{clip}(r_t(\theta ),1-ϵ,1+ϵ){\hat{A}}_t\right)\right]$$

When optimizing a combined policy and value function network (sharing parameters), the combined loss function includes the clipped surrogate, a squared-error value function loss, and an entropy bonus $S[{\pi }_{\theta }]$ to encourage exploration:

$$L_t^{\text{CLIP+VF+S}}(\theta )={\hat{\mathbb{E}}}_t\left[L_t^{\text{CLIP}}(\theta )-c_1{\left(V_{\theta }(s_t)-V_t^{\text{targ}}\right)}^2+c_{2}S[{\pi }_{\theta }](s_t)\right]$$

Advantages ${\hat{A}}_t$ are computed using Generalized Advantage Estimation (GAE):

$${\hat{A}}_t=\sum _{l=0}^{\infty }(\gamma \lambda {)}^l{\delta }_{t+l}^V$$

where ${\delta }_t^V=r_t+\gamma V(s_{t+1})-V(s_t)$.

Results (with the most important numbers)

• Continuous Control Benchmarks: Evaluated on MuJoCo locomotion environments (e.g., HalfCheetah, Hopper, Walker2d, Humanoid). PPO with the clipped objective ($L^{\text{CLIP}}$) outperformed the adaptive KL penalty version and surpassed TRPO across nearly all domains. On HalfCheetah-v1, PPO achieved an episodic reward score over 4000 within 1 million steps, well above TRPO and other baselines.
• Hyperparameter Sensitivity: Experiments testing $ϵ\in \{0.1,0.2,0.3\}$ demonstrated that $ϵ=0.2$ consistently yielded strong performance.
• Atari Games: PPO achieved higher average episodic scores than A2C on 30 out of 49 games, and surpassed ACER in both sample efficiency and final score across several challenging titles.

Limitations and open questions

• The clipping mechanism is a heuristic lower bound that does not formally guarantee that policy updates remain within a strict trust region.
• Performance remains sensitive to generalized advantage estimation hyperparameters ($\lambda ,\gamma$) and learning rate schedules.
• While sample efficiency improves over standard policy gradients, it remains lower than fully off-policy algorithms (e.g., Q-learning or DDPG) on continuous control tasks.