Playing Atari with Deep Reinforcement Learning

Reading guide

Problem and motivation

Reinforcement learning (RL) in sensory domains historically depended on hand-engineered features combined with linear value functions or policy approximations. Deep neural networks had demonstrated strong representation learning in computer vision and speech recognition, but applying them to RL posed fundamental challenges:

• Delayed and sparse rewards: Learning signals are scalar, noisy, and separated from actions by thousands of time steps.
• Non-i.i.d. observations: Successive environment states form highly correlated time series rather than independent samples.
• Non-stationary distributions: The data-generating policy shifts continuously during learning, creating feedback loops that can cause nonlinear function approximators to diverge.

The goal of this work is to create a single algorithm and neural network architecture capable of learning control policies directly from high-dimensional visual input ($210\times 160$ RGB video) across diverse Atari 2600 games without game-specific modifications.

Key ideas

• End-to-end sensory control: Map raw pixels directly to action-values ($Q$-values) without domain-specific vision systems or access to emulator internals.
• Experience Replay Buffer: Store agent interactions $e_t=(s_t,a_t,r_t,s_{t+1})$ in a finite memory buffer $\mathcal{D}$. Training draws uniform mini-batches randomly from this memory. This breaks sample correlations, reduces gradient variance, and stabilizes the data distribution across past policies.
• Action-value output parameterization: Instead of feeding both state and action as inputs to compute a single scalar $Q(s,a)$, the network takes only the state representation as input and outputs a vector containing the predicted $Q$-values for all legal discrete actions in a single forward pass.
• Frame stacking and downsampling: Because single frames suffer from perceptual aliasing (inability to infer velocity or trajectory), histories of four preprocessed grayscale frames are stacked into an $84\times 84\times 4$ input tensor.

Method (with the key equations)

The state sequence $s_t=(x_1,a_1,\dots ,x_t)$ is mapped via a preprocessing function $\varphi$ to an $84\times 84\times 4$ representation. The network parameters $\theta$ are trained using a sequence of mean squared Bellman error loss functions at each iteration $i$:

$$L_i({\theta }_i)={\mathbb{E}}_{(s,a,r,s^{\prime })\sim \mathcal{D}}\left[{\left(y_i-Q(s,a;{\theta }_i)\right)}^2\right]$$

where the target $y_i$ is computed using parameters ${\theta }_{i-1}$ from the previous iteration:

$$y_i=\{\begin{array}{ll}r& \text{for terminal }{s}^{\prime }\\ r+\gamma {\max }_{a^{\prime }}Q(s^{\prime },a^{\prime };{\theta }_{i-1})& \text{for non-terminal }{s}^{\prime }\end{array}$$

The gradient with respect to ${\theta }_i$ is:

$${\nabla }_{{\theta }_i}{L}_i({\theta }_i)={\mathbb{E}}_{(s,a,r,s^{\prime })}\left[\left(r+\gamma \underset{a^{\prime }}{\max }Q(s^{\prime },a^{\prime };{\theta }_{i-1})-Q(s,a;{\theta }_i)\right){\nabla }_{{\theta }_i}Q(s,a;{\theta }_i)\right]$$

Model Architecture and Training Setup

• Input: $84\times 84\times 4$ image produced by cropping downsampled grayscale frames.
• Conv 1: 16 filters of size $8\times 8$, stride 4, ReLU activation.
• Conv 2: 32 filters of size $4\times 4$, stride 2, ReLU activation.
• FC 1: 256 fully-connected ReLU units.
• Output: Fully-connected linear layer with 4 to 18 outputs (one per legal action).
• Optimizer: RMSProp with batch size 32.
• Exploration: $ϵ$-greedy with $ϵ$ annealed linearly from $1.0$ to $0.1$ over the first $10^6$ frames, then fixed at $0.1$.
• Reward Clipping: Positive rewards set to $+1$, negative rewards to $-1$, and neutral rewards to $0$ to standardize the error scale.
• Frame skipping: Agent observes and acts every $k=4$ frames (repeated action), except $k=3$ for Space Invaders to keep blinking laser sprites visible.

Results (with the most important numbers)

DQN was evaluated on seven Atari 2600 games using an $ϵ$-greedy policy with $ϵ=0.05$. Average scores compared to prior linear RL baselines and expert humans:

• DQN outperformed prior reinforcement learning methods (Sarsa and Contingency) on all seven games.
• DQN outperformed human expert scores on three games: Breakout (168 vs. 31), Enduro (470 vs. 368), and Pong (20 vs. -3).
• Tracking the average predicted maximum $Q$-value on a held-out set of states yielded smooth monotonic improvement during training, demonstrating stability without divergence.

Limitations and open questions

• Long-horizon strategic planning: Games requiring delayed, long-term credit assignment (such as Q*bert, Seaquest, and Space Invaders) lagged significantly behind human scores.
• Reward clipping side effects: Normalizing all positive rewards to $+1$ prevents distinguishing high-value targets from low-value targets.
• Uniform replay memory limitations: The replay buffer has a fixed capacity ($10^6$ frames) that overwrites older data and samples uniformly without weighting transitions by importance (unlike prioritized replay).
• Changing target values: While parameters ${\theta }_{i-1}$ are kept fixed during a single gradient step, they update continuously across SGD steps (a dedicated slowly updating target network was introduced in later work).