Atari ALE as a Benchmark for Deep RL Agents

The Atari Learning Environment (ALE) gave reinforcement learning a shared, reproducible testbed at a time when researchers were mostly building their own small, hand-crafted problems and comparing results that didn't actually compare to anything. Bellemare and colleagues introduced it in 2013, and the standard metrics that grew out of it, especially the "superhuman" label, now hide more than they reveal about sample efficiency, generalization, and what these agents have actually learned to do.

The platform runs on the Stella emulator, the same software people use to play original Atari 2600 cartridges on modern hardware, and it exposes agents to more than 50 arcade games: Pong, Breakout, Space Invaders, Pac-Man, Montezuma's Revenge, and dozens more. Every game shares the same interface. An agent sees raw pixels, a joystick with multiple directions plus a button gives it a discrete set of possible actions, and reward structures vary wildly from game to game, sparse in some, nearly continuous in others. That consistency is what made ALE useful as a Markov Decision Process testbed: researchers could point the same algorithm at wildly different games and see what happened without redesigning the problem each time.

ALE was built to answer a specific question: can an agent learn directly from pixels to actions, at scale, without a human hand-engineering features for each game? A second, related question was whether a single agent, using one set of hyperparameters, could handle enough different tasks that no narrow trick would dominate. The benchmark was also attractive because Bellemare's team built it independently of any lab trying to win a specific competition, it ran cheaply enough that most university labs could use it, and its variety made overfitting to one game's quirks a bad long-term strategy. The games test perception and quick eye-hand coordination more than they test planning, a limitation researchers have noted since the platform's early years. Before DQN made it famous, ALE was already showing up in multi-task and transfer learning work, including studies by Machado and colleagues and by researchers such as Parisotto and Rusu.

How DQN turned ALE into the field's central benchmark

Mnih and colleagues published the paper that changed everything in Nature in 2015 (volume 518, pages 529 to 533). Their reinforcement learning algorithm took in nothing but raw pixels and the game score, and it beat every prior algorithm while matching a professional human games tester across 49 Atari titles. The remarkable part wasn't the win itself, it was that the same network architecture, same algorithm, and same hyperparameters worked across all 49 games without per-game tuning. That generality was the actual claim, and it's the reason the paper still gets cited as a turning point rather than just a good result.

DQN's success set three conventions that the field still uses. Human-normalized score, or HNS, became the standard way to report results, measuring agent performance relative to both a random policy baseline and a human reference score. "Superhuman" became a real threshold, defined against a professional games tester rather than a world-record holder or a dedicated speedrunner. And the 57-game Atari suite became the field's default proving ground.

What followed was a fast climb. DQN scored a mean HNS of 228% in 2015. C51 hit 701% in 2017. UNREAL reached 880% that same year, and QR-DQN pushed to 915%. In 2018 alone, IQN scored 1019%, Rainbow reached 1189%, Ape-X DQN climbed to 1695%, and Ape-X DQfD posted 2346% across a 42-game subset. Numbers above 100% mean the agent is beating the average human tester by a wide margin, not that it's beating the best human who has ever played the game. That distinction affects how the next section should be read, since it explains why the games test perception more than planning.

What "superhuman" means under standard ALE metrics, and why the human baseline is weak

The human baseline behind HNS comes from a professional games tester playing each title briefly, not from an expert who trained for months, not from a speedrunner, and certainly not from whoever holds the actual world record. That baseline was reasonable enough for a 2015 paper trying to prove pixels-to-action learning was possible. It's a much shakier foundation for claims about superhuman performance made years later.

Fan's 2021 paper (revised in 2023, arXiv:2112.04145) makes the argument directly: current superhuman claims are built on an evaluation standard that underestimates what humans can actually do, which makes the entire "superhuman" framing misleading. Fan proposes measuring agents against human world records instead, a much higher bar on both final score and how much data an agent needs to get there. Under that stricter standard, only a handful of agents can match or beat top human scores on select games, and according to the ALE topic survey on Emergent Mind, no method matches world-record performance while also using anything close to a human-comparable number of practice attempts. World records represent a stable, meaningful ceiling rather than a moving target, and Fan's paper treats them as the appropriate upper bar for evaluating these claims.

Those big HNS numbers carry a second distortion. Agents that post scores well above 1000% do so across a mixed set of games, while still struggling on games that demand different capabilities. Those same agents often struggle badly on games that demand memory, planning, or exploring for a long time before any reward shows up. Averaging across games smooths all of that into one clean number, obscuring the variation in where agents succeed and where they fail. Read against this, the climb from 228% to 2346% documented above is real progress on a specific set of skills. It is not evidence that these agents are broadly superhuman, and treating the aggregate score as proof of that is where the field's reporting habits get sloppy.

What it took for Agent57 to clear even the weak "all 57 games" threshold

DeepMind's Agent57, released in 2020, was the first deep RL agent to beat the human baseline on all 57 games in the suite. Every agent before it had a persistent blind spot: strong average performance, but a handful of games it simply couldn't crack no matter how it was tuned.

Four games did the damage. Montezuma's Revenge and Pitfall both demand a long stretch of exploration before any reward shows up at all, and Pitfall goes a step further by handing out negative rewards that actively punish the kind of wandering an agent would need to do to find the good stuff. An agent exploring at random is, mathematically, almost never going to stumble into the right sequence of actions by chance. Solaris and Skiing pose a different problem: long-horizon credit assignment, where an agent has to string together information across a long stretch of time before it gets any usable signal about whether its choices were good ones.

Agent57 solved this by combining an exploration algorithm built to search efficiently with a meta-controller that adjusts, on the fly, how much the agent explores versus exploits. It's a genuinely clever piece of engineering. But it comes with a catch that coverage of the result tends to bury: Agent57 doesn't learn all 57 games at once. It retrains from scratch for each one, reusing the same algorithm but starting from zero every time a new game comes up. Clearing the human-baseline bar, one game at a time, against a casual tester, is a real result. It's a narrower one than the "superhuman across 57 games" framing suggests, and it raises the next question directly: if even the full-budget version of this result needs that many caveats, what happens once the data budget gets tight?

The Atari 100k sub-benchmark and the separate problem of sample efficiency

Atari 100k answers that question by restricting agents to exactly 100,000 environment interactions per game, roughly two hours of real-time play, across 26 games with discrete action spaces. Full-budget ALE results, the kind Agent57 posted, can involve agents playing for the equivalent of years of continuous gameplay, a scale that swamps any measure of learning efficiency. That's fine for measuring what's possible with enough compute. It says almost nothing about how efficiently an agent learns, and efficiency is a critical question for anyone thinking about real-world deployment.

The early results were rough. Rainbow, a strong performer under the standard full-budget setup, scored a normalized mean of just 0.10 at 100k steps. SimPLe, a 2019 model-based approach from Kaiser and colleagues that used a learned video-prediction model to simulate extra training data, improved that to 0.35, still well short of human-level play.

EfficientZero, published by Ye and colleagues in 2021, changed the picture. Built on top of MuZero, it adds self-supervised consistency objectives and an off-policy correction mechanism, and it reached 194.3% mean human performance and 109.0% median performance within the same 100,000-step budget, beating the human benchmark outright in 14 of the 26 games. It was the first agent to clear human mean and median performance under that constraint, a genuinely significant jump from where SimPLe had left things two years earlier.

The Atari 100k topic survey, updated in January 2026, lists newer entries building on that momentum, including the model-based methods DIAMOND and EfficientZero V2, and other agents pushing the benchmark further. DIAMOND complicates the "sample efficient" story in an important way. It uses a conditional diffusion model working directly on pixel frames and posted a mean human-normalized score of 1.46 in Alonso and colleagues' 2024 paper. But each training run took roughly 2.9 days on a single Nvidia RTX 4090, and running five seeds per game across the full benchmark added up to about 1.03 GPU-years of compute. The agent needed few environment interactions, but the compute bill to squeeze value out of each one was steep. Sample efficiency and computational efficiency are not the same currency, and conflating them is a mistake anyone comparing these results needs to avoid.

Why ALE results are harder to compare than they look

As more labs adopted ALE, they didn't all adopt the same rules for using it, and the differences aren't cosmetic. Some groups end an episode the moment an agent loses a single life. Others let play continue until the game shows a full "Game Over" screen. That single choice changes difficulty and score distributions enough to make head-to-head comparisons across papers unreliable unless both groups made the same call. Varying preprocessing pipelines, different environment parameter settings, and inconsistent checkpoints for when results get reported mean two papers claiming similar scores may not be measuring the same thing.

The reporting mess has a deeper problem: the original Atari 2600 games are fully deterministic. The original Atari 2600 games are highly deterministic, meaning fixed input sequences can produce consistent, reproducible outcomes. That let early agents cheat, in a sense, by memorizing a fixed open-loop sequence of button presses that happened to work, rather than learning a policy that could actually respond to what was on screen. Machado and colleagues fixed this in 2018 by introducing "sticky actions," where each action has some controllable chance of repeating instead of executing immediately, injecting enough randomness that memorized sequences stop working. The updated ALE now supports several game modes alongside that fix.

Standardized protocols, a norm of publishing code alongside results, and reporting at fixed, protocol-verified checkpoints rather than whatever point looks best exist as fixes for comparing results and are gaining ground. The ALE topic survey on Emergent Mind treats these as necessary practice, not nice-to-haves. Some of what looks like algorithmic progress in the published literature is actually just a difference in evaluation protocol, and the field often can't tell the two apart after the fact.

What ALE does not test: generalization, robustness, and shortcut learning

Standard ALE environments never change. Same visual assets, same level layout, same rules, every single episode. That means an agent trained and tested on the same fixed version of a game can never be checked for actual generalization within that game, because it never sees a version that's even slightly different.

HackAtari, published by Delfosse, Blüml, Gregori, Kersting and colleagues at the Technical University of Darmstadt in June 2024 (arXiv:2406.03997), was built to close that gap. It introduces controlled variation across 50 modifications spanning 16 Atari environments: color swaps, hidden game elements, altered reward signals, and simplified or harder versions of the same game meant for curriculum-style training. The results are not flattering to standard training. Agents trained the normal way show clear robustness failures the moment even small perturbations show up.

The paper's Pong example is the clearest illustration. An agent trained normally learns to track the enemy paddle's position rather than the ball itself, because the enemy paddle position correlates well enough with where the ball is going to produce a working policy during standard training. Importance maps make the agent look like it understands the game. Hide the enemy paddle, though, and the policy falls apart completely. That's shortcut learning, sometimes called goal misgeneralization: the agent found a shortcut correlated with the real objective and locked onto that instead of learning the objective itself.

A separate, older finding compounds the problem. Well-tuned classical function approximators can sometimes match deep neural networks on many ALE games, often needing less data to get there. That raises an uncomfortable question about what deep RL is actually contributing on tasks where a much simpler linear model gets you most of the way. A more recent test pushes the question further into large language models. TextAtari, published by Li and colleagues in June 2025, converts Atari games into pure text interfaces and evaluates LLMs on them. Without external help, game manuals or demonstration examples fed in as priors, LLM performance lags far behind human play, despite these models' general reasoning strength on other tasks. ALE's own documentation has long acknowledged its lane: strong for testing perception and fast eye-hand coordination, weaker for testing planning. HackAtari and efforts like it are attempts to stretch that lane wider, not replacements for the platform itself.

The four open challenges that keep ALE from being "solved" under rigorous criteria

Fan's paper, working from human world record benchmarks, identifies four open problems standing between current agents and anything that deserves to be called genuinely superhuman.

The first is a plain performance gap: even the best agents available today fall short of world-record human scores on the games where it matters most. The second is learning efficiency. Agents like Agent57 and the MuZero family need vastly more environment interactions than any human would to reach comparable scores, and Fan argues that a truly superhuman agent needs to beat humans on efficiency, not just on final score, otherwise the comparison isn't fair to begin with. Third, hard-exploration games, Montezuma's Revenge and Pitfall chief among them, remain largely unsolved for general-purpose agents working within anything close to a human-sized data budget, for the same mathematical reason noted earlier: random exploration essentially never finds the right sequence on its own. Fourth, off-policy stability remains an open research problem. Off-policy methods paired with function approximation are known to diverge or behave unpredictably, and a fully reliable, scalable fix hasn't landed yet.

These four problems share a structural cause. The standard HNS metric, averaged across all 57 games, lets an agent's enormous score on easy games paper over its failures on the hard ones, and the average hides exactly the profile a reader would need to see to judge where an agent actually stands. None of this means ALE has failed as a benchmark. It means the challenges the benchmark surfaces hold up under scrutiny, the benchmark is the thing that exposed them in the first place, and the single headline number the field keeps repeating is the part obscuring the picture.

Newer evaluation frameworks trying to close the gaps ALE leaves open

Three directions of work respond to these gaps directly, and each targets a specific weak point rather than trying to replace ALE. Harder human baselines, following Fan's world-record proposal, aim to fix the "superhuman" measurement problem at its root by raising the bar to something that actually reflects the ceiling of human skill rather than a brief professional tester's session. Controlled-variation frameworks like HackAtari test whether an agent's policy survives changes it was never trained on, addressing the generalization and shortcut-learning gap. And standardized reporting protocols, including SABER along with mandatory code release and fixed evaluation checkpoints, aim at making results comparable, so that a claimed improvement reflects a better algorithm and not just a looser evaluation setup. None of these frameworks make ALE obsolete. They make it possible to trust what its numbers are actually saying.