Sample Efficiency Comparisons Across Major RL Algorithms on Atari 100K

The Atari 100K benchmark, introduced by Kaiser and colleagues in 2019, caps an agent at 100,000 environment interactions, which works out to 400,000 frames under the default action-repeat setting. That's roughly 1.85 hours of play time, a constraint that shapes the human-normalized scoring at the benchmark's center. Standard Atari training runs closer to 50 million steps, so the 100K setting asks a genuinely different question: not how good an agent can eventually get, but how much it can learn from almost nothing.

The benchmark, defined plainly

That 500-fold gap between 100K and 50 million steps is why older brute-force results never told anyone much about sample efficiency. An agent that needs 50 million frames to beat a human isn't demonstrating learning efficiency, it's demonstrating that enough repetition eventually works. The 100K setting removes that crutch. It forces every design choice, from the replay buffer to the loss function, to justify itself under real scarcity.

Performance gets scored with human-normalized score (HNS): agent score minus random-play score, divided by human score minus random-play score. A score below zero means the agent is worse than random button-mashing. A score above 1.0 means it beat the human tester. This single number lets you compare a chess-playing tree search against a pixel-prediction network against a plain value-based learner, because it rescales everything to the same human-relative yardstick.

But a single point estimate, mean or median, hides more than it reveals. Atari 100K results are noisy enough, and skewed enough by a handful of outlier games, that mean and median comparisons alone can give an incomplete or misleading picture. The interquartile mean (IQM) addresses this by dropping the top and bottom quarters of per-game scores and averaging what's left. It's a trimmed mean, essentially, and it resists distortion from one game where an agent gets lucky or unlucky. Alongside IQM, papers now report the "optimality gap" (how far below human-level performance the agent still sits) and stratified bootstrap confidence intervals, so a claimed improvement can be checked against the noise floor rather than taken on faith.

The standard evaluation protocol runs an agent across 26 or 55 games from a standard game-based benchmark suite, with 100 evaluation episodes and five random seeds per game, a frame-skip of 4, and up to 30 no-op starts to prevent the agent from memorizing a fixed opening sequence. That's the frame this whole comparison sits inside.

The pre-2021 starting point: where DQN, Rainbow, SimPLe, DER, and CURL landed

DQN and Rainbow, the model-free workhorses that had defined Atari research for years, needed millions of steps to reach competence. Dropped into the 100K budget, they performed near random, sometimes below it. That failure illustrates the benchmark's central purpose: to surface how much agents rely on sheer volume of experience rather than efficient learning.

SimPLe (Kaiser et al., 2019/2020) was the first serious model-based answer. It trained a video-prediction model of the game and let a PPO policy learn inside that imagined environment rather than the real one. The result, a mean HNS of 0.35 against Rainbow's 0.10 at the same budget, was a multiple-fold jump that gave the model-based direction real credibility, even though both numbers were still far below human level.

Data-Efficient Rainbow, or DER (Van Hasselt et al., 2019), showed that retuning the hyperparameters of an existing model-free agent, without inventing anything new, could beat prior methods. That's a methodological lesson that outlasted the specific numbers. Other model-free refinements pushed the same line further and set the stage for what came next.

CURL (Laskin et al.) brought contrastive learning to the table, using augmented observations to train richer representations. Its median HNS of 17.5% beat SimPLe's 14.4% and Efficient Rainbow DQN's 16.1%. But look at the mean instead, and the story flips: SimPLe's 44.3% beats CURL's 38.1%, which beats Efficient Rainbow DQN's 28.5%. That inversion between mean and median isn't a typo, it's the exact problem Agarwal's group later formalized: a couple of outlier games can swing a mean without moving the median at all, and neither number alone tells you which method is actually better on balance.

DrQ (Kostrikov et al.) skipped the contrastive objective entirely and applied image augmentations as a regularizer on top of standard value-based learning. That alone pushed median HNS from 5.8% (Efficient DQN) to 26.8%. No auxiliary loss, no learned world model, just better-regularized pixels.

SPR (Schwarzer et al.) combined augmentation with a self-predictive objective, training the network to predict its own future latent representations. That future-prediction module pushed SPR to a mean HNS of 70.4%, the highest of any model-free method at the time. It also marked something like a ceiling. Contrastive learning plus augmentation plus self-prediction had taken model-free Atari as far as it seemed to go, and it still left every agent short of human level. Getting past that ceiling took a different kind of change.

EfficientZero: the first superhuman result

EfficientZero (Ye et al., 2021) broke through that ceiling, and it did so with an architecture, not just a bigger training run. Built on MuZero, it added self-supervised consistency objectives and other architectural refinements to the underlying MuZero framework.

The result: mean HNS of 194.3%, median HNS of 109.0%, or 1.904 and 1.160 in raw score terms. That's the first time any agent crossed human-level performance under the 100K budget, and it beat human testers outright in 14 of the 26 games tested. Against SPR, the strongest model-free agent, EfficientZero scored 170% higher on the mean and 180% higher on the median, a jump in kind, not degree.

Perhaps the more striking comparison is against full-scale DQN, trained on many times the frame budget: EfficientZero reached roughly the same performance using about 500 times less data. That's the core argument for architecture over brute force. The gain did not come from some clever shortcut through the data; the architecture's combination of a learned world model and tree-search planning extracted far more signal from each interaction than prior approaches had managed.

That result raised an obvious question for the field: EfficientZero needed MCTS at inference time to hit these numbers. Search is expensive computationally, and it introduces a serious asterisk. If superhuman performance can only be reached by planning ahead with a tree search, that's a meaningfully different claim than reaching it through representation learning alone. Two lines of research picked up that question from opposite directions: one asking whether a learned world model could get there without search, the other asking whether a model-free agent could get there without any world model.

The world-model line without lookahead search: IRIS, STORM, DreamerV3, Δ-IRIS, DART, and DIAMOND

All the figures below are mean HNS across 26 games, reported for agents that skip lookahead search entirely, so this is a fair comparison within one family: models that imagine forward but don't plan with a tree search at inference time.

IRIS (Micheli et al., 2023) used a transformer as its world model, built on discrete latent representations of each frame. It reached a mean HNS of 1.046, an IQM of 0.501, and an optimality gap of 0.512, beating humans in 10 of 26 games. Compared to SPR, that's a 70% gain on mean, 49% on IQM, and 11% on optimality gap. The IQM and optimality gap numbers are statistically sturdier figures, less prone to distortion from one or two standout games.

STORM (Zhang et al., 2023) pushed further, reaching a mean HNS of 1.266, the strongest result reported among no-search world-model agents in the diffusion-comparison literature. DreamerV3 (Hafner et al., 2023) landed at 1.097, a figure that matters less for its Atari ranking and more because DreamerV3 became the general-purpose baseline that EfficientZero V2 was later checked against across a much wider set of tasks.

TWM (Robine et al., 2023) took a different route to the same neighborhood, combining transformer-based dynamics modeling with techniques designed to improve training stability and sample quality. Its headline number has since been overtaken, but it belongs in the lineage as part of the same transformer-world-model wave.

Δ-IRIS (Micheli et al., 2024) is really an efficiency story more than a scoring one. It topped IRIS on aggregate metrics while training five times faster than its predecessor. DART used a tokenized representation with transformer components, reporting a median HNS of 0.790 and beating humans in 9 of 26 games; note that DART's headline figure is a median, not a mean, so it isn't directly stackable against the IRIS or STORM numbers above without that adjustment in mind.

DIAMOND (Alonso et al., 2024, presented at NeurIPS 2024) is the standout of this group by raw mean HNS: 1.46, the highest reported among agents trained entirely inside a world model with no search involved. It gets there with a diffusion model built on the EDM framework rather than the more familiar DDPM approach, because DDPM becomes unstable when generating frames autoregressively with only a handful of denoising steps. The idea behind DIAMOND is straightforward even if the machinery isn't: sharper, more detailed visual predictions should translate into a better policy, since a blurry or degraded world model teaches the agent bad habits it later has to unlearn.

That quality costs real compute. DIAMOND has substantial compute requirements, demanding significant VRAM and multi-day runtimes on a single GPU per run. The two approaches differ meaningfully in the wall-clock time required to reach the same number of environment steps, a gap that is relevant when weighing the cost of DIAMOND's stronger results. More recent work in the same diffusion-world-model direction suggests there are still real efficiency gains left on the table.

Line the mean HNS scores up: IRIS at 1.046, STORM at 1.266, DIAMOND at 1.46. The trend climbs steadily, but so does training cost. There's no free lunch here, wall-clock time tracks the performance gains almost step for step.

BBF: how model-free scaling reached superhuman performance without imagination or planning

BBF (Schwarzer et al., 2023, ICML 2023) asked whether a world model is required for superhuman Atari 100K performance. BBF says no. It's a value-based agent, no imagined rollouts, no planning, and it still clears human-level performance.

The gains come from a handful of distinct design choices, not one big idea. BBF widens and deepens the network, uses adaptive multi-step return targets instead of fixed-horizon bootstrapping, periodically resets parts of the network to fight overfitting on the tiny replay buffer, applies strong weight decay, and folds in the same SPR-style self-supervised auxiliary loss that drove the earlier model-free ceiling. Each of these solves a different problem: resets fight stale representations, weight decay fights overfitting, and the SPR loss supplies a learning signal beyond the sparse reward.

Work by Çagatan and Akgün found that SPR's published description of its self-supervised loss doesn't quite match what its code actually did: two undocumented tweaks, masking the SSL loss at episode bo... Work by Çagatan and Akgün found that SPR's published description of its self-supervised loss doesn't quite match what its code actually did: two undocumented tweaks, masking the SSL loss at episode boundaries and reweighting samples by their priority in the replay buffer, were quietly carried forward into every method that reused SPR's codebase, BBF included. Stripping both changes out of SPR drops its IQM by 18%. The two undocumented tweaks reduce IQM proportionally in BBF and SR-SPR too, even though both post substantially higher IQM scores than SPR to begin with. That's a real reproducibility issue, not a minor footnote, and the next section digs into what it means for how the leaderboard should be read.

On raw efficiency, BBF is hard to beat: it reaches superhuman performance in about 6 hours on a single GPU, roughly double SR-SPR's human-normalized IQM at nearly the same computational cost, and gets to the performance level of a top model-based baseline at four times less runtime or better. That's the practical argument for the model-free path: no learned dynamics model to train, no tree search to run at inference time, just a well-tuned value network and a handful of training tricks.

The catch is scope. BBF's result is a 26-game Atari benchmark, full stop. It wasn't built to generalize across task families the way DreamerV3 or EfficientZero V2 were later tested to do, and that narrower ambition shouldn't be read as a weakness so much as a different bet: go deep on one benchmark rather than wide across several.

SSL objectives inside SPR, SR-SPR, and BBF: the undocumented modifications that shaped the leaderboard

The Çagatan and Akgün finding, from Koç University, deserves its own space because it complicates how the whole 2021-2023 stretch of results should be read. SPR's terminal-state masking and prioritized-replay weighting weren't described in the original paper, yet every method that built on SPR's code, including SR-SPR and BBF, inherited them without documenting that inheritance either.

Removing both modifications drops SPR's IQM by 18%. That's not a rounding error, and the same relative hit occurs in SR-SPR and BBF, whose base IQM scores run roughly three and two times higher than SPR's respectively. So the modifications aren't just background noise, they're doing real work, work that wasn't visible in any of the published method descriptions.

There's a useful counterpoint buried in the same research: swapping in a feature-decorrelation objective from another self-supervised method, with none of SPR's domain-specific masking or weighting tricks, comes within 5% of SPR's original performance. That's a strong hint that the modifications are patching over a weakness in the base contrastive objective rather than adding something fundamentally necessary. A better-chosen loss function might not need the patches.

What this means for reading the leaderboard: some of the performance gap separating methods in the 2021-2023 window reflects undocumented implementation choices, not algorithmic invention. Researchers reusing these codebases picked up the masking and weighting tricks by default, often without knowing how much of their reported gain traced back to them. Going forward, SSL objective choices and RL-specific adaptations like masking and replay weighting deserve the same ablation scrutiny that architecture choices already get. Treating an SSL loss as a fixed, interchangeable component, rather than something that needs its own controlled comparison, is how a field ends up with a leaderboard nobody can fully explain.

EfficientZero V2: MCTS-based search at 100K steps and the current performance ceiling

EfficientZero V2 (Wang et al., 2024, an ICML 2024 Spotlight paper) is currently at the top of the Atari 100K leaderboard, with a mean HNS of 2.428 and a median of 1.286, ahead of both BBF and the original EfficientZero.

What makes that result notable isn't just the score; it's how it was reached. EZ-V2 uses fewer network parameters and a lower replay ratio than BBF, so the gain isn't coming from throwing more compute at the same idea. It comes from the environment model paired with Gumbel search during action selection, a refinement of the MCTS approach that made the original EfficientZero work.

The breadth of the result matters too. The ICML 2024 proceedings report that EZ-V2 beats DreamerV3 on 50 of 66 tasks spanning Atari 100K, Proprio Control, and Vision Control. That positions it as a general-purpose algorithm being tested across domains. According to a source (arXiv 2507.04075), one leading model-based method still holds the state-of-the-art spot on Atari 100K, using a tree-search planning method to pick the best action at every single step.

That search isn't free. It adds real inference-time compute that BBF and the no-lookahead world-model methods simply don't pay. Whether that trade-off is worth it depends entirely on the deployment context, this is a design choice with real costs on both sides, not a settled argument that search always wins.

A harder question follows from this: once mean HNS climbs past 2.0 and toward 2.4, how much of the remaining gap between top methods reflects genuine differences in capability, versus noise from a handful of games where one method got lu... Once mean HNS climbs past 2.0 and toward 2.4, how much of the remaining gap between top methods reflects genuine differences in capability, versus noise from a handful of games where one method got lucky? A statistical reporting approach built around robust central-tendency measures and confidence intervals exists precisely to answer that question, but it hasn't been applied uniformly across every paper cited here. Some of the horse-race framing in this exact section should be read with appropriate caution.

What the performance gaps reveal about which architectural choices drive sample efficiency

Three separate paths reached superhuman Atari 100K performance, and they got there by different routes: search in latent space (EfficientZero, EZ-V2), world-model imagination without search (IRIS, STORM, DIAMOND), and model-free value-network scaling (BBF). Each trades compute, wall-clock time, and generality differently, and none of the three has made the other two obsolete.

Across all three families, self-supervised auxiliary losses help, data augmentation as a regularizer helps, and moving past decade-old convolutional architectures helps. Those three ingredients recur in nearly every method that cleared the ceiling SPR set in the model-free era, from BBF's inherited SPR loss to EfficientZero's consistency objectives to the transformer and diffusion backbones running the world-model line.

What clearly didn't generalize is the idea that hyperparameter tuning alone, without any of those structural additions, could keep closing the gap. DER proved tuning could beat a stale baseline once. It never proved tuning could substitute for the auxiliary losses, augmentation, and architectural shifts that actually did the work in every method that followed. The Çagatan and Akgün finding sharpens that point further: even the auxiliary losses that did work carried undocumented implementation details doing a meaningful share of the lifting. The field's next real gain in sample efficiency probably won't come from a bigger network or a longer training run. It will come from someone finally isolating which piece of the recipe was actually doing the work all along.