Pretraining on Game Data for Downstream Sequential Decision Tasks

Game data offers an unusually practical foundation for pretraining agents on sequential decision tasks because it combines scalable trajectories, verifiable rewards, and recoverable action labels. Its value is not a ready-made general policy. Rather, games build behavioral skills, shared representations, and reasoning policies that can make downstream learning tractable, provided fine-tuning preserves useful knowledge and addresses the gap between the game and the target task.

What makes sequential decision tasks hard to pretrain on in general

A sequential decision changes the problem that follows it. Unlike one-shot prediction, each action reshapes the environment and helps determine what the agent observes next. A policy must link observations and decisions to what happens next over time, beyond mapping an input to an output. The difficulty grows when observations reveal only part of the state, or when the agent does not know which model of the world is correct.

Medicine, resource allocation, and robotics have plentiful logged data, but live experiments can be costly or unethical. Training a transformer from scratch also gives up the broad representations available in a pretrained model, while existing approaches often remain confined to fully observable settings. The practical bottleneck is labeled trajectory data at scale, including not just observations but records of which decisions were made and why.

Why game environments are a structurally privileged source of pretraining data

Games have three things that rarely come together: large numbers of cheap trajectories, built-in verifiable rewards, and action labels that can be attached to frames. Compute becomes the main constraint on generating interactions. Game environments can run faster than real time and in parallel, producing data without the cost of gathering equivalent experience in the physical world.

The game engine also defines success unambiguously. Reward signals need no human annotation, and their definition does not depend on assuming that every relevant confounder has been observed in a real-world offline dataset. Controller inputs provide another kind of supervision by linking visible changes to the actions that produced them. Online gameplay footage may lack those inputs, but recordings that pair them with footage can help models learn to fill in missing action labels.

Other data sources usually supply only part of this combination. Robotics has records of actions but is hard to scale, while internet video has plenty of footage without action labels. Synthetic tasks can provide verifiable outcomes without the richness of game observation spaces. The advantage extends well beyond Atari pixels. Minecraft, CS:GO, and arcade-style multimodal games differ in how they look and how players act in them, so game data spans more than a single benchmark format.

How behavioral pretraining from raw gameplay video works — the VPT approach

OpenAI’s VPT work, presented at NeurIPS 2022, turns a limited amount of annotated Minecraft play into supervision for a much larger video corpus. Contractors record both their screens and their inputs. An inverse dynamics model, or IDM, learns to infer actions from frames next to each other, then pseudo-labels footage that has no input records. VPT trains its behavioral cloning foundation model on 70,000 hours of IDM-labeled online video.

The resulting model acquires useful behavior before task-specific training. It can craft planks and crafting tables zero-shot, tasks requiring a proficient human a median of roughly 970 consecutive actions. The model has learned sequences of behavior that are nearly impossible to acquire through reinforcement learning from scratch. Still, all three VPT variants need fine-tuning to follow instructions, so knowing how to act leaves them short of task-directed control.

RL fine-tuning turns that competence into a sharper result. Starting from the foundation model produces the first agent to craft diamond tools in Minecraft, succeeding in a meaningful fraction of 10-minute episodes and matching human-level success rates on all prerequisite items. The diamond pickaxe result establishes the role of the prior. Pretraining supplies skills that make downstream RL tractable, while RL organizes those skills around an objective that behavioral cloning alone does not reliably achieve.

How joint world-action pretraining across multiple games transfers to new ones — the JOWA approach

JOWA, presented at ICLR 2025, extends the pretraining argument from one game to a distribution of games. It jointly trains a world model and an action model on multiple Atari games using billions of tokens. A shared transformer backbone connects the two objectives and stabilizes temporal difference learning at large model scales. The distribution is the important distinction from single-game training. Shared representations must work across different dynamics, beyond the rewards or quirks in the observations of any one game.

The strongest evidence comes from the held-out games. JOWA fine-tunes to new games using approximately 4 trajectories per game, which shows how much the pretrained model has already learned. The downstream data need not support learning the entire decision process from scratch. On 15 Atari games, the model also substantially exceeds contemporaneous model-based agents while using only a fraction of the available data.

MHWA (2026) provides a compute-efficient replication of this direction. It matches JOWA’s largest held-out performance with far fewer training FLOPs per sequence, indicating that the transfer result is not an artifact of scale alone. Capacity still has a predictable relationship with downstream performance. The evidence therefore supports both the value of multi-game pretraining and the possibility of obtaining its benefits more efficiently.

How games train reasoning skills that transfer beyond games — the ViGaL and GameQA findings

Visual Game Learning, or ViGaL, presented at ICLR 2026, has a different edge over specialist fine-tuning. It improves mathematical reasoning while preserving general visual benchmark performance, where models post-trained on benchmark-oriented data often degrade. Multimodal large language models play arcade-style games such as Snake and Rotation through RL, producing a 2.9% average accuracy improvement on mathematical reasoning benchmarks and surpassing specialized models trained on math data.

Games demand look-ahead planning, spatial reasoning, and coordinate tracking, which can be useful outside the environment that exercises them. Coordinate recognition learned through Snake, for example, transfers to visual math tasks. ViGaL’s improvements hold across diverse game mechanics, suggesting that the transferable resource is a learned reasoning policy. Although ViGaL is a post-training method, it demonstrates how gameplay can prepare capabilities for later use in another domain.

GameQA, also presented at ICLR 2026, broadens that evidence. RL training solely on GameQA enables multiple vision-language models to generalize across out-of-domain vision-language benchmarks. Increasing either game data volume or game diversity consistently improves both generalizable reasoning and game performance. As in language-model pretraining, a varied corpus exposes the model to a wider range of demands, although here the diversity includes the reasoning needed to act.

What determines whether pretrained game knowledge actually transfers to a downstream task

Transfer depends first on what the pretext game rewards. ViGaL’s ablations show that reward design shapes which cognitive skills develop, with distinct games unlocking different skills. RL fine-tuning also outperforms supervised fine-tuning in developing transferable capabilities. A precisely scored objective can still cultivate behavior that is poorly matched to the downstream task.

Directly applying standard RL fine-tuning to a pretrained policy can cause rapid behavioral degradation, driven by mismatches in rewards and additional observations available only downstream. Adaptation should layer new behavior onto the prior. When the gap is wide, constraining each update to a local region of the pretrained policy space is necessary to prevent skill loss.

Observation mismatch imposes a separate constraint on transfer. Microsoft Research’s visual encoder study, presented at ALA 2025, finds that pretrained encoders such as DINOv2 outperform end-to-end trained encoders for imitation learning in Minecraft and Minecraft Dungeons, but the benefit varies by game. Variation in how well natural-image representations handle the shift to game pixels makes the observation interface part of the diagnosis.

Partial observability leaves a deeper question unresolved today. Work on in-context decision-making with pretrained models tends to stay within fully observable MDPs, so it is still unclear whether game pretraining helps when state is hidden or the world model is ambiguous. Fine-tuned LLM theory draws a useful distinction but leaves that question open. More in-context demonstrations sharpen test-time inference, while more pretraining data reduces systematic bias acquired during pretraining. Those levers do different jobs and cannot substitute for each other.

The strongest objection — game pretraining produces game-specific biases, not universal priors

A policy learned from Minecraft or Atari is shaped by those games’ rewards, observation spaces, and ways of interacting. If downstream learning must first undo those habits, pretraining may consume part of the advantage it appears to offer. VPT’s performance degrades on objectives such as obtaining diamonds without fine-tuning, precisely where a broadly useful behavioral prior might be expected to provide the greatest help.

This objection limits claims of universality, but not usefulness. The successful diamond-tool fine-tuning shows that reusable skill components can be valuable even when the original policy cannot complete the target task. ViGaL’s reasoning transfer also shows how useful knowledge can carry over apart from game-specific behavior, while JOWA shows how training across a game distribution makes a model less tied to the conventions of any one game. These findings support transfer through representations, reasoning policies, and component skills. They do not establish that a game-trained policy is ready for arbitrary deployment.

The uncertainty is greater when downstream assumptions change. A new reward or visual environment is not the same challenge as hidden state or uncertainty about which world model applies. Evidence of successful adaptation within the former settings does not settle the latter. The evidence therefore supports game pretraining as a strong initialization for downstream learning.

Where the field is moving — blurring the line between pretraining and reinforcement learning

Reinforcement Pre-Training, or RPT (2025), makes the separation between pretraining and RL less fixed. It integrates scalable self-supervised pretraining with RL directly, producing a stronger foundation for subsequent fine-tuning and improving zero-shot performance. RL is no longer only the downstream adaptation step. This direction extends the role of interactive environments from supplying an initial corpus to shaping the foundation model during its construction.

Self-Play Pretraining with Zero Data (2026) pushes that boundary further. Two models co-evolve, with a generator proposing synthetic data and a learner training on it. Zero-shot loss on natural data improves in a predictable way as self-play compute grows, and the models find recognizable mathematical sequences without natural data. This is an existence proof of a route toward self-sustaining pretraining. It is not evidence that external data or downstream adaptation has become unnecessary.

GameQA’s scaling results make the game distribution itself a design variable in this broader training process. Its range of mechanics and volume of interactions influence the reasoning that develops, rather than merely determining how much data is available. As pretraining and RL converge, cheap trajectories, verifiable rewards, and action labels can support more of the training pipeline. The unresolved tests remain demanding and specific. Partial observability, model ambiguity, and long-horizon instruction-following without fine-tuning are still limits that the next round of work must tackle.