Transformer Architectures as Policy Networks in Game Domains

Transformer architectures succeed as policy networks in game domains because self-attention matches the structure of the decisions games require. It connects distant events and board positions without forcing their relationships through a recurrent memory bottleneck. Results from chess, Atari, and offline reinforcement learning support that fit, though online adaptation is still a major limit.

Why game states break feed-forward and recurrent architectures

Game states call for three linked things: handling long-range dependencies, dealing with partial observability, and reasoning about non-adjacent events. A move can matter because of something that happened many moves earlier or because of a position far across the board. With partial observability, the current observation may lack the information the policy needs to choose an action. A policy must therefore do more than recognize what is immediately visible. It must retain context and connect information separated in space or time.

Feed-forward networks are stateless by design, so processing an observation alone gives them no mechanism for relating it to earlier states. RNNs and LSTMs process sequences, but they propagate context through a fixed-width hidden vector. Information from earlier steps must make it through compression again and again before it can help with a later decision. Vanishing or exploding gradients make learning those dependencies harder over long horizons.

Wang et al. (2023) demonstrate a related spatial failure through a cyclic-adversary program that successfully attacks the convolutional KataGo at a high rate. KataGo fails to evaluate large circular groups globally across the Go board. Ladder patterns also call for reasoning across the whole board, beyond spotting a pattern in one part of it. Both are standard parts of strategy that human players handle easily in normal play. These failures expose a mismatch between local processing and global evaluation, rather than a shortfall that more training time alone can be assumed to remove.

What self-attention gives a policy network that recurrence cannot

Each position can attend to every other available position in a single pass, allowing distant information to interact without first passing through a fixed-width recurrent state. The operation, softmax(QKᵀ/√d_k)V, computes pairwise relevance across positions and uses it to combine their information. For a policy network, this means that the information relevant to an action can be selected from across the available context. Distance no longer dictates how many recurrent steps separate two events.

Multi-head attention lets the model learn different kinds of links at the same time. In chess, one head might track piece mobility while another keeps track of king safety. These are examples of possible roles for separate heads. BT4 uses 64 tokens, one per chessboard square, so attention can directly relate a rook on a1 to a pawn on h7.

That access is an architectural advantage. Can Go AIs Be Adversarially Robust?, presented at AAAI, shows that even Vision Transformer-based victims remain vulnerable to attacks. Attention supplies a route for combining distant information, but training must still produce a policy that uses it correctly. Basic self-attention also scales quadratically with sequence length. That cost is manageable for board-sized inputs and fixed Atari frame stacks, but matters when the policy needs very long contexts.

How transformers replaced convolutions as the backbone of chess engines

Chessformer, accepted at ICLR 2026, gives a concrete account of the move from convolutional to transformer backbones in Leela Chess Zero. Its BT4 architecture has 15 transformer layers with residual connections and 32 attention heads in each multi-head self-attention layer. Each layer operates on the board-square tokens, bringing global relationships into the backbone rather than leaving them to emerge through local processing. Chessformer reports that BT4’s raw policy is 270 Elo stronger than T78, which was the strongest convolutional model in the LC0 family. It achieves that gain with fewer parameters and less computation, separating the architectural improvement from a simple increase in scale.

Chessformer also reports an increase of over 100 Elo points in Leela’s overall playing strength. That is a different comparison from the raw-policy result and measures the gain in the complete engine. Major chess engine releases rarely surpass their predecessors by more than 50 Elo points, making the reported margin substantial. Chessformer describes Stockfish as the “perennial world champion” and reports that the resulting configurations defeated it in select elite computer chess tournaments. Those wins show that it can compete at that level, though it may fare differently in other tournaments or configurations.

Monroe and Chalmers (2024) offer more evidence with ViT policy and value networks that they trained by supervised learning from self-play. They achieved higher win rates when using either the policy network or the value network alone. Ju et al. at Academia Sinica extend the pattern beyond chess with ResTNet, which interleaves transformer and residual blocks within an AlphaZero framework. ResTNet improves win rates over pure-convolutional baselines in 9x9 Go, 19x19 Go, and 19x19 Hex, while reducing errors on ladder and circular patterns. Gains across these games suggest that processing information from the whole board helps beyond what tuning for chess alone can do.

Decision Transformer: when policy learning is recast as sequence prediction

Chen et al. (2021) use the Decision Transformer to bring this ability to link distant context to a different learning problem. Rather than learning through value-function optimization, the model learns to predict actions from recorded trajectories and a desired outcome. It can generate near-expert actions without computing Bellman backups. The distinction is not merely a change of network backbone. It changes the object being learned from estimates of value to actions conditioned on a target return.

That target, expressed as return-to-go, or RTG, is the total reward the agent wants from the current point forward. The input contains a history of RTG, state, and action tuples. During learning, this structure connects behavior in recorded trajectories with the returns associated with it. The target therefore tells the model what outcome to aim for, rather than certifying that it can achieve that outcome.

A decoder-only transformer naturally fits this task because it generates the next action from the preceding context, much as a language model generates the next token. The history gives the policy more information than a single observation, which is especially useful when the environment is only partly observable. It also lets the model associate earlier actions with later outcomes across a long trajectory. This offers a distinct approach to sparse rewards and long-horizon credit assignment without relying on bootstrapped value estimates. Its power, however, depends on what relationships the recorded trajectories make available to learn.

Multi-game generalization: one model, many games, near-human performance

Lee et al. (2022) use the Multi-Game Decision Transformer to model trajectories across games. A single set of weights, trained purely offline, plays up to 46 Atari games. The observations and action sequences differ from game to game, but the model uses the same learned architecture to process them. Attention over tokenized trajectories lets the model pick out the context it needs in the same way across games, without a reasoning mechanism for each one.

The training data has both expert and non-expert trajectories. This mixture matters because indiscriminate imitation would treat successful and unsuccessful behavior alike. RTG conditioning gives the model a way to distinguish actions associated with different outcomes. The multi-game result therefore supports more than breadth of recognition. It shows that one conditional sequence model can learn how to play from data that spans games and levels of play.

DeepMind’s Gato (2022) carries the shared-model principle beyond games. Trained on 604 distinct tasks, it plays Atari, captions images, chats, and controls a real robot arm with the same network and weights. Context determines what type of output it produces. This is evidence that tokenized context can support a common policy interface across very different tasks. For game domains, the important implication is that breadth need not come from an ensemble of game-specific networks.

Where offline power ends: the online adaptation problem

The conditional imitation that helps the Decision Transformer work well offline also makes online learning hard. A policy trained on recorded trajectories must now explore and learn to do better than the behavior in those trajectories. Asking for a higher RTG does not itself supply the actions needed to reach it. If the dataset lacks trajectories with the desired return, a sequence model cannot reliably infer successful behavior merely from that request. Conditioning on success is not equivalent to learning how to create previously unseen success.

Hindsight return relabeling introduces a further difficulty in online Decision Transformer variants. RTGs used during rollout are replaced by the returns observed afterward. That changes the conditioning information between collecting an action and optimizing from it, creating a mismatch in the importance ratios used by standard online reinforcement learning machinery. The resulting moving target can destabilize training.

When poor trajectories dominate, the model has little high-quality behavior on which to condition. More examples of poor behavior do not remove that absence. Unlike Bellman-based methods, which can reason about unseen returns through value relationships, the offline sequence formulation is primarily interpolative. The limitation matters most when play produces genuinely novel situations, because the policy must then do more than recover a successful pattern already represented in its data.

Two active lines of work that address the offline-only ceiling

Luo and Zhu, in work submitted to ICLR 2026, adapt GRPO to fine-tune pretrained Decision Transformers using pure reinforcement learning gradients. Their approach preserves offline knowledge while allowing online exploration, treating pretraining and interaction as complementary phases. They report state-of-the-art performance on several benchmarks. That is evidence of progress beyond offline imitation.

Shi et al. (2024) address the question from the theoretical side. They provide guarantees that pretrained transformers can learn to approximate Nash equilibrium in context in two-player zero-sum games, covering both decentralized and centralized learning settings. This result matters because it distinguishes the limitations of an offline training formulation from the capabilities of the transformer architecture itself. A transformer need not be restricted to retrieving behavior at return levels represented in a fixed dataset. The guarantee applies only within its stated game and learning settings.

Sinha et al. (November 2025) provide related algorithmic evidence through STACCA, or Shared Transformer Actor-Critic with Counterfactual Advantage. Its shared Graph Transformer Actor uses GAT preprocessing within a full Actor-Critic framework. Each agent attends over its 1-hop induced subgraph, which encodes state, control, topology, and shortest-path distances. STACCA enables zero-shot transfer across unseen network topologies, demonstrating a specific form of out-of-distribution generalization. Together, these results weaken an architecture-wide interpolation objection while leaving the broader online adaptation problem open.

What interpretability research reveals about how transformer policies actually reason

Lin et al. (April 2026) bring the argument back to Leela Chess Zero by examining its internal computation. Their sparse decomposition framework replaces MLP and attention modules with sparse replacement layers that capture the primary computation process. The resulting pathways expose interpretable tactical considerations that can be tested empirically. This moves the evidence beyond playing strength and attention’s theoretical capacity for global access. It asks whether a trained policy actually encodes recognizable relationships that matter to chess decisions.

Lin et al. describe their work, to their knowledge, as the first decomposition of both MLP and attention modules for transformer interpretability. They introduce three quantitative metrics showing that LC0 exhibits parallel reasoning consistent with the inductive bias of its policy head architecture. That correspondence links the network’s internal organization to its tactical behavior. It provides a concrete basis for auditing parts of the computation. Such access matters wherever an opaque policy is harder to trust, certify, or improve, including competitive games and safety-critical deployment.

The finding also clarifies why the architectural argument is stronger than a claim that transformers simply score better. Attention makes distant relationships directly accessible, and the LC0 analysis shows that a trained model can organize its computation into verifiable tactical pathways. These methods have yet to be shown to work elsewhere. Transformers are a principled choice for game policies because the relationships games demand can become both usable and legible within the architecture.