Hierarchical RL for Long-Horizon Strategy Game Tasks

A flat policy hits a structural ceiling in long-horizon strategy games, and hierarchical reinforcement learning exists to answer it, so that ceiling needs naming before you look at any solution. A flat policy, which maps states directly to actions with no intermediate abstraction, faces two compounding pathologies once a task stretches across thousands of timesteps: sparse delayed rewards and the collapse of credit assignment. The reward signal in a strategy game typically arrives only at the end, win or lose, and standard temporal difference methods struggle to propagate that signal back through the enormous chain of decisions that produced it. Nothing in a flat policy's structure can diagnose whether a failure at a long-horizon task traces to a poor strategic choice made early or to a botched execution made late. Every decision, good and bad, gets folded into a single gradient update, and conventional exploration strategies, faced with a sparse reward surface and no structural guidance, see the probability of stumbling onto a rewarding state shrink exponentially as the horizon grows, often stalling convergence or locking the policy into a mediocre local optimum.

How hierarchical decomposition compresses the planning problem

Hierarchical reinforcement learning answers this by replacing the single flat mapping with a layered structure: a high-level policy sets subgoals, and a low-level policy executes primitive actions to reach them. This idea has precedent in the options framework and in feudal architectures, both of which formalized the notion that a policy could operate at more than one timescale at once. The payoff is that the horizon faced by each individual controller shrinks dramatically. A high-level policy reasons over subgoals spaced far apart in time, so it sees a short decision sequence, and a low-level policy chasing one subgoal at a time sees a short decision sequence too. Decomposition also splits the credit assignment problem that flat RL had collapsed into one signal: did the high-level policy choose a bad subgoal, or did the low-level policy fail to reach a good one? So each question gets its own gradient, and both levels end up learning from a cleaner, less noisy target. The structure also buys skill reuse, since a low-level skill trained to reach one subgoal can be invoked by the high-level controller across many different strategic situations, compounding sample efficiency rather than relearning the same motion pattern from scratch each time. The HRL-SAC framework pairs high-level strategic planning with a continuous-control Soft Actor-Critic at the low level in a two-level hierarchy, both levels trained with entropy-regularized policy optimization, and the resulting system outperforms a flat SAC baseline on success rate, coverage efficiency, and convergence speed in sparse-reward, long-horizon search tasks.

How recent work addresses subgoal infeasibility

Hierarchy is not a free lunch, and the sharpest failure mode inside HRL deserves equal attention to its benefits. High-level policies can and do propose subgoals that the low-level policy has no structural way to reach, whether because of an obstacle in the environment or a gap in the low-level skill set. Hindsight experience replay, the standard fix for sparse-reward learning, makes this worse in this specific case: it relabels failed attempts as successes toward whatever goal the agent happened to reach, which teaches the low-level policy something, but it leaves the high-level policy never informed that the original subgoal was unreachable. The high-level planner keeps proposing goals in regions the executor cannot enter, and the gradient it receives stays poisoned by that mismatch. Strict Subgoal Execution, presented at ICLR 2026, constrains the high-level policy to propose only subgoals the low-level policy can reach in a single step, eliminating the infeasibility class at its source. SSE pairs that constraint with a decoupled exploration policy that systematically visits underexplored regions of goal space, plus failure-aware path refinement that adjusts edge costs in its subgoal graph according to the low-level policy's observed success rates. SSE reaches close to a perfect success rate while HIRO and HIGL fail to learn the task entirely, a qualitative gap between a method that solves subgoal infeasibility structurally and two that do not.

Scaling hierarchy depth: what recursive goal decomposition achieves in hard puzzles

Most HRL research in recent years worked with two or three levels of hierarchy, a shallow depth that raised a real question: does the architecture scale, or does it hit diminishing returns past a couple of layers? Recursive goal decomposition, where hierarchies run far deeper and are learned entirely end-to-end, answers that question directly and unlocks domains that shallower hierarchies cannot touch. Pastukhov's 2025 HRL framework for Sokoban builds a six-level policy hierarchy in which each higher level generates subgoals for the level immediately below it, and it injects no domain knowledge, provides no demonstrations, and hand-designs no abstractions anywhere in the pipeline. Sokoban is a demanding test case precisely because its combinatorial structure punishes naive exploration: an agent cannot stumble into a solution by chance, so a hierarchy that solves it must have developed genuinely coherent intermediate representations of puzzle state to generate subgoals worth pursuing. You go from two or three levels to six, learned from scratch, on a domain this unforgiving, and that shows hierarchy depth is a design parameter that can scale with the problem, not an engineering ceiling imposed by prior architectures.

HRL applied to StarCraft II: micromanagement, full games, and race generalization

Real-time strategy games give hierarchical decomposition a commercial, competitively benchmarked proving ground, and recent work shows it solving distinct problems at distinct levels of the game. Bai et al.'s 2026 hierarchical multi-Q-table architecture takes on StarCraft II micromanagement by splitting decision-making into an upper level that selects clustering strategy and a lower level that handles tactical execution, with reward allocated densely enough to generate real learning signal across six asymmetric combat scenarios. The architecture matches competitive deep RL baselines while offering clear advantages in sample efficiency and interpretability. That interpretability advantage carries weight beyond convenience: because the hierarchy separates strategic reasoning from tactical execution, an observer can audit which layer made which decision, something flat end-to-end systems like AlphaStar cannot offer, and that auditability matters directly in safety-constrained or regulated settings where an opaque policy is a liability regardless of its win rate. At the level of the full game, HIMA tested all nine possible race combinations across Protoss, Terran, and Zerg, the first time the literature has covered the complete matchup set rather than a single race pairing, and recorded high win rates across multiple difficulty settings. Earlier work in this space had proven the concept on one matchup. HIMA's result is a generalization claim, not a refinement of an existing benchmark, and it demonstrates that the hierarchical approach holds up across the full strategic variety the game actually contains.

What flat RL achieved without hierarchy, and its cost

Flat RL's strongest counterargument to all of this is that it already produced superhuman play in two of the hardest strategy game benchmarks that exist. OpenAI Five reached superhuman Dota 2 performance using PPO over a flat LSTM architecture with no explicit hierarchical macro-actions, and AlphaStar achieved comparable results in StarCraft II through a similarly flat design. Both are genuine achievements, but you should not read either as evidence that hierarchy is unnecessary in principle. What they depended on is the detail that keeps those results from generalizing: training at a compute scale available to a small number of organizations, reward functions hand-shaped with extensive domain expertise, and a specialization to one game that does not transfer to the next one. Dense reward shaping, the tool flat RL leans on to substitute for structural decomposition, trades one engineering cost for another. Building a hierarchy requires designing levels and interfaces, but shaping rewards densely enough for a flat policy to learn a long-horizon task requires comparably deep domain expertise, invested over and over for each new task, and it produces no reusable component when the task changes. If a hierarchical policy's high-level layer operates over reusable skills, its advantage compounds, because those skills transfer to new tasks and new environments, but a flat policy's hand-tuned reward function stays bound to the single game it was built for.

LLMs as high-level planners: what changes when strategic abstraction is pretrained

Classical HRL left one problem stubbornly open: automatic option discovery, the question of which skills the low-level policy should even be learning in the first place. In a classical hierarchy, the high-level policy has to learn two things at once, what subgoals to set and which skills exist to satisfy them, so a second hard learning problem stacks on top of an already difficult one. Using a pretrained large language model as the high-level controller sidesteps that problem rather than solving it from scratch: the LLM arrives already carrying semantic knowledge of task structure, which supports decomposing a task into sensible subgoals, inferring what phase of a strategic sequence the agent is in, and selecting among skills contextually, all without hand-coded logic to manage it. Option discovery does not disappear under this architecture. It moves from online learning into pretraining, so you solve it once, broadly, for every new task. Hösch et al.'s 2026 architecture puts this into practice: a pretrained LLM sits as a centralized strategic controller choosing among specialized RL skill policies for a team of agents, with the LLM supplying strategic intent and the RL policies supplying reliable low-level execution. The L2M2 system referenced in that work uses LLMs to generate navigation subgoals for cooperative multi-agent RL, and it cuts training samples by 80% compared to end-to-end approaches, a reduction that speaks directly to the compute-cost argument that favors flat RL. The pattern is not confined to games: LLMs augmented with action primitives for long-horizon manipulation tasks show the same LLM-as-planner structure succeeding in physical domains, which is evidence that the architecture is a structural advance rather than a trick specific to StarCraft II or Dota 2.

Credit assignment at the hierarchy boundary: HiPER's hierarchical advantage estimation

An LLM at the top of the hierarchy resolves option discovery, but it does not by itself fix how credit flows across the boundary between planner and executor, and that stays a live optimization problem no matter how the high level is implemented. When a plan succeeds or fails, the outcome reflects two things at once: whether the subgoal chosen was a good one, and whether the execution of it was competent. Standard advantage estimation mixes these into a single number, and a high-level policy trained on that mixed signal is learning from a noisy approximation of its own contribution. HiPER, presented at ICML 2026, introduces hierarchical advantage estimation, a two-timescale policy gradient method that factorizes the policy into a high-level planner proposing subgoals and a low-level executor carrying them out over multiple steps, assigning credit to action execution and to subgoal transitions separately. On ALFWorld and WebShop, two long-horizon task environments that demand sequential planning under uncertainty, HiPER reaches a 97.4% success rate on ALFWorld and strong results on WebShop as well. That outcome establishes something specific: explicit credit factorization at the hierarchy boundary produces qualitatively better performance than leaving the planner and the executor to share one undifferentiated advantage signal.

How symbolic planning integration adds compositional generalization

A third architectural direction addresses a limitation that neither an LLM planner nor a purely neural hierarchy fully resolves on its own: systematic reuse of learned skills when a genuinely new task demands a combination nobody trained for. Pure RL hierarchies learn skills tied to the distribution of tasks they were trained on, so when a new task needs known subtasks combined in a way the training distribution never presented, the high-level policy has no built-in way to compose them. It can only generalize insofar as the new task resembles something already seen. If you integrate symbolic planning into the hierarchy, the system gets an explicit representation of operators, and it can reuse and recombine them according to logical structure. That capacity for flexible extension is what produces compositional generalization, the ability to solve a task built from a new arrangement of familiar pieces, a property that purely neural hierarchies, however deep or well-trained, do not achieve on their own. Across all three directions, the LLM planner, the credit-factorized optimizer, and the symbolic layer, the throughline is the same: hierarchy compresses the planning problem that flat RL cannot solve at scale, and each new architectural refinement closes a specific gap that the previous layer of the hierarchy left open.