Abstract
Training vision-language-action (VLA) models to follow gameplay instructions across diverse games is limited by the cost of collecting demonstrations. We introduce a framework that automatically generates game environments and expert demonstrations. An LLM-powered agent uses game-building tools to construct executable environments and tasks targeting specified skills. Programmatic experts then use privileged simulator state to generate aligned instructions, rendered observations, and keyboard and mouse actions. We train our game policy, Script2Play, by post-training a 4B vision-language model with behavior cloning on 150.3 hours of synthetic gameplay in Minecraft and Roblox, without human gameplay demonstrations or reinforcement learning. Evaluation on multiple public game benchmarks demonstrates that Script2Play follows gameplay instructions effectively and generalizes across game engines, highlighting the effectiveness of targeted synthetic supervision.
Video
Method
Training data
We choose the game mixture by surveying games on mainstream platforms and tasks in public benchmarks, then estimating how frequently they require each gameplay skill. The 123.4 h from Roblox covers 41 generated games in six categories: combat and shooting (56.4 h), traversal and parkour (21.5 h), object interaction (20.8 h), dodging and survival (19.7 h), GUI operation (3.7 h), and ball games (1.3 h). The 26.9 h of Minecraft demonstrations comprise 31,494 verified episodes covering mining and workstation use (5.0 h), crafting and smelting (19.0 h), and mob combat (2.9 h).
Diversity within a generated skill
Diversity within a generated skill. Eight verified Melee arena episodes share a combat mechanic while varying the world theme, characters, camera, dynamics, and instruction wording. Player and enemy appearances are sampled from 20 and 9 Roblox avatar bundles, respectively; camera FOV ranges from 62–78°. Themes vary terrain, props, sky, and lighting. Arena size, enemy speeds, and control gains also vary across episodes. Labels give the theme, FOV, and instruction. Videos are cropped to the play area.
Results
Instruction following on OpenHA
Script2Play reaches 69.6% average success, exceeding the strongest prior trained policy, Game-TARS MoE-mini (64.5%), by 5.1 percentage points. This is the highest average among the compared trained game policies, using 150.3 h of synthetic demonstrations and behavior cloning.
| Method | Training data | Total data | MC data | Embodied | GUI | Combat | Avg |
|---|---|---|---|---|---|---|---|
| Commercial VLMs (low reasoning effort) | |||||||
| Claude Opus 5.5 | – | – | – | 72.9 | 80.6 | 53.1 | 68.9 |
| GPT-6.1-Sol | – | – | – | 68.7 | 84.9 | 69.4 | 74.3 |
| Gemini 4 Argon | – | – | – | 80.7 | 96.0 | 71.4 | 82.7 |
| Trained Game Policies | |||||||
| VPT | Human play | ~70,000 h | ~70,000 h | 6.0 | 0.8 | 3.6 | 3.5 |
| STEVE-1 | Human + VPT rollouts | ~70,750 h† | ~70,750 h† | 8.0 | 3.2 | 3.9 | 5.0 |
| ROCKET-1 | Human play | ~22,000 h† | ~22,000 h† | 18.9 | 0.0 | 27.9 | 15.6 |
| JARVIS-VLA | Human play | ~192 h† | ~192 h† | 30.0 | 25.1 | 18.5 | 24.5 |
| OpenHA | Human play | ~145 h† | ~145 h† | 30.1 | 32.5 | 31.9 | 31.5 |
| UI-TARS-1.5 | Human play + RL gym | ~2,600 h† | ~600 h† | 42.1 | 36.7 | 31.0 | 36.6 |
| HiSTEVE | Human play + RL gym | ~200 h† + RL | ~200 h† + RL | 46.0 | 27.2 | 39.0 | 37.4 |
| Game-TARS Dense 7B | Human play + RL gym | ~20,000 h | ~560 h† | 50.4 | 39.1 | 38.1 | 42.5 |
| GameWAM | Human play + scripted GUI | ~300 h† | ~300 h† | 47.5 | 60.0 | 32.2 | 46.6 |
| Game-TARS MoE-mini | Human play + RL gym | ~22,000 h | ~560 h† | 72.0 | 55.4 | 66.1 | 64.5 |
| Our method | |||||||
| Script2Play (4B) | Scripted teacher | 150.3 h | 26.9 h | 63.9[58.7, 69.1] | 66.4[63.7, 69.1] | 78.6[70.3, 86.9] | 69.6[66.3, 73.0] |
Success rate (%), with 95% confidence intervals in brackets for Script2Play; higher is better. Total data denotes the listed gameplay training scale; MC data denotes its Minecraft component. STEVE-1 includes inherited VPT pretraining. Script2Play uses 150.3 h in total: 26.9 h of Minecraft and 123.4 h of Roblox demonstrations. Bold marks the best score separately within commercial VLMs and trained policies (including ours). †Approximate hour equivalents inferred from reported frame or token counts, not matched training budgets.
Transfer to five solo games in OmniGameArena
We test the same checkpoint on UE5, which does not appear in post-training, and evaluate only OmniGameArena's 3D instruction-following games to align with the task setting of our training data. Script2Play achieves a mean score of 0.640, the highest score among all compared agents. It exceeds the best commercial VLM in this comparison, Gemini 3.1 Pro Preview (0.473), by 0.167, and the strongest human-gameplay policy, GameScaling-9B-S3 (0.361), by 0.279. The latter uses 1,600 h of human gameplay, compared with our 150.3 h of synthetic demonstrations.
| Agent | Training data | Data scale | Obstacle Run3D | Last Stand | Monster Shoot | Scene Escape | Cue Chase | Mean |
|---|---|---|---|---|---|---|---|---|
| Commercial VLMs | ||||||||
| Claude Opus 4.7 | – | – | 0.094 | 0.308 | 0.400 | 0.460 | 0.380 | 0.328 |
| Claude Opus 4.6 | – | – | 0.172 | 0.147 | 0.362 | 0.540 | 0.840 | 0.412 |
| Claude Sonnet 4.6 | – | – | 0.200 | 0.144 | 0.166 | 0.440 | 0.440 | 0.278 |
| GPT-5.5 | – | – | 0.133 | 0.416 | 0.464 | 0.720 | 0.580 | 0.463 |
| GPT-5.4 | – | – | 0.089 | 0.148 | 0.326 | 0.680 | 0.300 | 0.309 |
| Gemini 3.1 Pro Preview | – | – | 0.165 | 0.230 | 0.710 | 0.660 | 0.600 | 0.473 |
| Gemini 3.1 Flash-Lite Preview | – | – | 0.097 | 0.122 | 0.182 | 0.300 | 0.440 | 0.228 |
| Kimi K2.5 | – | – | 0.075 | 0.232 | 0.290 | 0.220 | 0.140 | 0.191 |
| Open-weight VLMs | ||||||||
| Qwen3.5-397B-A17B | – | – | 0.112 | 0.106 | 0.072 | 0.200 | 0.040 | 0.106 |
| Qwen3.5-122B-A10B | – | – | 0.034 | 0.093 | 0.024 | 0.060 | 0.040 | 0.050 |
| Trained Game Policies | ||||||||
| NitroGen (gamepad) | Human play | 40,000 h | 0.065 | 0.100 | 0.014 | 0.120 | 0.020 | 0.064 |
| Open-P2P (kbd-mouse) | Human play | 8,300+ h | 0.023 | 0.063 | 0.000 | 0.100 | 0.000 | 0.037 |
| GameScaling-0.8B-S3 | Human play | 1,600 h | 0.182[0.000, 0.416] | 0.102[0.068, 0.137] | 0.316[0.073, 0.559] | 0.200[0.048, 0.352] | 0.040[0.000, 0.108] | 0.168 |
| GameScaling-2B-S3 | Human play | 1,600 h | 0.097[0.000, 0.236] | 0.102[0.060, 0.144] | 0.314[0.000, 0.724] | 0.160[0.000, 0.386] | 0.000[0.000, 0.000] | 0.135 |
| GameScaling-9B-S3 | Human play | 1,600 h | 0.083[0.038, 0.129] | 0.080[0.053, 0.107] | 0.780[0.304, 1.000] | 0.520[0.298, 0.742] | 0.340[0.152, 0.528] | 0.361 |
| Our method | ||||||||
| Script2Play (4B) | Scripted teacher | 150.3 h | 0.374[0.104, 0.644] | 0.436[0.387, 0.485] | 0.972[0.924, 1.000] | 0.420[0.198, 0.642] | 1.000 | 0.640[0.582, 0.699] |
Scores use the benchmark's per-game normalized metrics (with 95% confidence intervals in brackets for locally evaluated policies) and their mean (higher is better), under the time-paused protocol. Script2Play receives no UE5 gameplay during post-training. Bold marks the best score in each column. Training scales describe the respective gameplay corpora, not matched compute budgets.
Transfer to ViZDoom
Script2Play scores 12.92 without ViZDoom trajectories or target-game RL, compared with 14.54 for the zero-shot Game-TARS MoE-mini, which uses a much larger model and substantially more gameplay data. GameWAM is trained directly on expert APPO trajectories from the four evaluation maps. The Battle maps require the agent to search for health and ammunition as well as enemies, although the instruction only asks it to kill enemies; these implicit subgoals and the need to find enemies outside the current view are difficult for our policy, which cannot track previously searched areas or supply locations from a single frame.
| Method | Data scale | Battle 1 | Battle 2 | Defend Line | Defend Center | Mean |
|---|---|---|---|---|---|---|
| In-domain (trained on ViZDoom APPO) | ||||||
| GameWAM | ~300 h† | 43.12 | 19.02 | 23.52 | 15.36 | 25.26 |
| Zero-shot (no ViZDoom training) | ||||||
| Game-TARS MoE-mini | ~22,000 h | 18.87 | 10.04 | 15.24 | 14.00 | 14.54 |
| o4-mini | – | 7.85 | 6.74 | 10.24 | 11.39 | 9.06 |
| GPT-5 | – | 12.59 | 8.99 | 10.24 | 13.46 | 11.32 |
| Gemini 2.5 Pro | – | 7.51 | 10.04 | 15.24 | 5.27 | 9.52 |
| Claude 4 Sonnet | – | 7.34 | 7.52 | 10.24 | 5.56 | 7.67 |
| Our method (scripted teacher) | ||||||
| Script2Play (4B) | 150.3 h | 17.00 | 4.08 | 22.94 | 7.66 | 12.92 |
Mean reward over 50 episodes per map; higher is better. GameWAM uses target-domain training; the other policies transfer zero-shot. †Approximate data scale.
Effect of the supervision source
Shared-backbone comparison. We compare three policies initialized from Qwen2.5-VL-7B: OpenHA trained on human play, HiSTEVE trained with human-derived supervision and RL, and a separate Script2Play variant trained by behavior cloning on 26.9 h of Minecraft teacher demonstrations, without Roblox data. This comparison fixes the backbone; data budgets and training procedures differ. We compare OpenHA success rates for all three and evaluate OpenHA and our policy zero-shot on OmniGameArena.
| (a) In-domain: OpenHA (Minecraft, success rate %) | ||||||
| Supervision | Method | MC data | Emb. | GUI | Combat | Avg |
|---|---|---|---|---|---|---|
| Human play | OpenHA | ~145 h† | 30.1 | 32.5 | 31.9 | 31.5 |
| Human + RL | HiSTEVE | ~200 h† + RL | 46.0 | 27.2 | 39.0 | 37.4 |
| Scripted | Script2Play (7B, MC only) | 26.9 h | 49.7[44.3, 55.1] | 39.0[36.2, 41.8] | 49.0[38.9, 59.1] | 45.9 |
| (b) Out-of-distribution: OmniGameArena (UE5, normalized score) | ||||||
| Method | Obstacle Run3D | Last Stand | Monster Shoot | Scene Escape | Cue Chase | Mean |
|---|---|---|---|---|---|---|
| OpenHA | 0.021 | 0.082 | 0.010 | 0.000 | 0.000 | 0.023 |
| Script2Play (7B, MC only) | 0.033[0.000, 0.073] | 0.066[0.041, 0.090] | 0.000[0.000, 0.000] | 0.040[0.000, 0.108] | 0.000[0.000, 0.000] | 0.028 |
Supervision comparison with Qwen2.5-VL-7B. Our 7B variant uses behavior cloning on 26.9 h of Minecraft data, independently of the main 4B model. Brackets show 95% confidence intervals; bold marks column bests. HiSTEVE weights are unavailable for (b). †Approximate hour equivalents.
Our policy achieves 45.9% OpenHA success versus 31.5% and 37.4%, leading both baselines across all three task groups (a), with similar zero-shot scores to OpenHA (0.028 versus 0.023; b). Thus, the in-domain advantage persists when the backbone is held fixed.
Matched human and scripted supervision. To isolate the supervision source, we train two Qwen3.5-4B policies on GameWAM's event-aligned human VPT clips or verified Minecraft teacher demonstrations. We use 429 training tasks and hold out 136 tasks. For each training task, we cap both sources at the smaller window count and sample whole clips, so both sources share the same task distribution and the same amount of training data for every task, approximately 8.7 h per source. Both use instructions sampled from OpenHA task descriptions, the same observation/action format and training recipe, and five epochs (2,405 steps, batch size 384). A larger-data human control uses all 40.4 h available for the same training tasks, for three epochs (6,683 steps). We evaluate each epoch on tasks from the OpenHA benchmark: 240 in-corpus tasks and all 136 held-out tasks. The in-corpus tasks also appear in training, but all evaluations use environments excluded from both training sources.
Results. Scripted supervision leads at all five epochs on both task sets, with 9.7–18.3-point held-out gains. Its one-epoch in-corpus success (51.6%) exceeds the best matched-human checkpoint (45.6%); from epoch two, it also beats the 40.4 h control. Gains are largest on Embodied and Combat; the larger human control leads on GUI. Scripted supervision improves data efficiency and generalization.
Which skills transfer?
We train seven Qwen3.5-4B policies on all nonempty subsets of Combat & Shooting (C), Traversal & Parkour (T), and Dodging & Survival (D). We evaluate transfer on three OmniGameArena games, one per skill category: Monster Shoot (C), Obstacle Run3D (T), and Last Stand (D). This ablation uses subsets of the Roblox data, containing 20.5, 12.3, and 15.8 h for C, T, and D, respectively. All arms use the Script2Play training recipe for 3,000 steps, with no Minecraft data. Within each batch, the selected categories share 85% equally, and web data supplies the remaining 15%. Each policy plays 10 episodes per game under the time-paused protocol.
| Training categories | Monster Shoot | Obstacle Run3D | Last Stand | Mean |
|---|---|---|---|---|
| C | 0.722[0.468, 0.976] | 0.066[0.064, 0.069] | 0.108[0.079, 0.138] | 0.299 |
| T | 0.000[0.000, 0.000] | 0.215[0.134, 0.295] | 0.139[0.088, 0.190] | 0.118 |
| D | 0.000[0.000, 0.000] | 0.000[0.000, 0.000] | 0.744[0.588, 0.900] | 0.248 |
| C+T | 0.985[0.956, 1.000] | 0.370[0.259, 0.481] | 0.118[0.109, 0.126] | 0.491 |
| C+D | 0.991[0.979, 1.000] | 0.014[0.005, 0.023] | 0.812[0.663, 0.961] | 0.606 |
| T+D | 0.000[0.000, 0.000] | 0.233[0.126, 0.341] | 0.427[0.354, 0.499] | 0.220 |
| C+T+D | 0.969[0.948, 0.990] | 0.288[0.170, 0.407] | 0.428[0.390, 0.467] | 0.562 |
| Reference | ||||
| Script2Play (4B) | 0.972 | 0.374 | 0.436 | 0.594 |
| Random | 0.008 | 0.000 | 0.088 | 0.032 |
Skill-category ablation on OmniGameArena. Normalized scores over 10 episodes (mean with 95% confidence intervals; higher is better). C: Combat & Shooting; T: Traversal & Parkour; D: Dodging & Survival. Bold marks the best ablation arm per column. Reference Script2Play uses 8,000 training steps on the full mixture and 5 evaluation episodes per game.
Without the corresponding training category, Monster Shoot scores 0 and the other two games remain near Random. Adding traversal data hurts Last Stand: policies keep moving forward and fall sooner. Adding combat data also halves the dodging-data share without the same drop. Transfer therefore depends on both relevant skill coverage and the compatibility of combined skills.
Closed-loop rollouts of Script2Play
Action sequences reveal how control transfers. In OpenHA combat, the policy turns and advances toward the target before attacking. In GUI tasks, it combines mouse positioning with item-manipulation clicks. In the held-out UE5 engine, ObstacleRun3D requires forward motion interleaved with timed jumps, while CueChase requires camera adjustment, approach, and interaction. The same action-chunk representation supports these distinct control behaviors in Minecraft and UE5. These successful episodes illustrate policy behavior; the benchmark scores measure its reliability.
Minecraft (OpenHA)
OmniGameArena
At 2× speed.
ViZDoom
BibTeX
@article{liu2026gamegen,
title = {From Game Generation to Game Playing: Fully Automated Ability Training
for Vision-Language-Action Models},
author = {Liu, Changran and Wu, Qi and Ma, Wenchao and Huang, Sharon X. and Jiang, Haomiao},
journal = {arXiv preprint arXiv:2611.11136},
year = {2026}
}