From Game Generation to Game Playing Fully Automated Ability Training for Vision-Language-Action Models

Changran Liu1Qi Wu1Wenchao Ma2Sharon X. Huang2Haomiao Jiang1
1Roblox2The Pennsylvania State University

Abstract

Training vision-language-action (VLA) models to follow gameplay instructions across diverse games is limited by the cost of collecting demonstrations. We introduce a framework that automatically generates game environments and expert demonstrations. An LLM-powered agent uses game-building tools to construct executable environments and tasks targeting specified skills. Programmatic experts then use privileged simulator state to generate aligned instructions, rendered observations, and keyboard and mouse actions. We train our game policy, Script2Play, by post-training a 4B vision-language model with behavior cloning on 150.3 hours of synthetic gameplay in Minecraft and Roblox, without human gameplay demonstrations or reinforcement learning. Evaluation on multiple public game benchmarks demonstrates that Script2Play follows gameplay instructions effectively and generalizes across game engines, highlighting the effectiveness of targeted synthetic supervision.

Video

Method

Overview: generate games from specifications, demonstrate with a scripted teacher, play with a VLM policy
Overview. Generate: a coding agent turns each specification into a game with scenes s0, tasks (g, V), and a scripted teacher π*; a game must pass five audit gates, and a failing game is repaired (up to three rounds) or discarded. Demonstrate: the teacher plans on privileged engine state but acts only through keyboard and mouse; the piano roll shows its recorded keys (W: forward, ←: camera left, →: camera right, E: object interaction) in one rollout, which the verifier accepts (V = 1). Only verified rollouts enter 𝒟. Play: behavior cloning on 𝒟 turns a pretrained VLM into the policy πθ, which writes each 200 ms action chunk as text. It is evaluated on OpenHA in Minecraft, a training engine, and zero-shot on OmniGameArena (UE5) and ViZDoom.
Generating a game and its demonstrations: specification, coding agent, four files, five audit gates, six templates; the teacher plays two arms of one scene
Generating a game and its demonstrations. A coding agent generates a game and scripted teacher π* from a skill specification. Here, two tasks use the same scene but ask the teacher to shoot all red or all blue targets. Only rollouts that complete the instructed task and pass verification enter 𝒟.

Training data

Verified teacher rollouts in generated Roblox games and in Minecraft.
Post-training data: Roblox and Minecraft data across different skills, with hours of verified play Representative frames of the Roblox and Minecraft training data
Training environments organized by reusable gameplay skills. Top: skill categories, mechanics, and verified gameplay hours. Bottom: representative frames. Roblox games cover six categories and Minecraft covers OpenHA's three groups, with multiple camera views.

We choose the game mixture by surveying games on mainstream platforms and tasks in public benchmarks, then estimating how frequently they require each gameplay skill. The 123.4 h from Roblox covers 41 generated games in six categories: combat and shooting (56.4 h), traversal and parkour (21.5 h), object interaction (20.8 h), dodging and survival (19.7 h), GUI operation (3.7 h), and ball games (1.3 h). The 26.9 h of Minecraft demonstrations comprise 31,494 verified episodes covering mining and workstation use (5.0 h), crafting and smelting (19.0 h), and mob combat (2.9 h).

Diversity within a generated skill

Forest · FOV 70°
“Defeat the enemies around you with your sword and avoid their attacks.”
Forest · FOV 69°
“Several swordsmen attack you: hit them and stay out of their swings.”
City · FOV 70°
“Survive the melee: strike the enemies and dodge their blows.”
Desert · FOV 73°
“Fight off the group of sword fighters.”
Snow · FOV 75°
“Several swordsmen attack you: hit them and stay out of their swings.”
Snow · FOV 64°
“Survive the melee: strike the enemies and dodge their blows.”
Desert · FOV 71°
“Fight off the group of sword fighters.”
Forest · FOV 72°
“Defeat the enemies around you with your sword and avoid their attacks.”

Diversity within a generated skill. Eight verified Melee arena episodes share a combat mechanic while varying the world theme, characters, camera, dynamics, and instruction wording. Player and enemy appearances are sampled from 20 and 9 Roblox avatar bundles, respectively; camera FOV ranges from 62–78°. Themes vary terrain, props, sky, and lighting. Arena size, enemy speeds, and control gains also vary across episodes. Labels give the theme, FOV, and instruction. Videos are cropped to the play area.

Results

Instruction following on OpenHA

Script2Play reaches 69.6% average success, exceeding the strongest prior trained policy, Game-TARS MoE-mini (64.5%), by 5.1 percentage points. This is the highest average among the compared trained game policies, using 150.3 h of synthetic demonstrations and behavior cloning.

MethodTraining dataTotal dataMC dataEmbodiedGUICombatAvg
Commercial VLMs (low reasoning effort)
Claude Opus 5.5–––72.980.653.168.9
GPT-6.1-Sol–––68.784.969.474.3
Gemini 4 Argon–––80.796.071.482.7
Trained Game Policies
VPTHuman play~70,000 h~70,000 h6.00.83.63.5
STEVE-1Human + VPT rollouts~70,750 h†~70,750 h†8.03.23.95.0
ROCKET-1Human play~22,000 h†~22,000 h†18.90.027.915.6
JARVIS-VLAHuman play~192 h†~192 h†30.025.118.524.5
OpenHAHuman play~145 h†~145 h†30.132.531.931.5
UI-TARS-1.5Human play + RL gym~2,600 h†~600 h†42.136.731.036.6
HiSTEVEHuman play + RL gym~200 h† + RL~200 h† + RL46.027.239.037.4
Game-TARS Dense 7BHuman play + RL gym~20,000 h~560 h†50.439.138.142.5
GameWAMHuman play + scripted GUI~300 h†~300 h†47.560.032.246.6
Game-TARS MoE-miniHuman play + RL gym~22,000 h~560 h†72.055.466.164.5
Our method
Script2Play (4B)Scripted teacher150.3 h26.9 h 63.9[58.7, 69.1]66.4[63.7, 69.1]78.6[70.3, 86.9]69.6[66.3, 73.0]

Success rate (%), with 95% confidence intervals in brackets for Script2Play; higher is better. Total data denotes the listed gameplay training scale; MC data denotes its Minecraft component. STEVE-1 includes inherited VPT pretraining. Script2Play uses 150.3 h in total: 26.9 h of Minecraft and 123.4 h of Roblox demonstrations. Bold marks the best score separately within commercial VLMs and trained policies (including ours). †Approximate hour equivalents inferred from reported frame or token counts, not matched training budgets.

Transfer to five solo games in OmniGameArena

We test the same checkpoint on UE5, which does not appear in post-training, and evaluate only OmniGameArena's 3D instruction-following games to align with the task setting of our training data. Script2Play achieves a mean score of 0.640, the highest score among all compared agents. It exceeds the best commercial VLM in this comparison, Gemini 3.1 Pro Preview (0.473), by 0.167, and the strongest human-gameplay policy, GameScaling-9B-S3 (0.361), by 0.279. The latter uses 1,600 h of human gameplay, compared with our 150.3 h of synthetic demonstrations.

AgentTraining dataData scaleObstacle Run3DLast StandMonster ShootScene EscapeCue ChaseMean
Commercial VLMs
Claude Opus 4.7––0.0940.3080.4000.4600.3800.328
Claude Opus 4.6––0.1720.1470.3620.5400.8400.412
Claude Sonnet 4.6––0.2000.1440.1660.4400.4400.278
GPT-5.5––0.1330.4160.4640.7200.5800.463
GPT-5.4––0.0890.1480.3260.6800.3000.309
Gemini 3.1 Pro Preview––0.1650.2300.7100.6600.6000.473
Gemini 3.1 Flash-Lite Preview––0.0970.1220.1820.3000.4400.228
Kimi K2.5––0.0750.2320.2900.2200.1400.191
Open-weight VLMs
Qwen3.5-397B-A17B––0.1120.1060.0720.2000.0400.106
Qwen3.5-122B-A10B––0.0340.0930.0240.0600.0400.050
Trained Game Policies
NitroGen (gamepad)Human play40,000 h0.0650.1000.0140.1200.0200.064
Open-P2P (kbd-mouse)Human play8,300+ h0.0230.0630.0000.1000.0000.037
GameScaling-0.8B-S3Human play1,600 h 0.182[0.000, 0.416]0.102[0.068, 0.137]0.316[0.073, 0.559] 0.200[0.048, 0.352]0.040[0.000, 0.108]0.168
GameScaling-2B-S3Human play1,600 h 0.097[0.000, 0.236]0.102[0.060, 0.144]0.314[0.000, 0.724] 0.160[0.000, 0.386]0.000[0.000, 0.000]0.135
GameScaling-9B-S3Human play1,600 h 0.083[0.038, 0.129]0.080[0.053, 0.107]0.780[0.304, 1.000] 0.520[0.298, 0.742]0.340[0.152, 0.528]0.361
Our method
Script2Play (4B)Scripted teacher150.3 h 0.374[0.104, 0.644]0.436[0.387, 0.485] 0.972[0.924, 1.000]0.420[0.198, 0.642] 1.0000.640[0.582, 0.699]

Scores use the benchmark's per-game normalized metrics (with 95% confidence intervals in brackets for locally evaluated policies) and their mean (higher is better), under the time-paused protocol. Script2Play receives no UE5 gameplay during post-training. Bold marks the best score in each column. Training scales describe the respective gameplay corpora, not matched compute budgets.

Transfer to ViZDoom

Script2Play scores 12.92 without ViZDoom trajectories or target-game RL, compared with 14.54 for the zero-shot Game-TARS MoE-mini, which uses a much larger model and substantially more gameplay data. GameWAM is trained directly on expert APPO trajectories from the four evaluation maps. The Battle maps require the agent to search for health and ammunition as well as enemies, although the instruction only asks it to kill enemies; these implicit subgoals and the need to find enemies outside the current view are difficult for our policy, which cannot track previously searched areas or supply locations from a single frame.

MethodData scaleBattle 1Battle 2Defend LineDefend CenterMean
In-domain (trained on ViZDoom APPO)
GameWAM~300 h†43.1219.0223.5215.3625.26
Zero-shot (no ViZDoom training)
Game-TARS MoE-mini~22,000 h18.8710.0415.2414.0014.54
o4-mini–7.856.7410.2411.399.06
GPT-5–12.598.9910.2413.4611.32
Gemini 2.5 Pro–7.5110.0415.245.279.52
Claude 4 Sonnet–7.347.5210.245.567.67
Our method (scripted teacher)
Script2Play (4B)150.3 h17.004.0822.947.6612.92

Mean reward over 50 episodes per map; higher is better. GameWAM uses target-domain training; the other policies transfer zero-shot. †Approximate data scale.

Effect of the supervision source

Shared-backbone comparison. We compare three policies initialized from Qwen2.5-VL-7B: OpenHA trained on human play, HiSTEVE trained with human-derived supervision and RL, and a separate Script2Play variant trained by behavior cloning on 26.9 h of Minecraft teacher demonstrations, without Roblox data. This comparison fixes the backbone; data budgets and training procedures differ. We compare OpenHA success rates for all three and evaluate OpenHA and our policy zero-shot on OmniGameArena.

(a) In-domain: OpenHA (Minecraft, success rate %)
SupervisionMethodMC dataEmb.GUICombatAvg
Human playOpenHA~145 h†30.132.531.931.5
Human + RLHiSTEVE~200 h† + RL46.027.239.037.4
ScriptedScript2Play (7B, MC only)26.9 h 49.7[44.3, 55.1]39.0[36.2, 41.8] 49.0[38.9, 59.1]45.9
(b) Out-of-distribution: OmniGameArena (UE5, normalized score)
MethodObstacle Run3DLast StandMonster ShootScene EscapeCue ChaseMean
OpenHA0.0210.0820.0100.0000.0000.023
Script2Play (7B, MC only) 0.033[0.000, 0.073]0.066[0.041, 0.090] 0.000[0.000, 0.000]0.040[0.000, 0.108] 0.000[0.000, 0.000]0.028

Supervision comparison with Qwen2.5-VL-7B. Our 7B variant uses behavior cloning on 26.9 h of Minecraft data, independently of the main 4B model. Brackets show 95% confidence intervals; bold marks column bests. HiSTEVE weights are unavailable for (b). †Approximate hour equivalents.

Our policy achieves 45.9% OpenHA success versus 31.5% and 37.4%, leading both baselines across all three task groups (a), with similar zero-shot scores to OpenHA (0.028 versus 0.023; b). Thus, the in-domain advantage persists when the backbone is held fixed.

Matched human and scripted supervision. To isolate the supervision source, we train two Qwen3.5-4B policies on GameWAM's event-aligned human VPT clips or verified Minecraft teacher demonstrations. We use 429 training tasks and hold out 136 tasks. For each training task, we cap both sources at the smaller window count and sample whole clips, so both sources share the same task distribution and the same amount of training data for every task, approximately 8.7 h per source. Both use instructions sampled from OpenHA task descriptions, the same observation/action format and training recipe, and five epochs (2,405 steps, batch size 384). A larger-data human control uses all 40.4 h available for the same training tasks, for three epochs (6,683 steps). We evaluate each epoch on tasks from the OpenHA benchmark: 240 in-corpus tasks and all 136 held-out tasks. The in-corpus tasks also appear in training, but all evaluations use environments excluded from both training sources.

OpenHA success by training epoch for scripted-teacher and human-play supervision at 8.7 h each, and a 40.4 h human control
Matched human versus scripted supervision. OpenHA success by epoch for Qwen3.5-4B with matched per-task data budgets (8.7 h per source). Bands show 95% task-bootstrap confidence intervals. Dashed line: 40.4 h human control trained for three epochs. Supplementary Table S6 of the paper reports paired differences.

Results. Scripted supervision leads at all five epochs on both task sets, with 9.7–18.3-point held-out gains. Its one-epoch in-corpus success (51.6%) exceeds the best matched-human checkpoint (45.6%); from epoch two, it also beats the 40.4 h control. Gains are largest on Embodied and Combat; the larger human control leads on GUI. Scripted supervision improves data efficiency and generalization.

Which skills transfer?

We train seven Qwen3.5-4B policies on all nonempty subsets of Combat & Shooting (C), Traversal & Parkour (T), and Dodging & Survival (D). We evaluate transfer on three OmniGameArena games, one per skill category: Monster Shoot (C), Obstacle Run3D (T), and Last Stand (D). This ablation uses subsets of the Roblox data, containing 20.5, 12.3, and 15.8 h for C, T, and D, respectively. All arms use the Script2Play training recipe for 3,000 steps, with no Minecraft data. Within each batch, the selected categories share 85% equally, and web data supplies the remaining 15%. Each policy plays 10 episodes per game under the time-paused protocol.

Training categoriesMonster ShootObstacle Run3DLast StandMean
C0.722[0.468, 0.976]0.066[0.064, 0.069]0.108[0.079, 0.138]0.299
T0.000[0.000, 0.000]0.215[0.134, 0.295]0.139[0.088, 0.190]0.118
D0.000[0.000, 0.000]0.000[0.000, 0.000]0.744[0.588, 0.900]0.248
C+T0.985[0.956, 1.000]0.370[0.259, 0.481]0.118[0.109, 0.126]0.491
C+D0.991[0.979, 1.000]0.014[0.005, 0.023]0.812[0.663, 0.961]0.606
T+D0.000[0.000, 0.000]0.233[0.126, 0.341]0.427[0.354, 0.499]0.220
C+T+D0.969[0.948, 0.990]0.288[0.170, 0.407]0.428[0.390, 0.467]0.562
Reference
Script2Play (4B)0.9720.3740.4360.594
Random0.0080.0000.0880.032

Skill-category ablation on OmniGameArena. Normalized scores over 10 episodes (mean with 95% confidence intervals; higher is better). C: Combat & Shooting; T: Traversal & Parkour; D: Dodging & Survival. Bold marks the best ablation arm per column. Reference Script2Play uses 8,000 training steps on the full mixture and 5 evaluation episodes per game.

Without the corresponding training category, Monster Shoot scores 0 and the other two games remain near Random. Adding traversal data hurts Last Stand: policies keep moving forward and fall sooner. Adding combat data also halves the dodging-data share without the same drop. Transfer therefore depends on both relevant skill coverage and the compatibility of combined skills.

Closed-loop rollouts of Script2Play

Action sequences reveal how control transfers. In OpenHA combat, the policy turns and advances toward the target before attacking. In GUI tasks, it combines mouse positioning with item-manipulation clicks. In the held-out UE5 engine, ObstacleRun3D requires forward motion interleaved with timed jumps, while CueChase requires camera adjustment, approach, and interaction. The same action-chunk representation supports these distinct control behaviors in Minecraft and UE5. These successful episodes illustrate policy behavior; the benchmark scores measure its reliability.

Five sampled observation-action pairs from successful episodes in OpenHA combat, embodied and GUI tasks and OmniGameArena's ObstacleRun3D and CueChase
Closed-loop policy actions across engines. Each row shows five sampled observation–action pairs from a successful Script2Play episode. Actions contain mouse displacement (dx, dy) in pixels and four 50 ms key groups (·: no key; LB/RB: left/right mouse button; Spc: space). Each step covers 200 ms.

Minecraft (OpenHA)

“Trim the tree by mining the acacia leaves.”
“Mine the birch log from the tree.”
“Break the coal ore to collect coal.”
“Harvest carrots from the farmland.”
“Mine the lily pad from the water.”
“Break the bell in the village.”
“Mine the netherrack in the Nether.”
“Break the cocoa pod.”
“Break the snow block.”
“Mine the kelp plant underwater.”
“Mine sand by the water.”
“Hunt the drowned.”
“Combat the zombie.”
“Combat the creeper.”
“Hunt a spider.”
“Combat the guardian.”
“Hunt the pillager.”
“Hunt a polar bear.”
“Hunt the witch.”
“Hunt the husk.”
“Hunt the skeleton.”
“Harm a turtle.”
“Craft an acacia sign.”
“Craft a stone sword at the crafting table.”
“Put obsidian and an eye of ender in the crafting grid to craft an ender_chest.”
“Smelt a baked potato.”
“Cook beef with the furnace.”
“Open the crafting table.”
“Use an anvil by right-clicking on it.”
“Use the loom by right-clicking.”

OmniGameArena

Obstacle Run3D
Score 0.54
Last Stand
Score 0.53
Monster Shoot
Score 1.00
Scene Escape
Score 0.70
Cue Chase
Score 1.00

At 2× speed.

ViZDoom

Battle 1
Reward 25
Battle 2
Reward 15
Defend the Line
Reward 38
Defend the Center
Reward 11

BibTeX

@article{liu2026gamegen,
  title   = {From Game Generation to Game Playing: Fully Automated Ability Training
             for Vision-Language-Action Models},
  author  = {Liu, Changran and Wu, Qi and Ma, Wenchao and Huang, Sharon X. and Jiang, Haomiao},
  journal = {arXiv preprint arXiv:2611.11136},
  year    = {2026}
}