RoJev

TL;DR

Background

We started with following Lumine' s recipe, our first policy looked at a frame and wrote out avatar's action in next 200ms as text. The output was a string like W A ; W A ; W A ; W A: the mouse movement, then the keys to hold for four 50 ms slices.

It worked. The policy learned to move around the game, and later to follow instructions. But it was slow. Writing a string means decoding it token by token, and in the live game the loop topped out at about 5 decisions per second. At 5 fps a lot happens between two decisions: the avatar walks off a ledge, or a fireball arrives before the model has even looked again.

The idea for a fix came from Jev, a decision model from TypeSafe AI that we work with. Jev never writes text. Each call gives it a state and a question, and the question lists the possible answers. Jev returns a probability for every answer in one pass, so it can only ever answer with an option that it was offered.

This makes Jev fast enough to play games in real time, and people already use it that way. In each of these setups, Jev reads the game's structured state in text, not pixels, and it picks one tool call from a fixed list instead of writing a button-level action out. Each option is a command that the harness implements in code, such as "run right and jump", "build a wall" or "attack this unit". Jev only chooses the call, and the harness carries it out in the game. The table compares the input and the output of some of these setups with RoJev:

RoJev plays like a player instead. At every decision, it sees one screen frame and one instruction, and it picks one of 16 key combinations to hold for the next 50 ms. The answer is a single letter, so a decision takes about 37 ms, which is fast enough for 20 fps even with 480p frames as inputs. Hence the name: RoJev, a Jev-style policy for Roblox games.

How RoJev makes a decision

For each decision, RoJev receives these items:

• A system line: "Apply the question to the state. Choose exactly one of the listed options. Respond with only its uppercase letter…"
• One RGB frame from the default third-person camera. The frame is 832 px wide and includes the game UI.
• One instruction, for example "Walk to the red box ..."
• The question "Which keys should the player hold for the next 50 ms?" and the list of lettered options.

RoJev selects one of these 16 options:

Training

The base model is Qwen3.5-4B. We train it in four stages, and each stage starts from the previous one.

Stage 1 — Pretraining with human gameplay

RoJev first copies human players of Natural Disaster Survival (NDS), with the non-historical pretraining recipe of Lumine. All the human gameplay that RoJev trains on comes from Pixel2Play (P2P), an open dataset from Elefant AI (paper). Each sample is one frame and the keys that the player holds in the next 200 ms, written as a key string. There is no instruction and no history. The data is 16,374 NDS recordings from P2P, about 4.7 million samples, plus about 20% web data.

Stage 2 — Instruction tuning

Next, each frame gets a short instruction, for example "Climb the ladder onto the roof." The data is about 235,000 frames from the same P2P recordings of NDS. The answer is still a key string.

Stage 3 — The choice format

Now the answer becomes one letter. The data stays the same, but each sample becomes a question:

• Label: the keys held in the first 50 ms.
• Options: shuffled for every sample, so the model reads the list instead of memorizing letters.
• Loss: cross-entropy over the listed letters only.

The ability datasets also use NDS frames from P2P, and they are built from contrastive pairs: the same frame comes with different instructions and different answers, such as two targets in one scene, or a left and a right beacon drawn on one image. Each frame also has a mirrored copy that swaps left and right in the answer. So the image alone never gives the answer, and the model must read the instruction.

Stage 4 — Learning from scripted teachers

After stage 3, the model follows simple instructions, but it does not find a route, steer with the camera, use objects or react to danger. Stage 4 teaches these skills with scripted teachers.

• The teachers: each teacher is a script that reads the game state, such as the player's position, the goal and a pathfinding route. Every 200 ms, it picks the keys that follow the route, and it turns the camera toward the route.
• The data: for each choice of a teacher, we save the frame that the model sees and use the teacher's keys as the label.

Stage 4 runs in two rounds. Round 1 teaches route following and camera steering. Round 2 uses practice places with random layouts, such as climbing a ladder, crossing lava, and pressing buttons.

How we serve RoJev

We use vLLM to serve RoJev. Each decision is one small request:

• A prompt of approximately 770 tokens.
• One frame of 832 × 480 px, which is 390 vision tokens.
• One output token.

At batch size 1, the GPU uses much of the time to start many small kernels. Thus, we use CUDA graphs. A CUDA graph records a fixed block of work one time. Then the GPU replays the full block with one launch.

• Vision encoder: we record a graph for a budget of 400 tokens. This budget is sufficient for one frame.
• Prefill: we record a graph for a padded prompt length of 822 tokens. This length is sufficient for all prompts of the choice format.

Thus, each decision replays only recorded graphs. The table shows the median time for one decision. We used one B200 GPU, batch size 1 and 150 saved live frames.

The graphs do not change the answers of the model. On the 150 frames, the served model and an fp32 reference agree on 147 frames. This result is the same with and without the graphs. A longer prompt or a second image is possible, but the model then runs without the graphs.

Playing NDS with a planner

RoJev follows short instructions, but it does not decide where to go. In a live NDS round, a reasoning model writes the instructions for it, and the two models run side by side:

• The planner (Qwen3.8-27B) answers about every 2.5 s. It sees the current frame, its last two observations with the frames they came from, up to four remembered facts (each one expires after 180 s), and its recent instructions.
• Its answer is one instruction in the short sentences that RoJev was trained on, such as "Go to the red brick building.", "Move backward.", "Rotate your view left." or "Stay where you are.", plus a one-sentence rationale and a confidence. Below a confidence of 0.5, RoJev gets "Stay where you are." instead.

Limits and next steps

• Limits of RoJev:

  • No memory and no mouse. RoJev sees one frame at a time, no historical frames.

• Limits of the planner:

  • Qwen3.8-27B does not plan well enough. A stronger model such as GPT-6 Astra takes about 6 s per call, compared with about 2.5 s for Qwen3.8-27B.

• Next steps:

  • Add more games, not only NDS.
  • Give the model a short context of more than one frame.
  • Post-train the reasoning model, so that it plans better at about the latency of Qwen3.8-27B.
  • Improve RoJev with reinforcement learning.
  • Support GUI interaction with the mouse, such as menus, shops and dialog buttons.