teaching a 9B model to solve the rubik’s cube

tl;dr

I trained Qwen3.5-9B, a small open model, to solve a Rubik’s cube from a real scramble. It reads the cube as text, turns it with a move tool, and has no solver and no code execution. It started out solving almost nothing. It now solves 66% of real scrambles end to end, and 88% of them within two tries.

→ the model is open source on Hugging Face, with a one‑file script to run it in the same environment it was trained in.

I set it up as a reinforcement learning problem: an environment with the cube and its tools, a reward for progress, and GRPO on top. That was the baseline, and on its own it barely moved. What moved the numbers was post‑training with SFT: writing out the reasoning I wanted the model to do, generated and checked by a cube engine, and rewriting it every time the model got stuck. Every time I turned a search in its head into a lookup, it jumped. RL came back at the end to polish, once the model could already solve most positions some of the time.

Watch it solve. Pick one of the 28 real scrambles it solved, press play, and read what it wrote at every turn:

open the solver full page →

background

The cube looks like a toy, but it’s a brutal test for a language model. You have to track the exact state of 54 stickers across dozens of moves, understand how each turn moves them in 3D, and plan a long way ahead. One wrong move quietly wrecks everything you built.

Frontier models are bad at it. CubeBench tested GPT‑5, Gemini 2.5 Pro, Claude Sonnet 4 and others as cube‑solving agents: on scrambles 8 or more moves from solved, every model scored 0%. Give them a solver tool and GPT‑5 jumps to 95%. So the models know what a move does. What breaks is the long‑horizon planning.

And a small network trained from scratch doesn’t get around this: classic solvers like DeepCubeA lean on a search algorithm to do the planning, CubeBench’s own RL‑trained network solved 0% of long scrambles without one, and my model has to read the cube as text and plan with no search at all.

I wanted to know if a 9B model could learn the planning itself.

I’d never done RL before this. I talked to a couple of friends at different RL‑environment startups, who pointed me at what to read (the sources are at the end), and I learned the rest by running it.

methods & results

The approach had three parts:

1. An RL environment as the baseline: the cube, three tools, and a reward a program can check exactly, trained with GRPO.
2. A curriculum: solve the cube in CFOP stages, one at a time, each starting from the model before it.
3. SFT post‑training: engine‑written reasoning, rewritten wherever the model got stuck, then RL once it could solve a stage some of the time.

The model sees the cube as six 3×3 grids of color letters, one per face, and acts through three tools: move (e.g. "R U R'"), peek and state. Thinking mode is off, so all of its reasoning is visible in its reply. The CFOP stages:

1. cross: the 4 white edges around the white center.
2. F2L (first two layers): the 4 corner‑edge pairs under the cross.
3. OLL (orient the last layer): make the top face yellow.
4. PLL (permute the last layer): move the top pieces into place.

Training used prime‑rl and verifiers with LoRA. I started on GPUs rented through Prime Intellect, but the deals weren’t stable (offers vanished or machines came up broken), so I moved to Modal.

cross

first attempts

I started on the easiest possible tasks, and spent the first week there: crosses and “undo” puzzles only 1–2 moves from solved. Here’s everything I tried, in order. Hover over a row to preview it, or click to keep it open, and see what happened, with a real excerpt of what Qwen wrote.

held‑out evals on 1–2‑move tasks (fractions are solved / episodes), except where noted

I need to analyze the current state of the cube and solve it to reach a white cross.

**Current State Analysis:**

1.  **White Center**: I need to identify the white center to know which face is the "top" (U) for the white cross.
    *   Looking at the U face: `R W R / Y Y B / R R G` - Center is Y (Yellow).
    *   D face: `B W B / W W W / G O O` - Center is W (White).

[… about 6,800 more characters …]

[thinking]
Let me analyze the cube state to understand where the white cross is and what needs to be done.
...
The goal is to make a white cross. This means:
1. White should be on the center of U face

turn 1: move("L")
turn 2: move("L'")
turn 3: move("F2")
turn 4: move("B2")
turn 5: move("U' R'")
...

The white center is on D. I'll locate the four white edges and their side colors before moving them.
The white edges are at U-B, F-R, D-L, and U-R; the D-L edge is flipped. I'll lift that edge into the upper layer while preserving the others.   -> move("L2")
That turn moved the white-red edge to F-L. I'll use F' to bring it to U-F with white facing up.   -> move("F'")

**Solving cross positions efficiently**

I need to solve the cross of a Rubik's Cube efficiently. The centers are mapped as follows: U, Y, D, W, F, G, R, O, L, R, B, B. I can see the white edges' positions: U top is blue (UB), U mid left is blue (UL), U mid right is red (UR) ...

Three white-red edges are on the bottom face, and the white-blue edge is on top. I'll move blue down first, then place the red edge.
...
The white-blue edge is now correctly placed. The white-red edge is still at the right-front position, so an L turn should put it under the red center.

I have analyzed the current state of the cube.

**Current State Analysis:**
- **U (Up):** Mostly Yellow, but has a row of Red (R R R) in the middle. This is incorrect; U should be all White or all Yellow ...
[…]
What if we do **L2**?
Moves U-col0 to D-col0.
...
Then maybe rotate D?
If we do **D

I have analyzed the current state of the Rubik's cube.

**Current State Analysis:**
- **U (Up):** Contains R and Y. Not solved.
- **R (Right):** Mostly O and Y. Not solved.
- **F (Front):** All Green (G). This face is solved.
[…]
       So D-col0 is filled by B-col2.
       B face is all Blue. B-col2 is Blue.
       So Os are not directly moved here.
       Wait, the Os are on D-row 2.

I have analyzed the current state of the Rubik's cube.

**Current State Analysis:**
- **Up (U):** All Yellow (Solved)
- **Down (D):** All White (Solved)
- **Front (F):** Top layer is Red, bottom two layers are Green.
[…]
- New L-top = Old F-top (R) -> Correct for Red face.

The cube will be fully solved.

Plan:
Execute `U`.

...
  "Give a sequence ... as short as you can."
  Shortest is 0.
  I'll output an empty string inside the tags.

  Wait, looking at similar problems, sometimes if the state is solved, the answer is empty.
  I will provide an empty answer.

<answer> </answer>

why rejection sampling worked

The one clear win was the simplest: let the RL model attempt thousands of tasks, keep only the attempts the engine verified as solved, and fine‑tune on those. Undo solves went from 16% to 34%, and 2‑move undos went from 0% to 29%, a skill it didn’t have before. Why it beat everything else:

• Nothing is hidden. Unlike the teacher’s explanations, every kept example is reasoning the model really wrote, which really led to a solve. The work is in the tokens.
• It’s in the model’s own voice. There’s no style gap to bridge. It learns which of its own habits work, not someone else’s formatting.
• The verifier does the filtering. Only correct solves survive, so it can’t learn confident nonsense.

But it had a hard ceiling, exactly the one Will Brown describes: it can only lock in what the model can already sometimes do. Crosses had only 45 kept examples and stayed flat. Depth 3 had zero solves to keep, so it stayed at zero.

That pointed at the real move. Rejection sampling worked because the examples were verified reasoning with the work written out. My engine can produce that for any position, at any depth, in unlimited quantity. So from here on, I wrote the reasoning myself, generated by the engine and checked line by line.

hill‑climbing the reasoning

Each version below is the same model and the same kind of SFT run. The only thing that changed was the shape of the reasoning in the training data. After every run I graded the model’s own working against the real cube to find the step that broke, then rewrote that step.

short crosses: 1–4 moves, 16 per depth (v5/v6: 3–4 moves on the position bank). real scrambles: a fixed bank of positions, 8 tries each, weighted like real scrambles (5–7 moves).

• step: learned the format but misread the cube: edges stated as facts meant finding stickers in one silent jump.
• scan: copying the squares first fixed reading the stickers (100%), but naming edges was still a search.
• scan v2: naming edges in copy order made reading a pure lookup; choosing a move was still a search.
• deep: more examples of a hard step didn’t make it easy. Worse per move, dropped.
• v4 (RL): slow gains (moves closer 38% → 43%), polishing a step that was still a search.
• v5: one listed move per edge turned choosing into a lookup: 14.5% → 64.2%.
• v6 (RL): v5 already solved most positions some of the time, so RL had signal: 75.9%.

Moves:
  W-B (1 move from home): B2 brings it home, keeps the home edges.
  W-G (1 move from home): F2 brings it home, keeps the home edges.
  W-R (2 moves from home): F2 brings it 1 closer, keeps the home edges.
-> B2
B2: W-B to D row 2 col 1 (home).
After: 2 of 4 home.

F2L

you can’t imitate a search

F2L is where the lesson really landed. Every version here starts from the v6 cross model.

F2L solved, by how many pairs were left to place. Prompting v6 with the method (no training) also stayed at 0.

• F1, copy the expert’s plan: a plan from a search is a magic string; its moves matched the expert’s 7% of the time.
• F3, memorize cases: every move became a lookup (extract, set up, or play the case’s algorithm), but 236 algorithms was too many to remember.
• F4, one table seen from one side: the 4 slots are the same 59 cases, so read each from the front‑right and swap letters back. Each case seen ~4× as often.

Step: case. BR corner in its slot (DRB), white on R; BR edge at UL, B on U.
FR view (R>F, B>R, L>B, F>L): corner DFR, white on F; edge UB, R-color on U.
FR algorithm: R U' R' F' U' F
For BR (F>R, R>B, B>L, L>F): B U' B' R' U' R

I’d planned RL on F2L next, and launched it. But its first eval showed F4 already solving 1–2 pairs nearly every time. With every group of 8 a sweep of wins, RL could only shave a few moves off. I stopped it after 11 minutes.

OLL + PLL

mostly memorization, then the full solve

We gave it the beginner two‑look algorithms (16, all face moves) and had it solve whole cubes, cross to last layer, in one conversation.

conclusions

what i learned

• The basics of RL: GRPO only learns from problems the model already solves some of the time.
• RLVR: a perfectly checkable task makes both the reward and the training data trustworthy.
• Picking a navigable problem: hard enough that frontier models fail, cheap enough to iterate on in ~$15 runs.
• Lookups, not searches: a model can imitate a lookup, not a search, and small tables beat big ones.
• SFT moves it, RL polishes it: post‑training did the heavy lifting; RL sharpened what was already there.

what’s next

Choosing an unverifiable problem. The cube made everything checkable, and most problems worth solving aren’t like that: writing, analysis, research. The next step is learning how to post‑train when there’s no exact answer to check, where the reward is a rubric or a judge and the training data can’t be checked line by line.

Getting really good at one specific task with LLMs. The recipe here seems general: build an environment with a reliable check, write the reasoning as lookups instead of searches, keep the lookup tables small, SFT first, then RL once the model sometimes succeeds. I want to see how far that goes on a task that matters outside of puzzles.

And for the cube: rerun the full solves with sensible limits to get the true solve rate, fix the weakest stage (about 78% on 6‑move crosses) with DAgger, letting the model play and having the engine label the moves it should have made, then the two PLL cases it fumbles (H and Z perms), and eventually RL on whole solves.

sources

• Paper Instruments, Ultramarine 1 Flash: Scaling Non‑Verifiable RL. My intro article. I saw it going around on X and ended up revisiting it once I’d started.
• Hugging Face, The ultimate guide to RL environments. My baseline understanding of what an environment is in the LLM age, and why it matters.
• Cameron R. Wolfe, Reinforcement Learning for LLMs: The Complete Guide. Extremely in depth: the equations, and what each piece is actually used for.
• Will Brown, On SFT, RL, and on‑policy distillation. I came across him at an event and checked out his writing afterwards. Really good writing.
• Chakra Labs, RL Environments are Worthless. Recommended by my friend Viren Govin. A good read for a feel of how RL works in practice.
• Sean Cai, on X. Good posts on the RL environment space.
• Also: CubeBench (Gao et al.) for how frontier models do on the cube, and DeepCubeA (Agostinelli et al.) for the classic network‑plus‑search approach.