Translating the full HTML, visible text only, structure and markup preserved.Writing the full English translation carefully, then emitting the complete HTML.
Research brief · XiaoHu Explains

Two API settings took GPT-5.6 Sol from 13.3% to 38.3% on ARC-AGI-3

Same model. Different harness: keep private reasoning, and compact instead of deleting. Output tokens per game fell to about one-sixth.
One-minute overview
  • OpenAI walked through a failed benchmark run of its own model: same GPT-5.6 Sol, same tasks, only two API settings flipped on—and the ARC-AGI-3 public-set score jumped from 13.3% to 38.3%.
  • The bug was in the harness that runs the model. After every step, the model's private reasoning was thrown away, so the next step had to re-learn the game from scratch.
  • Second problem: when context filled up, the oldest messages were deleted—including past keypresses. The model effectively had no past.
  • The fix is two switches: pass the previous response ID through the Responses API so reasoning sticks around, and turn on compaction so full context compresses into a summary at the limit instead of truncating the front.
  • The cost win is bigger than the score win: at max reasoning, output tokens per game fell from 2.9M to 0.49M. The new harness at "high" matched the old harness at "xhigh" on score, with about 12× fewer tokens.
  • ARC's own May analysis pointed elsewhere—three failure modes around never building a world model. And only the official-harness scores of 13.33% (public set) and 7.78% (semi-private set) are ARC Prize–verified.
⚑ Source material is OpenAI's official blog about its own model and APIs. The 38.3% result is OpenAI's self-run score. The only numbers independently verified by ARC Prize are 13.33% and 7.78% under the official harness—keep that boundary in mind when reading. Extra sources we checked ourselves are listed at the end.
Opening

Same model, same tasks, nearly 3× the score

OpenAI published a concrete postmortem: same GPT-5.6 Sol, same benchmark tasks, only two API settings turned on—score from 13.3% to 38.3%, and output tokens per game down to about one-sixth. If you run any multi-step job over the API, those two switches decide both score and bill.

When a model scores low, the usual first guess is "the model is weak." This time the answer is different: the harness was wiping the model's memory. They tested on ARC-AGI-3—puzzle games with no manual—and ran all five reasoning-effort levels.

What is a harness?
AI self-improvement starts outside the weights: Lilian Weng on harness engineering
The external program that runs a model is called a harness. What it controls—and how far it can go—is covered in full there.
13.3 → 38.3%ARC-AGI-3 public set, max reasoning
2.9M → 0.49MOutput tokens per game ≈ 1/6
48%Human average (OpenAI estimate from public logs)

Calling this a failure only makes sense because Sol looks nothing like one elsewhere. It resolved the cycle double cover conjecture in graph theory open for decades; cleared Pokémon FireRed from pixels alone with no game API; beat Slay the Spire in Codex; and finished the early levels of Baba Is You. In a log Wharton's Ethan Mollick shared, it won the Slay the Spire 2 daily random challenge in Codex—a game released after Sol's knowledge cutoff.

Ethan Mollick showing GPT-5.6 Sol clearing the Slay the Spire 2 daily challenge in Codex
Ethan Mollick showing GPT-5.6 Sol winning the Slay the Spire 2 daily random challenge in Codex. The game shipped after Sol's knowledge cutoff, so it was not in training data. Image source: Ethan Mollick's post as cited on the OpenAI blog.

That same model scored a little over 10% on 25 short 2D puzzle games. The side-by-side video below is the same game (id cd82) and the same Sol: official harness on the left, OpenAI's Responses API rewrite on the right. The game has six levels; no frontier model on the leaderboard clears level one. With the new harness, Sol clears all six.

Sped-up gameplay. Left: official harness. Right: Responses API harness with retained reasoning and compaction. Same model—GPT-5.6 Sol at max reasoning. Each side shows three counters: steps taken, levels cleared, output tokens burned. Video source: OpenAI.
Don't mix these two scores: 7.78% and 13.33% are both official harness at max reasoning. They differ only by set: 7.78% is semi-private, 13.33% is public—both ARC Prize–verified. The 38.3% used later for comparison is on the public set, so its baseline is 13.33%.
Rules of the test

No manual—and the score is not "how many puzzles solved"

ARC-AGI-3 is an ARC Prize benchmark: 135 hand-built mini-game environments, with 25 demos public so anyone can play at arcprize.org. One rule runs through everything: no manual. No object list, no rule text, no goal description, no intermediate reward scores.

Each step, the model gets only this: a 64×64 grid of cells colored 0–15, plus seven buttons it can press.

What it sees
A 64×64 grid; each cell is a color ID from 0 to 15
What it can press
RESET
restart run
ACTION1
up
ACTION2
down
ACTION3
left
ACTION4
right
ACTION5
confirm / delete
ACTION6
click a cell (x, y each 0–63)
Schematic we drew. Left grid is a casual color layout, not a real level. Right-side key names and roles match ARC's open-source definitions. Which keys matter in each game is also hidden—the model has to discover that.

The only way through: act, guess the rules, revise when wrong. Press a key, watch what changes, infer what that key does, then test the next step.