OpenAI nearly tripled its ARC-AGI-3 score without changing the model
The public-set score for the same GPT-5.6 Sol rose from 13.3% to 38.3% after two context settings changed. The useful lesson is not that a model became magically smarter. It is that the way an agent is run is part of what a benchmark measures.
This was a systems change, not a model upgrade
ARC-AGI-3 puts an agent in unfamiliar two-dimensional games. It receives no rulebook: it has to observe, act, make a hypothesis, and update that hypothesis across a sequence of moves. That makes continuity of state central to the task.
OpenAI found two constraints in the generic harness. Private reasoning was discarded after every game action. And, as the conversation became long, rolling truncation removed the oldest actions. The agent could see some recent moves, but not the plans and deductions that had produced them.
- Private reasoning is not carried to the next turn.
- Old history falls out of a rolling window.
- The agent repeatedly rebuilds its model of the game.
- Reasoning is retained across calls.
- Older context is compacted into usable state.
- Plans, failed paths and observations can carry forward.
What the two settings actually change
Retained reasoning keeps the rationale attached to the next action
In a multi-step task, visible tool output is not the same thing as the model's working plan. If that plan disappears after each action, the model has to infer the rules again. OpenAI used the previous response in the Responses API so the next action could build on earlier reasoning.
Compaction preserves the useful residue of old context
Truncation manages length by deletion. Compaction instead carries forward a compressed account of goals, evidence, rejected routes and outstanding questions. It is not lossless memory, but it is materially different from forgetting the beginning of the run.
Why did the score rise while tokens fell?
More context does not necessarily mean more repeated work. OpenAI's explanation is that the agent stopped re-deriving its understanding before every move. It could spend tokens on the next uncertainty instead of replaying prior analysis. The result is a useful engineering heuristic: inspect the points where an agent forgets before increasing model size or reasoning budget.
Benchmarks rarely measure models in isolation. They also measure less visible choices about API settings, harness design, and prompting.
OpenAI, paraphrased
What to take into an agent evaluation
- Treat context policy as a capability. Tool traces, decisions and failure reasons need an explicit path into the next turn.
- Report the runner. API, prompt, tool set, context policy and budget belong next to a benchmark result.
- Do not equate compaction with simple truncation. A useful summary preserves constraints, verified facts and unresolved branches.
- State the question being answered. A minimal, standardized harness and a production-grade agent system are both valid targets, but they answer different questions.