Deep Dive · XiaoHu Explains

Former OpenAI CTO Mira Murati: AI Should Amplify Humans, Not Replace Them

From tacit knowledge to interaction bandwidth to model alignment — the case for why AI development can't do without human participation.

TL;DR
  • Thinking Machines Lab published a mission blog proposing an AI development path centered on "extending human will and judgment," positioned as a counterpoint to the industry's "autonomous intelligence" path.
  • The company lists three technical investments: training frontier models, building Tinker — a tool that lets users fine-tune model weights themselves — and researching "interactive models" that support real-time, multimodal interaction.
  • Using chess and mathematical proofs as examples, the piece argues that every domain where AI can autonomously surpass humans meets two conditions: the goal is static and clearly expressible, and there's no hidden tacit knowledge involved — conditions most real-world work fails to meet.
  • The article criticizes current evaluation systems — such as METR's "autonomous task duration" — for measuring only what AI can do on its own, not how well humans and AI perform together.
  • The article makes the case for "decentralized alignment," pointing out that labs commonly use their previous flagship model to generate training data and reward signals for the next one — a practice that homogenizes model "personality" across generations.
⚑ This piece is compiled from Thinking Machines Lab's official mission blog. All claims, arguments, and descriptions of technical direction are the company's own statements — not independent third-party evaluation or verification.
1Company & Mission

What Kind of AI Is This Company Trying to Build

Thinking Machines Lab (an AI lab founded by former OpenAI CTO Mira Murati) has published a mission blog stating the company's mission as "AI should extend human will and judgment," and laying out three technical paths to get there.

The blog sets a direction: use AI to amplify human will and judgment, then breaks that direction down into three concrete technical investments.

What makes this piece notable is that, in the form of a company mission, it openly proposes a technical direction distinct from the industry's mainstream "autonomous intelligence" path, and publicly lists three concrete investments: training strong models, building the fine-tuning tool Tinker, and researching interactive models.
Train Strong Models

Keep pushing model capabilities — multimodal interaction, customizability — toward the frontier, since human judgment can only take effect through a strong model.

Build Tinker

A toolkit that lets users fine-tune model weights themselves, reshaping AI into a version built for their own needs.

Research Interactive Models

Give models native support for real-time, multimodal human-AI interaction, training this capability directly into the model itself.

The company also says it will make the related research and practical handbooks public to the scientific community, so more people can gain access to this fine-tuning capability.

2Core Argument

AI Can Win Solo at Chess and Math Proofs; Most Jobs, It Can't

AI serves the knowledge that people rely on to get things done: how to do something well, what's even worth doing. This kind of knowledge is generated continuously by the people doing the work. A chef refining a new dish, a shopkeeper adjusting how goods are arranged and priced on a shelf — both are chasing a complex, outsider-illegible set of goals, using their own tricks of the trade. Those tricks keep updating with real-world feedback; they can't be organized into a static list you file into a database, and they're local — change the restaurant, change the shop, and both the goal and the method change with it. The full knowledge of any chef or shopkeeper lives scattered across each individual chef's and shopkeeper's own hands.

This dispersal of knowledge is the source of the whole system's diversity, adaptability, and resilience — and it's why free markets beat planned economies. Planned economies fail for a reason rooted in the nature of productive knowledge itself: it's tacit, tied to the immediate situation, always changing, and held privately by the people doing the work. Trying to centralize that knowledge and hand it to a single centralized intelligence runs into the exact same wall.

This kind of knowledge-you-can't-quite-say has a name: "tacit knowledge" — the judgment calls people make on the job from feel, experience, and instinct, the kind that can't be written into a manual or filed into a database for someone else to use. It can only be built up and updated by a person, continuously, through the work itself.

Analogy

A seasoned chef tastes the soup and just knows how much salt to add — but can't tell you the exact gram count. That's tacit knowledge.

Core Argument

The article argues that every domain where AI can surpass humans on its own meets two conditions at once: first, the goal is static and can be clearly stated — win a game of chess, prove a theorem; second, the domain has no hidden tacit knowledge — the rules are fully public, and every move on the board is visible to everyone. Chess and mathematics meet both conditions: the strongest chess engines can out-train humans through self-play alone, and frontier models are independently solving hard math problems. Step outside the board and the proof, and intelligence alone isn't enough.

Chess / Mathematical Proofs
  • The goal is fixed from the start: win a game, or prove a theorem
  • The rules are fully public — every move on the board is visible to everyone
  • AI can surpass humans just by training against itself
Kitchens, Shops, and Most Real-World Work
  • The goal keeps shifting — every shop wants something different and is chasing something different
  • It runs on the feel of the person doing the work, the kind you can't put in a manual
  • The knowledge is scattered across every chef and every shopkeeper — there's no way to package it up and feed it to one AI

The AI worth building should help an organization keep cultivating and updating this unique knowledge, letting it grow alongside the people who hold it.

3Case Study

Toyota Brought the Master Craftsmen Back to the Line

In 2014, Toyota — long known for its automation — brought veteran craftsmen back onto the production line, explicitly to cultivate craftsmanship and skill. The man who led the effort, Mitsuru Kawai, explained it this way: "To master the machines, you first need the knowledge and skill to teach them."

Producing knowledge and applying intelligence reinforce each other. The work humans do will keep changing, shifting more toward the parts only a human can bring — but the best organizations will push both to the limit. AI should help every organization become the best version of itself, on its own path, preserving the differences that already exist between organizations.

4Collaboration Bottleneck

Between Humans and AI Sits a Text Box That's Too Narrow

Keeping humans continuously involved in setting goals and sharing knowledge with AI is not the same as resisting automation. Whatever a machine can reliably do on its own, it should do. But it also needs to know when to act on its own and when to check in and wait for feedback — the same way people do when working together on a team. This raises new technical requirements for how AI is designed and evaluated.

Right now, the biggest bottleneck to bringing human knowledge and judgment into AI's work is the communication channel between human and AI: a small text box, followed by a long wait. That channel is too narrow to carry the full richness of human judgment, and too slow to keep pace with continuous feedback. Human-to-human collaboration works best in real time — interrupting, correcting, glancing back, gesturing while talking, changing your mind mid-sentence.

Now

Type into a small box, hit send, wait for the model to finish a whole paragraph, then read it over to decide what to say next.

What Interactive Models Envision

Interrupt in real time, correct on the spot, gesture while talking, change your mind mid-sentence — just like talking to a person — and the model understands and keeps pace, directly.

This vision has a name: "interactive models" — training the capability for "real-time, multimodal interaction" directly into the model itself, so that as the model gets smarter, its ability to interact grows right along with it. An interface designed this way doesn't just permit human participation — it actively invites and rewards it.

5Evaluation Blind Spot

The Yardstick We Use for AI Progress Only Measures Half the Picture

Another challenge is setting the right target for evaluation and optimization. The most common yardstick for AI intelligence right now is how long it can independently execute software tasks without human help — that's exactly what METR and similar organizations' charts track. These metrics will keep improving, but they only measure what AI can do alone, not what humans and machines can accomplish together.

What METR Measures

How long AI can complete software tasks independently, without human help — the longer it lasts, the more capable it's considered.

What It Doesn't Measure

Whether what humans and AI produce together, as a team, is good enough — or enough of it.

Measuring that second kind of outcome is more complicated, and no single lab can do it alone — every organization has to judge for itself whether AI is sharpening its judgment, growing new knowledge, and helping it reach its goals. Building AI that makes its users stronger over the long run also aligns everyone's incentives: a lab where every customer shares the same model survives by absorbing what's distinctive about each user; but build AI that can be customized and collaborated with, and the lab's interests become tied to customers realizing their own distinctive strengths. Getting that edge requires the organization to own and shape the AI itself.

6Power & Alignment

If AI's Values Are Decided by Just a Few Places

A person's values, like a person's knowledge, live in that person's own head — hard to consolidate into one place. Yet right now, AI's values and its say-so are being decided by only a handful of places. A unified alignment authority, no matter how well it's run, becomes a center of power worth seizing.

This is especially dangerous in a world where most valuable work is done by AI alone, with little need for human involvement. The social contract between governments, businesses, and citizens ultimately rests on individual productive capacity: government sovereignty and corporate profit are both built on top of it. Once that kind of power no longer needs to draw anything from people, its incentive to look after human needs and values disappears too — and it turns instead to perpetuating itself.

7Decentralized Alignment

Every New Model Generation Inherits the Last One's Temperament

No matter how good the intentions, a model trained in a single place ends up encoding just one set of values — its owner's.

Core Mechanism

The common practice across labs right now is to use their own previous flagship model to generate the training data and scoring criteria for the next one. Whatever "personality" grows out of that loop, every user gets the same one — and it passes from generation to generation: each generation grows up on the previous one's output, graded by the previous one's taste. A single alignment standard suppresses creativity and diversity, and slows down progress for the whole field. Free speech and free markets let new ideas, products, and services keep emerging and competing with each other — instead of averaging everyone's preferences at one moment in time and setting that as the sole standard.

Previous Flagship Current Flagship Next Flagship Generates data + scores Generates data + scores Same "temperament," passed down the line

For organizations and individuals to align AI with their own values, those values need to be encoded into the model's weights. If a user's values and needs can only sway the model through prompting, its surface behavior shifts while its deeper habits stay put — and a centralized model that can be dramatically reshaped by prompts alone is also easier to break through repeated jailbreak attempts. The power to substantially change a model is the same power that can break it. Keeping machines safe is an ongoing process of judgment and choice, one with no finish line. Thinking Machines Lab says it wants to give the people making those judgment calls stronger tools, without taking that ownership away from them.

8Vision

Let Everyone Write Their Own Answer

The tech industry has already made great progress teaching machines how to think — but what machines should think about has to stay in human hands: what's worth pursuing, what's worth doing, how to spend this stretch of time. The article's hope is to give everyone the ability to make their own answer part of frontier AI's development.

Humanity got here on the strength of individual distinctiveness and the creative tension between people. The alignment this piece envisions is a property of an entire ecosystem: many AIs, raised in different places, each different from the others, arguing with each other, competing with each other, learning from each other. Preserving that difference is the point in itself.

The mainstream path AI development is on right now pushes toward centralization and autonomy, treating human participation as a trade-off — participation versus capability, ownership versus safety and alignment, pick one. The article frames these as technical problems to solve instead: make AI stronger because it encourages human participation, let organizations benefit in the long run from shaping AI into their own image, and let alignment grow naturally out of a diverse population of AI shaped by its own users.

Left: a single model absorbs everyone and produces one shape. Right: multiple differently-shaped AIs each connect on their own, keeping their own shape.

The article's conclusion is that different roads lead to different futures, and which road to take is a choice. Thinking Machines Lab says the technology they're building is meant to let "the born" and "the built" walk that road together.

Technology carries good and bad usages so close together that it is difficult to separate the lion from the lamb.John von Neumann, 1955
Content sourced from Thinking Machines Lab's official mission blog (thinkingmachines.ai/blog/the-future-worth-building-is-human), compiled by XiaoHu Explains.