Guest: Wu Yi, assistant professor at Tsinghua University’s Institute for Interdisciplinary Information Sciences and former OpenAI researcher, as introduced at release | Host: Qu Kai
Published: 2025-04-05 | Duration: 72:49 | Prepared: 2026-10-18
Official episode
Why can a model solve mathematics yet get stuck when a website changes? Why might a coding assistant do better by asking about the environment version before editing code? Wu places reinforcement learning inside a continuous decision problem: a system takes actions, receives feedback, and improves its choices. For product managers, the useful question is how a good outcome is defined, how the environment responds, and which information remains missing.
These are observations from April 2025. Comparisons between companies and predictions about technical directions retain that date and Wu’s attribution; they are not current model rankings.
1. A result can be scored without every step having a correct answer
Listen: 01:55–06:14. Classifying a cat or dog involves a labeled answer and one decision. Playing a game requires many choices—left, right, or jump—and several paths may win. Reinforcement learning, or RL, improves an action policy using rewards received after acting. It addresses processes with multiple possible choices whose quality becomes clear only later.
This changes product acceptance. Booking travel cannot be checked solely by whether one answer mentions the right cities. The system must search, compare, choose, and finish within the required constraints. There need not be one script for the intermediate steps, but the outcome needs a definition. Evaluating only a single textual response can miss failures across a sequence of actions.
Wu immediately limits his analogy with life: algorithms usually assume a known reward function, whereas people may not know what they want to optimize. Calling life RL is a metaphor for exploration, not a promise that more attempts necessarily improve it. Products can also optimize the wrong objective. An easily measured indicator is a reward proxy; whether it represents the user’s desired outcome is a separate question. This last application is editorial interpretation.
2. From instruction following to extended reasoning, verification becomes the bottleneck
Listen: 06:39–19:00. Wu describes an early GPT example: asked to explain the moon landing, a continuation model might repeat the question. Continuing text and completing an instruction are different training objectives. Reinforcement learning from human feedback, or RLHF, uses people’s comparisons to train a reward model and improve instruction following. This is a simplified teaching account, not a claim that all alignment must use one method.
Extended reasoning enlarges the search space. A model can generate a long intermediate reasoning sequence before answering, but people cannot easily judge whether ten thousand words of reasoning are better than twenty thousand. Mathematics and other tasks with checkable final answers therefore became important training settings. The process explores; the final result supplies a score, much like the earlier game example.
Thinking longer is not sufficient for product quality. Excessive reasoning on a simple question adds latency and cost. Wu praises Anthropic’s control of reasoning time at that point, while explicitly saying he does not know whether it used several routed models or better training. An observed experience cannot establish an internal architecture. After scientific and mathematical training, preserving open-ended discussion, expression, and ordinary conversation also requires work. A reasoning score cannot substitute for the whole experience.
3. Understanding and decisions complement each other across environments
Listen: 24:03–34:01; 43:22–47:23. In Wu’s working definition, a model becomes an agent when its output affects a world beyond text. Outputting “off” and connecting that instruction to a light switch is a minimal example. Early agents assembled prompts, scripts, and workflows manually. Stronger decision models may reduce that manual decomposition and explore tasks that were not written out as scripts.
That does not mean RL alone solves everything. Wu recalls early website booking and robotics attempts, arguing that weak understanding limited generalization. Practicing on dozens of airline websites does not guarantee recognizing the text and layout of a new one. His “understanding × decision making” expresses the complementarity of a foundation model and RL. It is an intuitive framework, not a measured equation. The historical accounts and explanations of failure remain his recollections, not independently verified findings here.
Slower returns from pretraining do not mean pretraining has stopped working. He discusses deeper data cleaning, synthetic multimodal data, and preserving capability in smaller models. The product implication is to distinguish a failure to understand the environment from choosing the wrong action or failing during execution and feedback. “More reasoning” cannot diagnose every problem. Better decisions also do not remove the need for business permissions and outcome checks. This diagnostic approach is editorial interpretation.
4. Asking questions is part of handling incomplete instructions
Listen: 36:43–41:17; 55:12–58:47; 01:02:02–01:02:46. Coding benchmarks often test an isolated problem, while real users revise requirements, debug, and supply missing context. Wu identifies one strength of Claude at the time: before debugging, it might ask which packages and versions were installed. Such a question can change the next action; it is not merely conversation.
Users rarely provide every constraint at once. Literal execution may proceed fluently in the wrong direction. Asking about everything transfers the burden back to the user. Evaluation should therefore examine when the system asks, what it asks, and whether the answer changes its plan. That is closer to decision quality than counting questions alone. This is an editorial extension of the discussion.
Organizational boundaries matter too. Pretraining and post-training can have different goals without becoming isolated. A post-training team discovering a missing foundation capability needs to return that gap to the teams responsible for data and pretraining. Wu emphasizes practical work across boundaries. The talent shortage includes setting up environments, reproducing results, and diagnosing complex systems, alongside algorithm knowledge. Different company stages require different people; he does not offer a universal hiring formula.
5. Training infrastructure changes the number of experiments you can evaluate
Listen: 19:15–23:37; 38:40–39:40; 58:55–01:05:38. Even mathematical reasoning training involves generation, scoring, and training. Agents add an interactive environment, increasing the number of modules and making interactions less predictable. A multi-turn task is more complex than simply repeating one call.
A system must first run stably and then run faster. Wu describes reducing his team’s training time for a 7B model from roughly a week to roughly two days. More results within the same calendar time mean more opportunities to revise data, parameters, and algorithms. This is his project experience, not an efficiency multiplier transferable to every team. The official description’s AReaL-boba result of 61.9 on AIME24 is also a contemporaneous project report, not a replication conducted for this article.
His ordering of infrastructure well ahead of data, then algorithms, should be read as a judgment about practical bottlenecks at the time. It supplies neither a fixed ratio nor a claim that algorithms do not matter. For an application team without stable tasks, feedback, or verification, starting RL training may merely add a difficult subsystem. Establishing a reproducible task environment before deciding whether training addresses a specific gap is an editorial inference.
6. Exploration can reveal goals, while reality imposes costs
Listen: 01:06:34–01:12:35. Wu is interested in diversity-driven RL: a policy must earn rewards while differing from solutions already found, rather than endlessly repeating one winning strategy. He also uses a cooperative cooking game to show how interaction provides signals about a human partner’s hidden intentions.
Product exploration should likewise avoid treating the first workable solution as the only demand. Teams can deliberately test different task descriptions, clarification methods, and collaboration paths, then observe why users choose or abandon them. Qu notes a crucial limit: simulations allow many retries; life does not. A “maximum entropy” life is Wu’s personal outlook. Real exploration needs an affordable loss boundary before action. Algorithmic freedom to experiment does not directly justify career, financial, or health decisions.
Exercise: separate the causes of one agent failure
Choose a real failed run. Preserve the input, interface, tool responses, and additional user information, then make four judgments in order:
- Did it understand? Mark any misread text, layout, or task constraint.
- Did it choose the next action well? Given the same information, write another feasible action and explain why it fits better.
- Did it miss a necessary question? Add a question that would change the plan. If the answer changes no action, consider removing it.
- Did it verify the outcome? List evidence of a tool call separately from evidence that the user’s goal was satisfied.
Only then decide whether to revise prompts, add feedback, change models, or build a training environment. This exercise is editorial synthesis, not a training method promised by Wu. Its purpose is to connect an improvement to an observable gap.
Sources and listening guide
Prepared from the complete transcript and official description and chapters. Audio was not replayed, and no internal company training systems were inspected. Capability comparisons, training times, and project scores retain their historical attribution. Life analogies are not scientific claims or personal decision advice.
- 01:51: Sequential decisions and rewards.
- 06:22: Combining RL and language models.
- 25:11: How stronger decisions change agents.
- 43:21: Understanding and transfer across websites.
- 56:10: Generalization, coding, and agent specializations.
- 58:55: Frameworks, stability, and experimental throughput.
- 01:06:05: Diversity and the real limits of exploration.
Related themes: agent architecture, model evaluation, and training infrastructure. Continue with Ding Ding on model evaluation to separate “intelligence” into specific capabilities and actual experience.