42 Zhangjing episode analysis · Benchmark and Evaluation
Guest: 丁丁, introduced as a former product manager at WeChat, Meituan and Moonshot, previously responsible for Kimi App | Host: Qu Kai
Published: May 5, 2025 | Duration: 41 minutes 12 seconds | Prepared: October 18, 2026
Official episode and introduction
Why can a model look expert on a leaderboard and still fail at work?
A benchmark is a set of test tasks and corresponding standards. Evaluation is the process of using standards to analyze and judge performance. Dingding begins with this distinction: public tasks reflect some capabilities without necessarily covering a product’s actual inputs and processes.
Users also encounter system prompts, search, knowledge bases and interfaces. Failure in one component can spoil the experience despite a high-scoring foundation model. Yao Shunyu’s article about AI’s “second half” introduces the conversation. This analysis examines the interview’s interpretation and practices rather than replacing an independent reading of that article. Model preferences and industry assessments retain their May 2025 context.
1. More users do not automatically supply better capability tests
Episode reference: 05:50–09:27. Dingding does not dismiss DAU, or daily active users. Real users supply industry perspectives and input forms a team may never anticipate. She rejects equating scale directly with progress in intelligence.
A light entertainment request and a professional’s research report developed over dozens of exchanges expose different requirements. High-quality data must align with the capability being improved. This is not a ranking of people by identity: the question is what problem an input reveals and whether it produces useful feedback.
Pretraining, post-training and infrastructure also contribute to the final experience. A company’s larger user base therefore cannot independently establish superior intelligence. Editorial interpretation: Specify the capability to improve before collecting feedback, then ask whether the data fills its gaps. User count signals scale; it is not a conversion formula for training value.
2. Good tasks contain input, output and judgment, with varying difficulty
Episode reference: 10:08–11:58, 17:21–19:41 and 22:14–24:22. Dingding gives a deep-search task: retrieve ten years of a company’s financial reports and predict changes in this year’s net profit. Besides input and output, the task needs standards for evidence, completeness and inference. This is an evaluation example, not a financial forecast supplied here.
As models improve, tasks everyone solves lose distinguishing power. Harder tasks and additional dimensions become necessary. Work previously completed through several professional exchanges may be abstracted into a fuller test. However, improving A while losing B is a repeated concern. New tasks must not obscure regression on earlier capabilities.
Asked how many questions a startup needs, she says she might start with 400, then immediately emphasizes sufficient discrimination over count. Four hundred is not an industry requirement. Selecting only the 400 most frequent inputs can also omit the long tail. A hypothetical head accounting for only 20% of traffic illustrates this; it is not observed traffic data.
Editorial interpretation: Beyond quantity, check whose tasks are covered, how many kinds exist and whether difficulty is uniform. Frequent requests test the main path, negative feedback reveals floor failures and the long tail prevents serving only easy cases. Version the set and retain regression tasks so score changes remain interpretable.
3. Without one correct answer, define boundaries and preferences
Episode reference: 11:58–16:45 and 25:56–28:43. A research report may need comprehensive, factual and structured output. Someone describing a breakup may not want a numbered list recommending exercise and meeting friends. Some want to tell the story, others want solutions, and some prefer a friend’s blunt response.
Dingding separates emotional change, intent understanding, conversational language and memory relevance, then assigns priorities. If prior conversation genuinely established a relationship’s condition, asking about the change may help. Without that history, inventing “last week you said everything was fine” to sound familiar is unacceptable. Different priorities in companionship do not remove factual boundaries.
Personalization also need not mean one model and test set for every person. Memory, preference understanding and instruction following can support different styles. Asked whether liking a “silly character” conflicts with a smarter model, Dingding argues that performing the role requires reliable instruction following. Character presentation differs from underlying capability.
Editorial interpretation: Separate boundaries from preferences. Invented shared experiences cross a boundary; reassurance before advice depends on context and user choice. Counseling-like responses in the conversation are product examples, not clinical standards. Her explanation of DeepSeek’s style as reflecting team taste also does not establish its internal training process.
4. Offline evaluation, live outcomes and AI judges must check one another
Episode reference: 19:41–21:51 and 29:35–30:02. Dingding imagines a possible Manus indicator: few steps followed by a result being downloaded or cited. It is a candidate idea, not a published official core metric. She stresses continued alignment between evaluation and user indicators. If offline scores improve without better experience, the evaluation needs investigation.
The same applies to automated judges. One model scoring another does not naturally equal user judgment. Compare it repeatedly with human assessments of the full experience, retaining disagreement cases. Agreement among similar models alone cannot establish improved quality.
She suggests maintaining questions unknown even to the algorithm team to reduce adaptation and score gaming. Editorial interpretation: Make capability goals and standards clear while retaining independent acceptance samples. Concealing every rule can leave developers unsure what to improve. Separate development and acceptance sets to guide improvement while checking generalization.
Organization affects this loop too. Repeated handoffs between labeling, strategy and application teams can lose the original problem. Small teams may shorten feedback, but expanding audiences still require additional standards. Early users’ tastes cannot remain the only criteria indefinitely.
5. Product managers still translate needs, with evaluation added to their deliverables
Episode reference: 30:24–35:20 and 35:54–40:59. Dingding calls the shared skill between conventional and AI product management translation: discovering problems and abstracting business situations into structure and design. Evaluation standards and observable signals now become additional outputs.
She suggests running a proposed task completely with models, comparing versions and APIs, building prototypes and learning what works and where it stops. Reading principles at an appropriate depth complements practice. “Full stack” increases responsibility for the complete process rather than eliminating engineering and design expertise. Recruitment can likewise ask which model someone used for which problem, why and what they built.
WeChat experience adds another layer: design product structure before isolated features. A request for multiple avatars may represent wanting to display more photographs, a need that Moments could address without complicating the avatar control. This is her interpretation of previous work rather than independent confirmation of historical decisions.
Running twenty A/B tests and picking a locally positive result does not replace judgment about the whole product. One department’s gains can damage overall experience. Tone and structure do not emerge automatically from a local metric. Editorial interpretation: Evaluation must locate errors and ask whether an improvement serves a coherent product objective. Otherwise the score itself becomes the substitute being optimized.
Start a test with one failure example
As an editorial exercise, select real negative feedback, retain necessary context and describe both unacceptable behavior and several acceptable responses. Compare it with a similar input from a person with different preferences.
| Task component | Example for a companion conversation |
|---|---|
| Real input and history | The user reports a breakup; include only confirmed relationship information |
| Boundaries | Invent no past events and offer no mechanical guarantees |
| Dimensions | Understand current intent, respond naturally and ground memory in evidence |
| Acceptable differences | Listening, direct advice or friendly bluntness according to preference |
| Validation | Where do people and automated judges disagree, and does subsequent user feedback align? |
Run both old and new versions, checking whether fixing this example harms formerly successful tasks. A small sample reveals problems without representing everyone. Expand through actual input distribution, negative feedback and domain judgment rather than an attractive total question count.
Listening guide and related reading
- 01:27: The distance between public benchmarks and real products.
- 07:07: Aligning data with capability goals.
- 17:55: Realism, difficulty and distinguishing power.
- 22:14: Four hundred questions and high-frequency coverage.
- 25:42: Evaluating companionship without one correct answer.
- 29:28: Independent acceptance samples and adaptation risk.
- 35:38: Structure, tone and local metrics.
Related themes: complete-process evaluation, preferences and boundaries, and product judgment. Compare Hidecloud’s practical experimentation and Pokee’s multi-step action approach for turning prototypes into verifiable delivery.
Source boundary: Based on the complete transcript, earlier reading note and official introduction, without sentence-by-sentence audio checks. Yao Shunyu’s article, internal model training and company metrics were not independently verified. Model comparisons, product history and evaluation recommendations retain contemporary attribution. Companion response examples are not psychological treatment advice.