中文精读
好评测要解释真实体验：从模型榜单走到用户任务
42章经单期精读 · Benchmark与Evaluation
嘉宾：丁丁，当期介绍为前微信、美团、Moonshot产品经理，曾负责Kimi App｜主持：曲凯
发布：2025年5月5日｜时长：41分12秒｜整理：2026年10月18日
官方单集与节目介绍
榜单上像专家，工作中为什么仍交付不好
Benchmark是一套测试题及相应标准，Evaluation是用标准分析、判断表现的过程。丁丁从这一区分开始：公开题目能反映部分模型能力，却不必然覆盖一个产品真正处理的输入与流程。
用户最终接触的，还有系统提示、搜索、知识库和接口。某一环节出错，即使基础模型得分很高，整段体验仍会失败。本期以姚顺雨关于AI“下半场”的文章为引子，以下分析的是访谈中的理解与实践，不替代对原文的独立阅读。所有模型偏好及行业判断保留在2025年时点。
1. 用户多，不等于得到更多有用的能力题
线索：05:50–09:27。 丁丁没有否定DAU，即日活跃用户数；真实用户能带来团队没想到的行业视角和输入方式。但她反对把规模直接等同模型进步。
一段轻娱乐输入，与专业用户经过几十轮讨论产出研究报告，暴露的任务要求不同。所谓高质量数据，需要与准备提升的能力对齐；不是用一个身份标签判断哪类人更好，而是看输入提供了什么问题、能否形成可用反馈。
她还指出最终体验受预训练、后训练和基础设施等多环节影响。不能仅因某家公司用户多，就断言它一定拥有领先智能。编辑推论：采集反馈前先写明准备改哪项能力，再判断数据是否补足缺口。用户数是规模信号，不是训练价值的换算公式。
2. 好题包含输入、输出和判断，也要有难度梯度
线索：10:08–11:58、17:21–19:41、22:14–24:22。 丁丁举一个深度搜索任务：找出一家公司的十年财报，再预测今年净利润变化。这里不仅需要用户输入和模型输出，还要明确怎样评价数据依据、完整性和推断。它是题目示例，并非本篇提供的金融预测。
随着模型进步，原来人人能做的题失去区分度，需要增加更难、不同维度的任务。专业用户原本用多轮才完成的工作，也可能被抽象成更完整的题目。但训练好A却丢B，是她反复提醒的回归风险；新题不能遮住旧能力退步。
主持人问创业公司要多少题，她说自己可能先做400道，随即强调足以衡量表现即可。400不是行业门槛。只取最高频400种输入也可能遗漏长尾；节目用头部只占20%流量的假设说明这一点，不是实际流量统计。
编辑推论：题量之外，更应检查覆盖谁、任务有几类、难度是否单一。高频场景用于检查主路径，负反馈帮助找底线失败，长尾则防止只服务最容易的一群。更新题集时保留版本与旧任务回归，才能解释分数变动。
3. 没有唯一答案，也可以定义底线与偏好
线索：11:58–16:45、25:56–28:43。 研究报告可能需要全面、真实的结构化回答；失恋倾诉却未必希望收到“一、出去运动；二、见朋友”的建议清单。有人希望先说故事，有人想解决问题，也有人喜欢朋友式的直话。
丁丁将这类评估拆成情绪变化、意图理解、口语化、记忆关联等维度，再决定优先级。若历史对话确实说过关系状态，回问变化可能有价值；若没有，不能为了显得熟悉而捏造“上周你说还好好的”。陪伴的重点不同，也不意味着事实性可以无限放弃。
个性化不必一人一套模型和题集。记忆、偏好理解与指令遵循，可以支撑不同表达。主持人问喜欢“笨角色”的用户是否与更聪明模型冲突，丁丁认为装笨恰恰需要可靠遵循设定：角色表现与底层能力不是一回事。
编辑推论：把底线和偏好分别写清。不可凭空编造共同经历，是底线；先安慰还是直接建议，则要看上下文与用户选择。节目中的心理咨询式举例是产品讨论，不是临床标准。关于DeepSeek文风反映团队审美的解释，也未验证其内部训练过程。
4. 离线评测、线上结果与AI裁判必须互相校验
线索：19:41–21:51、29:35–30:02。 丁丁为Manus设想一个指标：尽量少的步骤，且结果被下载或引用。这是候选思路，不能写成Manus已公布的核心指标。她强调评测与用户指标要持续对齐，否则离线变好而真实体验不变，评测就需要追查。
自动评审也一样。用一个模型给另一个模型打分，并不天然等于用户判断；需要与人对完整体验的评价反复校准，尤其保留分歧案例。不能仅用同类模型彼此赞同来宣布质量提升。
她建议留一套算法团队也不知道的测试题，降低训练适配和刷分风险。编辑推论：可以公开能力目标与标准，保留独立验收样本；若把所有规范都藏起来，团队反而不知道要优化什么。开发集和验收集分开，才能既指引改进又检查泛化。
组织流程同样影响质量。标注、策略、功能端之间多次传话，容易丢失用户问题；小团队能缩短反馈链，但用户扩大后仍要补不同人群的标准，不能永远只按早期用户偏好打分。
5. 产品经理仍在翻译需求，只是交付物增加了评估
线索：30:24–35:20、35:54–40:59。 丁丁把传统与AI产品经理共同的能力叫“翻译”：发现问题，把具体业务抽象成结构与设计；如今还要抽象为标准和观察信号。
她建议把想做的任务先用模型完整跑一遍，使用不同版本与接口，亲手搭原型，理解哪些能做、哪些卡住。阅读适合自己深度的原理资料，与实践互补。“全栈”在这里增加端到端责任，并不抹去工程、设计专业分工。招聘时也能追问：用什么模型解决什么问题，为什么，自己做过什么。
微信经验补上了评估之外的一层：先想产品结构，再做单点功能。多头像诉求背后可能是展示更多照片，需求也许可由朋友圈承接，无需在头像处堆功能。这个解释属于嘉宾对既往工作的理解，不是本篇对历史设计决策的独立确认。
跑二十个A/B实验挑指标正向，也不能替代对全局体验的判断。某部门局部增长可能损伤产品整体，调性与结构也不是由一个局部指标自动生成。编辑推论：评测既要能定位错误，也要回答改进是否服务同一个产品目标；否则高分本身也会成为被优化的替身。
用一个失败样例开始写题
以下为编辑练习。选择一条真实失败反馈，保留完整必要上下文，分别写出“不能这样做”和“允许有哪些好回答”，再找一条相似但偏好不同的输入对照。
| 题目部分 | 以陪伴倾诉为例 |
|---|---|
| 真实输入与历史 | 用户说分手；只提供已经确认的关系信息 |
| 底线 | 不编造旧事，不机械保证结果 |
| 关注维度 | 是否理解此刻意图，是否自然接话，记忆是否有依据 |
| 可接受差异 | 倾听、直接建议或朋友式回应，依据用户偏好选择 |
| 校验 | 人与自动裁判哪里分歧；用户后续反馈是否一致 |
再用旧版本和新版本一起跑，检查改好这道题以后，原来擅长的任务有没有变差。小样本能暴露问题，还不能代表全体用户；题集扩展应来自真实分布、负反馈与专家判断，而非追求一个好看的总题数。
回听导航与关联阅读
- 01:27：公开Benchmark与真实产品的距离。
- 07:07：数据怎样与能力目标对齐。
- 17:55：真实、梯度和区分度。
- 22:14：400题及高频输入的覆盖问题。
- 25:42：没有标准答案的陪伴评估。
- 29:28：独立验收样本与训练适配风险。
- 35:38：结构、调性和局部指标。
相关主题：端到端评测、偏好与底线、产品判断。可对照Hidecloud的动手实践与Pokee的多步行动路线，比较能力原型怎样变成可验证交付。
资料边界：依据完整转写、旧精读及官方介绍整理，未逐句复听。没有额外核验姚顺雨原文、模型内部训练或各公司指标。模型优劣、产品历史与评估建议保留当期归因；陪伴回应示例不作心理治疗建议。
English edition
Good Evaluation Must Explain Real Experience: From Model Rankings to User Tasks
42 Zhangjing episode analysis · Benchmark and Evaluation
Guest: 丁丁, introduced as a former product manager at WeChat, Meituan and Moonshot, previously responsible for Kimi App | Host: Qu Kai
Published: May 5, 2025 | Duration: 41 minutes 12 seconds | Prepared: October 18, 2026
Official episode and introduction
Why can a model look expert on a leaderboard and still fail at work?
A benchmark is a set of test tasks and corresponding standards. Evaluation is the process of using standards to analyze and judge performance. Dingding begins with this distinction: public tasks reflect some capabilities without necessarily covering a product’s actual inputs and processes.
Users also encounter system prompts, search, knowledge bases and interfaces. Failure in one component can spoil the experience despite a high-scoring foundation model. Yao Shunyu’s article about AI’s “second half” introduces the conversation. This analysis examines the interview’s interpretation and practices rather than replacing an independent reading of that article. Model preferences and industry assessments retain their May 2025 context.
1. More users do not automatically supply better capability tests
Episode reference: 05:50–09:27. Dingding does not dismiss DAU, or daily active users. Real users supply industry perspectives and input forms a team may never anticipate. She rejects equating scale directly with progress in intelligence.
A light entertainment request and a professional’s research report developed over dozens of exchanges expose different requirements. High-quality data must align with the capability being improved. This is not a ranking of people by identity: the question is what problem an input reveals and whether it produces useful feedback.
Pretraining, post-training and infrastructure also contribute to the final experience. A company’s larger user base therefore cannot independently establish superior intelligence. Editorial interpretation: Specify the capability to improve before collecting feedback, then ask whether the data fills its gaps. User count signals scale; it is not a conversion formula for training value.
2. Good tasks contain input, output and judgment, with varying difficulty
Episode reference: 10:08–11:58, 17:21–19:41 and 22:14–24:22. Dingding gives a deep-search task: retrieve ten years of a company’s financial reports and predict changes in this year’s net profit. Besides input and output, the task needs standards for evidence, completeness and inference. This is an evaluation example, not a financial forecast supplied here.
As models improve, tasks everyone solves lose distinguishing power. Harder tasks and additional dimensions become necessary. Work previously completed through several professional exchanges may be abstracted into a fuller test. However, improving A while losing B is a repeated concern. New tasks must not obscure regression on earlier capabilities.
Asked how many questions a startup needs, she says she might start with 400, then immediately emphasizes sufficient discrimination over count. Four hundred is not an industry requirement. Selecting only the 400 most frequent inputs can also omit the long tail. A hypothetical head accounting for only 20% of traffic illustrates this; it is not observed traffic data.
Editorial interpretation: Beyond quantity, check whose tasks are covered, how many kinds exist and whether difficulty is uniform. Frequent requests test the main path, negative feedback reveals floor failures and the long tail prevents serving only easy cases. Version the set and retain regression tasks so score changes remain interpretable.
3. Without one correct answer, define boundaries and preferences
Episode reference: 11:58–16:45 and 25:56–28:43. A research report may need comprehensive, factual and structured output. Someone describing a breakup may not want a numbered list recommending exercise and meeting friends. Some want to tell the story, others want solutions, and some prefer a friend’s blunt response.
Dingding separates emotional change, intent understanding, conversational language and memory relevance, then assigns priorities. If prior conversation genuinely established a relationship’s condition, asking about the change may help. Without that history, inventing “last week you said everything was fine” to sound familiar is unacceptable. Different priorities in companionship do not remove factual boundaries.
Personalization also need not mean one model and test set for every person. Memory, preference understanding and instruction following can support different styles. Asked whether liking a “silly character” conflicts with a smarter model, Dingding argues that performing the role requires reliable instruction following. Character presentation differs from underlying capability.
Editorial interpretation: Separate boundaries from preferences. Invented shared experiences cross a boundary; reassurance before advice depends on context and user choice. Counseling-like responses in the conversation are product examples, not clinical standards. Her explanation of DeepSeek’s style as reflecting team taste also does not establish its internal training process.
4. Offline evaluation, live outcomes and AI judges must check one another
Episode reference: 19:41–21:51 and 29:35–30:02. Dingding imagines a possible Manus indicator: few steps followed by a result being downloaded or cited. It is a candidate idea, not a published official core metric. She stresses continued alignment between evaluation and user indicators. If offline scores improve without better experience, the evaluation needs investigation.
The same applies to automated judges. One model scoring another does not naturally equal user judgment. Compare it repeatedly with human assessments of the full experience, retaining disagreement cases. Agreement among similar models alone cannot establish improved quality.
She suggests maintaining questions unknown even to the algorithm team to reduce adaptation and score gaming. Editorial interpretation: Make capability goals and standards clear while retaining independent acceptance samples. Concealing every rule can leave developers unsure what to improve. Separate development and acceptance sets to guide improvement while checking generalization.
Organization affects this loop too. Repeated handoffs between labeling, strategy and application teams can lose the original problem. Small teams may shorten feedback, but expanding audiences still require additional standards. Early users’ tastes cannot remain the only criteria indefinitely.
5. Product managers still translate needs, with evaluation added to their deliverables
Episode reference: 30:24–35:20 and 35:54–40:59. Dingding calls the shared skill between conventional and AI product management translation: discovering problems and abstracting business situations into structure and design. Evaluation standards and observable signals now become additional outputs.
She suggests running a proposed task completely with models, comparing versions and APIs, building prototypes and learning what works and where it stops. Reading principles at an appropriate depth complements practice. “Full stack” increases responsibility for the complete process rather than eliminating engineering and design expertise. Recruitment can likewise ask which model someone used for which problem, why and what they built.
WeChat experience adds another layer: design product structure before isolated features. A request for multiple avatars may represent wanting to display more photographs, a need that Moments could address without complicating the avatar control. This is her interpretation of previous work rather than independent confirmation of historical decisions.
Running twenty A/B tests and picking a locally positive result does not replace judgment about the whole product. One department’s gains can damage overall experience. Tone and structure do not emerge automatically from a local metric. Editorial interpretation: Evaluation must locate errors and ask whether an improvement serves a coherent product objective. Otherwise the score itself becomes the substitute being optimized.
Start a test with one failure example
As an editorial exercise, select real negative feedback, retain necessary context and describe both unacceptable behavior and several acceptable responses. Compare it with a similar input from a person with different preferences.
| Task component | Example for a companion conversation |
|---|---|
| Real input and history | The user reports a breakup; include only confirmed relationship information |
| Boundaries | Invent no past events and offer no mechanical guarantees |
| Dimensions | Understand current intent, respond naturally and ground memory in evidence |
| Acceptable differences | Listening, direct advice or friendly bluntness according to preference |
| Validation | Where do people and automated judges disagree, and does subsequent user feedback align? |
Run both old and new versions, checking whether fixing this example harms formerly successful tasks. A small sample reveals problems without representing everyone. Expand through actual input distribution, negative feedback and domain judgment rather than an attractive total question count.
Listening guide and related reading
- 01:27: The distance between public benchmarks and real products.
- 07:07: Aligning data with capability goals.
- 17:55: Realism, difficulty and distinguishing power.
- 22:14: Four hundred questions and high-frequency coverage.
- 25:42: Evaluating companionship without one correct answer.
- 29:28: Independent acceptance samples and adaptation risk.
- 35:38: Structure, tone and local metrics.
Related themes: complete-process evaluation, preferences and boundaries, and product judgment. Compare Hidecloud’s practical experimentation and Pokee’s multi-step action approach for turning prototypes into verifiable delivery.
Source boundary: Based on the complete transcript, earlier reading note and official introduction, without sentence-by-sentence audio checks. Yao Shunyu’s article, internal model training and company metrics were not independently verified. Model comparisons, product history and evaluation recommendations retain contemporary attribution. Companion response examples are not psychological treatment advice.