中文精读
合成数据不是多生成几条：孟繁青谈环境、评测与自我改进
42章经单期精读 · 数据质量与研究闭环
嘉宾：孟繁青，Evolvent AI联合创始人、前Kimi RL实习生（当期介绍）｜主持：曲凯
发布：2026年8月8日｜时长：59分29秒｜整理：2026年10月18日
官方单集与节目介绍
训练入口简单了，判断数据是否有用仍然很难
孟繁青在模型厂里使用成熟训练平台，创业后却要从零搭平台、找与基础模型互补的切口。服务化可以让提交数据和启动训练变简单，不能自动让数据有价值。
本期最具体的主线，是把任务、环境、评分、轨迹、训练和再评测连成闭环。官方称他为NUS PhD，转写自述为博士生，因此本篇不推断学位完成状态。公司排名、模型路线与数据商业机会均保留2026年8月观点，不据单项榜单建立全面领先结论。
1. Agent评测从考题进入环境，评分本身也要验收
线索：01:38–07:09。 过去的静态评测像考生做题，新评测还要看能不能动手。Agent指能使用工具推进任务的模型系统：白领工作不只靠回答，还要操作飞书、Notion等软件，模拟环境试图把这些环节带进测试。
先要验证环境：相同输入下，模拟软件与真实软件是否有一致行为。一次相同输出只能检查一个样例，不能证明整个环境全对。再要定义多维得分点，例如改到哪条记录、有没有保留约束，而不只是模型说“完成了”。
孟繁青说评分设计仍依赖人和Agent共同检查。编辑推论：环境、任务和评分都可能有错；被测者通过测试，可能只是找到评分漏洞。应留真实任务对照、异常和人工抽审，避免模型自评形成封闭的成功故事。
2. 好数据要经过训练效果，也要经过退化检查
线索：07:19–12:41。 模型厂内部已把训练做成服务，研究员能把更多精力放在数据上。创业公司和平台建设者仍需处理环境、资源与运行问题，不能把内部便利误解为任何人训练都没有门槛。
数据可先检查答案正确、轨迹是否作弊、难度是否适中。太简单没有新增信号，太难可能学不到；这些都只是初筛。最终还得训练，检查目标任务是否提高，并看写作等旧能力有没有受损。
自动修回答格式，是他观察到Agent能完成的机械改进；提出高价值的数据洞察仍需要研究经验。编辑推论：数据条数、格式整齐与有效学习信号不是一回事。采购或自建都应保留基线、独立评测和旧任务回归，防止只针对一个分数制造进步。
3. RSI要先说清改了哪一层
线索：22:08–25:17、29:51–35:10。 RSI在本期指递归自我改进，但词的范围并未完全统一。曲凯拆成修订最终结果、修改运行框架、训练更强模型三层；孟繁青认为可放在共同形式里：环境、目标、操作、反馈、继续改进。模型从零训练并超越自己，是更强的目标，不能与自动修改一份输出混为一谈。
Harness是组织工具与上下文的运行框架。孟繁青认为它会与模型配套、可能更简洁，未必消失。他把部分自动研究比作更聪明的深度优先搜索：写GPU kernel，评分，坏了回溯，再选择可能更好的路径。Kernel是GPU执行计算的一段程序。
世界模型、价值判断和长上下文被他用来解释搜索效率，但“没有独立RSI基模”是个人论点，不是研究定论。未来模型公司只需十几人的推演也未验证。
编辑推论：自我改进必须说明目标指标、允许改动、反馈来源和计算预算。连续运行、自己改代码与得到更强模型，是三种不同验收；若评分有偏，循环只可能更熟练地优化偏差。
4. 强模型建设、小模型运行，是有条件的经济选择
线索：13:12–21:14、25:46–29:01。 对国内外模型差异，孟繁青提出一个非共识解释：资源限制倒逼预训练架构创新，后训练数据积累的时间差反而难追。他明确不知道是否行业共识；具体公司内部原因也有不了解之处。
榜单上交替领先，不等于真实体验相同，更不等于一家公司全面超越。他还强调组织设计影响整体迭代速度，不能把竞争简化成只有卡或只有架构。
在垂直任务中，他设想用强模型构建流程、指导较小开源模型改进，日常再用小模型运行，以求更快更省；隐私需求也可能影响选择。但主持人预测价格趋近免费，他保留不同意见：定价还受竞争和商业策略影响。
编辑推论：应在相同任务、质量、并发与维护范围比较总成本。本地模型未必更省，强模型生成过框架，也不能证明小模型稳定运行。建设成本和持续服务成本需要分开算。
5. 外采数据可以补分布，商业窗口也会移动
线索：35:28–42:29、46:25–48:14。 普通标注、专家出题、合成环境轨迹与真人工作轨迹并存。模型越强，专家越需要提供没见过的新问题；真人轨迹也有噪声和使用边界，个人聊天记录不自动成为可出售资产。
模型厂即使能造数据，也可能外采来补内部经验偏差。但评测和任务收敛后，有能力的客户又可能转自建。孟繁青预期，训练另一个模型并改善它的长轨迹是下一波机会；十几小时或一天产生一条轨迹，只说明生产复杂，不保证高售价或一定有用。
蒸馏在这里涉及用外部模型产生训练信号，合成数据的价值还包括环境、问题与评分设计。他认为它是加速手段，不是国内追赶决定因素，这是技术与竞争观点，不能顺势推导任何模型输出都允许用于训练。
编辑推论：来源、许可、隐私和合同范围要先明确，再讨论质量。所谓“合成”不自动消除权利问题；材料价格单位不清处也不能自行补成报价。
6. 长任务让数据、算法与基建互相牵动
线索：43:11–46:01、48:49–58:47。 他用一百条任务说明尾部问题：九十九条一分钟完成，一条需要一百分钟，整批却一直等待，部分卡闲置。更难、更长的数据会推动调度、并行乃至算法改变。
“算法就是数据，数据就是Infra”是强调相互制约的修辞，不是三者真的等同。数据质量提升若令吞吐大降，仍要看单位时间得到多少有效改进，不能只看最长轨迹有多惊人。
代码、科学与自动研究是当期探索方向，节目没有独立核验宣传中的科学成果。孟繁青长期更看好初创模型团队，认为懂相近技术后，组织如何跑快很重要；招聘重工程动手，是其团队阶段的偏好，不是普遍否定学术能力。
编辑推论：同时记录有效结果与等待瓶颈。自动化省下的时间，若被长尾失败、人工修数据和环境维护吃掉，就还没形成整体效率。
练习：让一条轨迹经过四道检查
选择一个可回读小任务，依次检查环境是否对应真实操作、评分是否对应目标、轨迹是否正确且无作弊、改进后旧任务是否退化。再记录生产和验证总耗时。
若只改提示或知识，也保留同样的旧样例比较。不要先追更多数据，先找哪种失败需要新信号。这个练习是编辑整理，不是Evolvent内部流程，也不承诺可以据此实现RSI。
回听导航与关联阅读
- 04:37：工作环境怎样进入评测。
- 10:35：数据质量的最后检验。
- 12:48：预训练优势的非共识观点。
- 21:50：RSI究竟改哪一层。
- 39:07：自建与外采的关系。
- 43:00：长任务如何牵动基建。
- 46:11：合成数据与蒸馏。
相关主题：Agent评测、数据采购、研究吞吐。可对照吴翼谈强化学习系统和朱亦博谈共同优化，把能力、成本与反馈放在同一任务里检查。
资料边界：完整阅读所供转写、旧精读和官方介绍，未逐句复听、复现模型或审查数据合同。专名按官方确认，含混版本、售价与科研成果不猜补；模型排名、蒸馏贡献、RSI路线及组织预测保留2026年8月观点，不构成独立技术验收。
English edition
Synthetic Data Means More Than Generating More: Meng on Environments, Evaluation and Self-Improvement
42 Zhangjing episode analysis · Data quality and research feedback
Guest: 孟繁青, cofounder of Evolvent AI and former Kimi RL intern, as introduced at release | Host: Qu Kai
Published: August 8, 2026 | Duration: 59 minutes 29 seconds | Prepared: October 18, 2026
Official episode and introduction
Easier training does not make data quality easy to judge
Meng used a mature training platform inside a model company. Starting a business meant building platforms and finding a complementary opportunity. Turning training into a service simplifies data submission and execution without automatically making the data valuable.
The central mechanism connects task, environment, scoring, trajectory, training and reevaluation. The introduction says NUS PhD while Meng calls himself a doctoral student, so completion of the degree is not inferred. Rankings, research routes and commercial opportunities retain their August 2026 context. One benchmark cannot establish comprehensive leadership.
1. Agent evaluation enters environments, and scoring also needs acceptance
Episode reference: 01:38–07:09. Static tests resemble examinations; newer evaluations also test action. Agents are model systems using tools to advance tasks. Office work requires operating software such as Feishu and Notion, beyond producing answers. Simulated environments bring these interactions into tests.
First validate the environment: does the same input produce behavior consistent with the real software? One matching output checks one case rather than proving the entire environment correct. Then define multiple scoring points, such as which record changes and which constraints remain, rather than accepting a model’s statement that it finished.
Meng says people and agents still inspect scoring together. Editorial interpretation: Environments, tasks and graders can each be wrong. Passing may reflect exploiting a grader. Retain comparisons with real tasks, exceptions and human samples to prevent self-evaluation becoming a closed success narrative.
2. Useful data need training results and regression checks
Episode reference: 07:19–12:41. Training services let researchers concentrate more on data. Startups and platform builders still manage environments, resources and failures; internal convenience does not mean training has no entry cost everywhere.
Initial checks include answer correctness, cheating trajectories and suitable difficulty. Easy examples may add little signal; excessively difficult ones may teach little. These are filters. Training must then test target improvement while checking whether previous abilities, such as writing, deteriorate.
Meng observes agents mechanically repairing response formats, while valuable data insights still require research experience. Editorial interpretation: Counts, tidy formatting and effective learning signals differ. Buying or building data requires baselines, independent evaluation and old-task regression checks, beyond improvement on one score.
3. RSI needs a stated level of change
Episode reference: 22:08–25:17 and 29:51–35:10. Recursive self-improvement, or RSI, has no fully shared scope in the discussion. Qu separates revising an output, modifying the execution framework and training a stronger model. Meng places them within an environment-goal-action-feedback loop. Training from scratch and surpassing oneself is a stronger objective than automatically revising a document.
A harness organizes tools and context. Meng expects it to accompany a model and potentially become simpler, rather than disappear. Some automated research resembles smarter depth-first search: write a GPU kernel, score it, backtrack after poor results and choose a promising route. A kernel is a program executing computation on a GPU.
World models, value judgments and longer context explain his view of search efficiency. His claim that there is no separate RSI foundation model is a personal argument, not research consensus. Future model companies needing only roughly ten to twenty people is also an unverified scenario.
Editorial interpretation: State the target metric, allowed changes, feedback source and compute budget. Running continuously, editing its own code and obtaining a stronger model are different acceptance claims. A biased grader may simply produce increasingly effective optimization of that bias.
4. Strong models for construction and small models for operation require an economic test
Episode reference: 13:12–21:14 and 25:46–29:01. Meng offers a non-consensus account of Chinese models: resource limits encouraged pretraining architecture innovation, while accumulated post-training data creates a harder time gap. He explicitly does not know whether this is industry consensus and lacks knowledge of some internal company causes.
Alternating benchmark leaders do not imply equivalent experience or comprehensive superiority. Organization also affects iteration; competition cannot be reduced to cards or architecture alone.
For specialized tasks, he imagines strong models building workflows and improving smaller open models, with the latter operating daily at lower cost and latency. Privacy may influence the choice. Qu forecasts prices approaching free; Meng disagrees that this is inevitable because competition and commercial strategy also affect pricing.
Editorial interpretation: Compare total cost under equal task, quality, concurrency and maintenance conditions. Local models are not necessarily cheaper. A strong model generating a harness does not establish stable operation by a smaller one. Separate construction from continuing service costs.
5. External data can broaden distributions while commercial windows move
Episode reference: 35:28–42:29 and 46:25–48:14. Ordinary labeling, expert questions, synthetic environment trajectories and human work traces coexist. Stronger models demand more novel expert questions. Human traces still contain noise and usage restrictions; personal conversations do not automatically become saleable assets.
Model companies may buy externally to supplement internal biases even while generating their own data. As evaluations and tasks converge, capable customers may build internally. Meng sees trajectories that train and improve another model as a possible next opportunity. Taking many hours or a day to produce one trajectory establishes complexity rather than a guaranteed price or usefulness.
Distillation here involves external models providing training signals. Synthetic data’s value also includes environment, questions and scoring. Meng calls distillation an accelerator rather than the decisive factor in Chinese progress. That competitive and technical position grants no permission to train on arbitrary model outputs.
Editorial interpretation: Establish provenance, rights, privacy and contractual scope before quality. Synthetic material does not automatically remove rights questions, and unclear price units should not be reconstructed into quotations.
6. Long tasks link data, algorithms and infrastructure
Episode reference: 43:11–46:01 and 48:49–58:47. Meng illustrates the tail problem with a hundred tasks: ninety-nine finish in a minute, one takes a hundred minutes and the batch waits while resources sit idle. Longer, harder data can require scheduling, parallelization and algorithm changes.
“Algorithms are data; data are infrastructure” emphasizes interdependence rather than literal identity. If improved data reduce throughput drastically, examine effective improvement per unit of time rather than the impressiveness of one long trajectory.
Coding, science and automated research are contemporary directions, without independent checks of promoted scientific results. Meng favors startup model teams over the long term, emphasizing organizational speed when technologies are widely understood. Engineering-focused hiring reflects his team’s stage rather than a general rejection of academic ability.
Editorial interpretation: Record useful results and waiting bottlenecks together. Time saved through automation may be consumed by long-tail failures, manual repairs and environment maintenance, leaving no overall efficiency gain yet.
Exercise: send one trajectory through four checks
Choose a small task with readable results. Check whether its environment matches real operation, scoring matches the goal, the trajectory is correct without cheating and previous tasks regress after improvement. Record production and validation time together.
Retain old examples even when changing only prompts or knowledge. Before generating more data, identify which failure needs a new signal. This is editorial synthesis rather than Evolvent’s internal procedure or a promise of achieving RSI.
Listening guide and related reading
- 04:37: Work environments entering evaluation.
- 10:35: The final test of data quality.
- 12:48: A non-consensus pretraining thesis.
- 21:50: Which level RSI changes.
- 39:07: Internal versus external data.
- 43:00: Long tasks and infrastructure.
- 46:11: Synthetic data and distillation.
Related themes: agent evaluation, data sourcing and research throughput. Compare Wu Yi on RL systems and Zhu Yibo on joint optimization, testing capability, cost and feedback within one task.
Source boundary: The complete supplied transcript, earlier note and official introduction were read without sentence-by-sentence audio, model replication or contract inspection. Confirmed official names are used; uncertain versions, prices and scientific results are not reconstructed. Rankings, distillation’s contribution, RSI routes and organizational forecasts remain August 2026 views rather than independent technical acceptance.