Guest: Zhu Yibo, cofounder of 阶跃星辰, former head of AI infrastructure at ByteDance, previously at Microsoft and Google, as introduced at release | Host: Qu Kai
Published: 2025-08-02 | Duration: 58:08 | Prepared: 2026-10-18
Official episode
Why can the same model become more expensive or slower under another system—and why can infrastructure limit its eventual capability? Zhu begins with a neglected question: what is the team optimizing? Maximizing hardware utilization, maximizing model scores, and minimizing the user’s wait can be three different calculations. AI infrastructure, the systems supporting reliable and efficient model operation, must weigh hardware, models, and tasks together.
This is a practitioner’s view from August 2025. Competitive judgments, technical directions, and cost forecasts retain that date. Depending on infrastructure does not imply that every application company should build its own.
1. Infrastructure becomes central when computation is an initial constraint
Listen: 01:07–07:11; 08:40–11:20. Zhu reduces the underlying problems to computation, communication, and storage. GPUs need networks, switches, storage, operations, resource scheduling, and training and inference frameworks. His comparison with cloud service layers is an explanatory framework, not the only possible industry taxonomy.
A small internet business can often begin with general services and hire specialists when its scale demands more. A foundation model company faces large data and computation requirements from the outset, making infrastructure part of what it can build. Zhu compares this with early search engines processing huge collections of webpages: background capabilities can become the starting point of product competition.
Building internally still requires a scale calculation. He imagines a monthly compute bill of RMB100 million, where a 10% improvement represents roughly RMB10 million in value. This is a hypothetical example, not disclosure of his company’s spending. A small application’s usage may be too low for the same percentage gain to cover a dedicated team. Cloud and model services provide a baseline. The threshold depends on actual costs, control needs, and expected returns, not sector excitement.
2. Waiting time, training utilization, and inference cost are different objectives
Listen: 08:40–09:43; 16:15–22:06. In a chat product, time to first token determines when the user sees a response. Subsequent speed and consistency determine whether the interaction feels smooth. A system can have high aggregate throughput while individual users wait a long time. A single tokens-per-second figure cannot establish the user experience.
Training teams often monitor MFU, a measure of actual model computation relative to theoretical hardware capability. Zhu argues that an attractive local metric does not guarantee a good overall choice. His interpretation of DeepSeek is that architecture oriented toward inference cost became advantageous when reinforcement learning required repeated generation and scoring. This is his explanation of an industry trajectory, not independent verification of internal decisions or a complete account of the company’s success.
He separates prefill, which processes input, from decoding, which produces output step by step. Decoding speed and cost are his priority at the time because they affect both online serving and the rate at which training receives feedback. That priority depends on the date and workload. Long documents or multimodal understanding may expose different bottlenecks. An editorial application is to identify the waiting that users most dislike before choosing the metric—and to evaluate whole-task success alongside it.
3. Handing a finished architecture to systems engineers can come too late
Listen: 22:06–28:41; 47:54–51:51. Zhu describes a triangle of algorithms, systems, and data. One architectural choice might offer a small quality improvement at a large performance cost, or the reverse. If algorithm researchers finish the structure before infrastructure is invited to accelerate it, some costs are already embedded.
He discusses an unnamed model that looks small in parameter count or active parameters and scores well, yet may run less efficiently than a larger model. A parameter-versus-quality chart is not an actual-cost-versus-quality chart. Communication, memory access, and implementation affect the bill. This remains Zhu’s example; the article neither identifies the company nor presents it as a public benchmark replication.
Mixture of experts, or MOE, illustrates cost constraints influencing architecture. Activating only some experts can retain a large capacity while reducing the work per use. Zhu recalls exploring MOE because business teams found a model too expensive, emphasizing that systems engineers can participate in design. This does not prove every early MOE adopter has a better organization, or that the architecture is inherently cheaper. It must be tested on actual hardware and workloads.
The potential advantage of a small team is shared discussion of tradeoffs, not small headcount itself. Joint responsibility requires aligned objectives and decision authority. Asking engineers to show more initiative cannot by itself remove organizational resistance.
4. Durable third-party value requires influence upstream or downstream
Listen: 11:20–16:15; 28:41–39:28. If everyone can access the same models and hardware, a middle layer offering temporary acceleration can be caught and pushed into price competition. Zhu advises practitioners to understand models or hardware deeply enough to join their design. He compares model services with platforms whose exclusive content gives users a reason to stay beyond generic aggregation.
Joint design also carries technical direction risk. The issue is participation in changing the architecture, rather than passively betting on a partner. Popular open models encourage adaptation while concentrating engineering investment around existing designs. A new architecture can force some of that work to be redone. Engineering problems with a visible route to completion and paradigm problems without one need different expectations.
At release, 阶跃星辰 sought to adapt models to Chinese chips and build mutual distribution benefits. Free commercial use was a company statement in the conversation. Actual use requires checking the corresponding version and license; a podcast is not an authorization document. Zhu also does not assert that every company must eventually build chips. His point is that a competitor integrating models and hardware particularly well could pressure others.
5. Visual reasoning matters when text loses spatial and operational relations
Listen: 39:28–43:48. Recognizing objects in a photo differs from deciding how to retrieve an obscured item from a cabinet. The latter requires understanding obstruction, location, and sequence: what moves first, and what can then be taken? Compressing the scene into text may discard those relationships, motivating direct reasoning from visual information.
Zhu’s robot “cerebrum and cerebellum” analogy separates task planning from balance and movement control. He also mentions locating a photograph and solving a photographed problem. Producing steps does not establish safe robot execution; location judgments introduce error and privacy concerns. These safety and privacy limits are editorial additions to prevent demonstrations from becoming claims of field reliability.
Models and systems therefore need joint assessment. A more direct input path may preserve information, while its actual latency, cost, and task accuracy still require measurement. Statements about multimodal leadership and future reductions in generation cost remain the company’s contemporary assessment and predictions.
6. Methods that benefit from more computation still need present delivery value
Listen: 52:58–57:58. Zhu invokes The Bitter Lesson: methods that exploit computation effectively may outperform short-lived tricks over time. For infrastructure, the question is whether a system and architecture keep benefiting as scale grows, rather than extracting a one-time advantage from a fixed setup. This does not eliminate engineering detail, and the conversation’s hardware history should not be treated as a rigorous physical law.
He sees both interdependence and pressure between agents and models. Application teams use engineering to make models “stand on tiptoe” to complete tasks; model providers may later absorb the capability natively. Yet providers may not discover those scenarios first. Zhu explicitly says he cannot see the final ecosystem. This cannot be reduced to a prediction that all applications disappear. An editorial question is whether today’s engineering advantage builds user relationships, reliable workflows, or other value that can accumulate.
Exercise: draw two curves for the same task
Choose a frequent task and compare two model-and-system options. One curve records model quality or scores. The other records cost per successful task under identical task, concurrency, and hardware conditions, with time to first token, total completion time, and failure recovery annotated. Separate input processing from output so an average speed does not conceal the bottleneck.
Ask product, algorithm, and engineering colleagues to write one shared tradeoff: which user benefit justifies which cost? If utilization improves without improving the target result, explain where its value comes from. Finally estimate savings using actual monthly usage before choosing procurement, internal development, or further observation. This is an editorial exercise, not the company’s internal acceptance procedure.
Sources and listening guide
Prepared from the complete transcript and official description and chapters, without replaying audio or testing models. Many transcript answers are mislabeled as the host; the synthesis follows question-and-answer meaning and official guest information. Uncertain hardware names and parameter counts are not reconstructed. Cost examples, company judgments, and commercial statements retain their source boundaries.
- 01:07: Infrastructure layers and shared objectives.
- 10:09: The scale calculation for internal investment.
- 17:45: How optimization objectives change outcomes.
- 22:06: Joint algorithm, system, and data tradeoffs.
- 36:35: Coordinating model and chip adaptation.
- 47:54: Parameters versus actual operating cost.
- 55:43: Long-term computation and engineering value.
Related themes: AI infrastructure, model evaluation, and organizational coordination. Continue with Wu Yi on reinforcement learning and Ji Yu on Nvidia for connections between training throughput and hardware-software ecosystems.