Reiner Pope
lecture 2026-05-04 · Dwarkesh Podcast
Reiner Pope: The Math Behind How LLMs Are Trained and Served
🔑 Key Insights
"If you do not batch together many users, the cost and the economics you get can be a thousand times worse than if you do batch many users together."
💡 不 batch 用户的成本比 batch 用户高 1000 倍
"FLOPs over memory bandwidth is a dimensionless constant of around 300 on most GPUs — from A100 to H100 to B100 it has remained remarkably stable."
💡 FLOPs/内存带宽 ≈ 300，A100→H100→B100 变化不大
"Because of RL, models may be 100x over-trained beyond Chinchilla-optimal."
💡 RL 信号稀缺，模型可能超出 Chinchilla 最优点 100 倍
"DeepSeek V3 activates 32 out of 256 experts. This gives a sparsity of 8, which means you need a batch size of roughly 300 × 8 = 2,400."
💡 DeepSeek V3 最优 batch size ≈ 2400
Summary
- Roofline Model：推理时间 = max(计算时间, 内存时间)，batch size 决定权衡
- 两个核心公式：$t_{compute} = B·N_active/FLOPs$, $t_{mem} = N_total/B_w + B·L·s/B_w$
- 延迟 vs 成本：延迟有下限，成本随 batch 从双曲趋近常数
- 最优 batch size：$B ≈ 300 × sparsity$，与模型规模无关
- HBM 全读一遍 ≈ 15-20ms（A100→H100→B100→Rubin 变化不大）
- Pipeline bubble：层间等待时间在大 batch 下严重拖累利用率
- 从 API 定价可反推 KV cache 内存成本结构