Beyond an algorithm name

An RL training recipe includes how prompts are sampled, how responses are generated, how rewards are assigned, and how policy updates are scheduled. DAPO presents an open-source large-scale LLM RL system and training choices. Reading the complete recipe is more informative than attributing a result to one acronym.

Watch the distribution of useful experience

If sampled responses have nearly identical rewards, a batch may provide little relative learning signal. Track the proportion of informative samples, output lengths, truncation, invalid answers, and reward variance. A mean reward alone can hide changes in which examples actually drive updates.

Keep systems improvements visible

Faster rollouts can allow more experience or more experiments within a budget. Report both throughput and learning outcomes. A systems improvement may be valuable even if the policy objective is unchanged, but it should not be mistaken for an isolated algorithmic advantage.

Publish a reproducible training contract

Record model and data revisions, objective settings, sequence limits, hardware, rollout policy, reward code, and evaluation checkpoints. Match inference budgets for the final comparison, then test transfer beyond the reward’s own task family. Treat reproducibility as part of the evidence for the claimed improvement.

References

  1. DAPO: An Open-Source LLM Reinforcement Learning System at Scale ↗

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue et al. · 2025

    All 35 authors

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, Mingxuan Wang

  2. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models ↗

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi et al. · 2024

    All 11 authors

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, Daya Guo