Mechanism

Observes that PPO/GRPO/DAPO zero out the gradient of any token whose importance ratio ρi,t(θ)\rho_{i,t}(\theta) exceeds 1+ϵ1+\epsilon. During long-CoT RL, rare "fork/reflection" tokens (Wait, However, Let's re-evaluate, Alternatively) have low initial probability under πθold\pi_{\theta_{\text{old}}} and thus experience large ratio jumps on the first mini-batch update—meaning standard PPO/GRPO clips and drops their gradients after a single step! CISPO instead optimizes an unclipped REINFORCE loss weighted by a detached, clipped importance weight: LCISPO(θ)=−Ei,t[sg(clip(ρi,t(θ),1−ϵlow,1+ϵhigh))A^i,tlog⁡πθ(yi,t∣x,yi,<t)]\mathcal{L}_{\text{CISPO}}(\theta) = -\mathbb{E}_{i,t}\left[\text{sg}\left(\text{clip}(\rho_{i,t}(\theta), 1-\epsilon_{\text{low}}, 1+\epsilon_{\text{high}})\right) \hat{A}_{i,t} \log \pi_\theta(y_{i,t} \mid x, y_{i,<t})\right] so every token continues to receive gradient updates bounded in magnitude.

Practical Notes

Achieves 2× faster RL convergence than DAPO on AIME while preserving reflective reasoning tokens.

References

  1. MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention ↗

    MiniMax, :, Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang et al. · 2025

    All 128 authors

    MiniMax, :, Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, Chengjun Xiao, Chengyu Du, Chi Zhang, Chu Qiao, Chunhao Zhang, Chunhui Du, Congchao Guo, Da Chen, Deming Ding, Dianjun Sun, Dong Li, Enwei Jiao, Haigang Zhou, Haimo Zhang, Han Ding, Haohai Sun, Haoyu Feng, Huaiguang Cai, Haichao Zhu, Jian Sun, Jiaqi Zhuang, Jiaren Cai, Jiayuan Song, Jin Zhu, Jingyang Li, Jinhao Tian, Jinli Liu, Junhao Xu, Junjie Yan, Junteng Liu, Junxian He, Kaiyi Feng, Ke Yang, Kecheng Xiao, Le Han, Leyang Wang, Lianfei Yu, Liheng Feng, Lin Li, Lin Zheng, Linge Du, Lingyu Yang, Lunbin Zeng, Minghui Yu, Mingliang Tao, Mingyuan Chi, Mozhi Zhang, Mujie Lin, Nan Hu, Nongyu Di, Peng Gao, Pengfei Li, Pengyu Zhao, Qibing Ren, Qidi Xu, Qile Li, Qin Wang, Rong Tian, Ruitao Leng, Shaoxiang Chen, Shaoyu Chen, Shengmin Shi, Shitong Weng, Shuchang Guan, Shuqi Yu, Sichen Li, Songquan Zhu, Tengfei Li, Tianchi Cai, Tianrun Liang, Weiyu Cheng, Weize Kong, Wenkai Li, Xiancai Chen, Xiangjun Song, Xiao Luo, Xiao Su, Xiaobo Li, Xiaodong Han, Xinzhu Hou, Xuan Lu, Xun Zou, Xuyang Shen, Yan Gong, Yan Ma, Yang Wang, Yiqi Shi, Yiran Zhong, Yonghong Duan, Yongxiang Fu, Yongyi Hu, Yu Gao, Yuanxiang Fan, Yufeng Yang, Yuhao Li, Yulin Hu, Yunan Huang, Yunji Li, Yunzhi Xu, Yuxin Mao, Yuxuan Shi, Yuze Wenren, Zehan Li, Zelin Li, Zhanxu Tian, Zhengmao Zhu, Zhenhua Fan, Zhenzhen Wu, Zhichao Xu, Zhihang Yu, Zhiheng Lyu, Zhuo Jiang, Zibo Gao, Zijia Wu, Zijian Song, Zijun Sun