Mechanism

Rather than training only on clean (x,y⋆)(x, y^\star) gold trajectories, trains the model on multi-step trajectories (x,ydraft,ccritique,yrepaired⋆)(x, y_{\text{draft}}, c_{\text{critique}}, y_{\text{repaired}}^\star) containing realistic model failures, localized first-bad-step critiques, and verified repairs so the model learns how to detect and recover from its own errors in-context.

Practical Notes

Crucial for overcoming the "intrinsic self-correction blind spot" where models trained only on error-free SFT traces fail to revise mistaken steps.

References

  1. CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing ↗

    Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan et al. · 2023

    All 7 authors

    Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, Weizhu Chen

  2. Generating Sequences by Learning to Self-Correct ↗

    Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi et al. · 2022

    All 7 authors

    Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, Yejin Choi

  3. Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning ↗

    Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Heng Huang, Jiuxiang Gu et al. · 2023

    All 7 authors

    Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Heng Huang, Jiuxiang Gu, Tianyi Zhou