Mechanism

Augments standard teacher outputs with rich system prompts ("Explain like I'm five", "Think step-by-step and justify every assumption") to elicit detailed intermediate reasoning explanations from a frontier teacher, then trains the student on the resulting explanation traces (or erases the system prompt in Cautious Reasoning / Orca 2 so the student chooses the best reasoning strategy autonomously).

Practical Notes

Crucial precursor to modern reasoning distillation; teaches small models how to structure reasoning rather than merely imitating terse answers.

References

  1. Orca: Progressive Learning from Complex Explanation Traces of GPT-4 ↗

    Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, Ahmed Awadallah · 2023

  2. Orca 2: Teaching Small Language Models How to Reason ↗

    Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agarwal et al. · 2023

    All 15 authors

    Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agarwal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, Hamid Palangi, Guoqing Zheng, Corby Rosset, Hamed Khanpour, Ahmed Awadallah

  3. AgentInstruct: Toward Generative Teaching with Agentic Flows ↗

    Arindam Mitra, Luciano Del Corro, Guoqing Zheng, Shweti Mahajan, Dany Rouhana, Andres Codas et al. · 2024

    All 14 authors

    Arindam Mitra, Luciano Del Corro, Guoqing Zheng, Shweti Mahajan, Dany Rouhana, Andres Codas, Yadong Lu, Wei-ge Chen, Olga Vrousgos, Corby Rosset, Fillipe Silva, Hamed Khanpour, Yash Lara, Ahmed Awadallah