The wider training picture.

1. Executive Summary: The Five Eras of LLM Post-Training (2020–2026)

Post-training has evolved from a lightweight formatting wrapper on top of pretraining into a compute-intensive, multi-stage learning loop that often accounts for the majority of frontier capability gains in mathematics, software engineering, scientific discovery, and long-horizon agency.

1. Era 1 — Multi-Task Instruction Tuning & Classic PPO RLHF (2020–2022):
   • Core methods: Multi-task SFT (FLAN, T0), Human Demonstration SFT + Bradley-Terry Outcome Reward Model (ORM) + PPO with KL penalty and pretraining mix (PPO-ptx in InstructGPT, ChatGPT, Claude 1, Llama 2).
   • Bottleneck: High memory/compute overhead of maintaining four models in memory (Actor, Critic, Reference, Reward Model), unstable value-function estimation, and susceptibility to reward hacking (length bias, sycophancy).
2. Era 2 — Offline Direct Preference Optimization & Synthetic Instruction Scaling (2023–Early 2024):
   • Core methods: Direct Preference Optimization (DPO) reparameterized the RLHF objective to eliminate the explicit reward model and online rollouts; followed by IPO, KTO, ORPO, SimPO, and synthetic data engines (Self-Instruct, Evol-Instruct, Orca, UltraFeedback).
   • Bottleneck: Offline preference datasets quickly go off-policy as the model updates, causing likelihood collapse on chosen responses and ceiling effects on complex multi-step reasoning.
3. Era 3 — Iterative Online Alignment, Rejection Sampling & Process Supervision (2023–2024):
   • Core methods: Multi-round Rejection Sampling + DPO (Llama 3 / 3.1), On-Policy Knowledge Distillation (Gemma 2), Process Reward Models (PRM800K, Math-Shepherd, OmegaPRM), and STaR / $\text{ReST}^{EM}$ loops.
   • Breakthrough: Closing the loop between policy sampling, verification/judging, and policy updating prevented off-policy drift.
4. Era 4 — RL with Verifiable Rewards (RLVR) & Critic-Free Group Policy Optimization (2024–2025):
   • Core methods: GRPO (DeepSeekMath, DeepSeek-R1), RLVR (Tülu 3), Online Mirror Descent (Kimi k1.5), and Long-CoT Cold-Start + Multi-Stage RL + Distillation (OpenAI o1/o3, DeepSeek-R1, QwQ, Qwen3).
   • Breakthrough: Replacing learned neural reward models with deterministic, executable verifiers (exact math checkers, unit tests, Lean/Isabelle proof kernels, sandboxed compilers) eliminated neural reward hacking and induced emergent test-time behaviors (self-verification, backtracking, subgoal decomposition).
5. Era 5 — De-biased, MoE-Stabilized, Asynchronous & Stateful Agentic RL (2025–2026):
   • Core methods: DAPO (ByteDance Seed), VAPO (value-augmented long-CoT RL), Dr.GRPO (removing length and standard-deviation biases), GSPO (Qwen sequence-level importance ratios for MoE stability), CISPO (MiniMax-M1 clipped importance weights preserving reflective tokens), REINFORCE++, SAO / V-SAO (GLM-5.2 & Ulam single-rollout asynchronous optimization), MuonClip (Kimi K2/K3 QK-Clip optimizer stability), and Multi-Turn Stateful Environment RL across Docker terminals, browsers, and MCP tool ecosystems (GPT-6 Astra, Claude Opus 5.5, Gemini 4 Argon, Kimi K3.1 Pro, DeepSeek V4.1, Qwen3.5/3.8).

---

2. Training recipes disclosed in technical reports

Model or experiment	What the report documents	Source
InstructGPT	Demonstrations, a preference reward model and PPO with a pretraining-data mixture.	Long Ouyang, Jeff Wu, Xu Jiang et al. (2022), Training language models to follow instructions with human feedback
DeepSeek-R1	R1-Zero experiments and a staged pipeline with cold-start data, reasoning RL and rejection-sampled supervision.	DeepSeek-AI, Daya Guo, Dejian Yang et al. (2025), DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
Qwen3	Reasoning-oriented post-training, thinking and non-thinking behavior, and distillation.	An Yang, Anfeng Li, Baosong Yang et al. (2025), Qwen3 Technical Report
Llama 3	Iterative supervised and preference training, with data filtering and domain-specific evaluation.	Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri et al. (2024), The Llama 3 Herd of Models
Tülu 3	A reproducible post-training stack including SFT, preference optimization and verifiable rewards.	Nathan Lambert, Jacob Morrison, Valentina Pyatkin et al. (2024), Tulu 3: Pushing Frontiers in Open Language Model Post-Training
Kimi k1.5	Long-context reinforcement learning and long-to-short transfer.	Kimi Team, Angang Du, Bofei Gao et al. (2025), Kimi k1.5: Scaling Reinforcement Learning with LLMs
MiniMax-M1	Reasoning reinforcement learning with the CISPO optimizer.	MiniMax, :, Aili Chen et al. (2025), MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention
Kimi K2	Agentic data and training systems, including MuonClip.	Kimi Team, Yifan Bai, Yiping Bao et al. (2025), Kimi K2: Open Agentic Intelligence
Deliberative alignment	Training models to reason about explicit safety specifications.	Melody Y. Guan, Manas Joglekar, Eric Wallace et al. (2024), Deliberative Alignment: Reasoning Enables Safer Language Models
DeepSeek V4.1 Flash / Flash 1017	The Flash technical report describes a supervised starting stage, reinforcement learning across task domains, then distillation from an ensemble of teachers. It emphasizes synthesized tasks and scalable interactive environments. Supervised fine-tuning; On-policy distillation; Routing replay; Stateful agent training.	DeepSeek-AI (2026), DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
Kimi K3.1 Pro	K3.1 Pro starts from Kimi K3 and expands joint reinforcement learning across many downstream domains. The announcement ties the scale-up to long-context inference, sandbox infrastructure, rollout scheduling and environment checks. Stateful agent training; RL training systems (background).	Moonshot AI (2026), Introducing Kimi K3.1 Pro
Qwen3.5 post-trained family	The reviewed card describes progressively harder agent environments and asynchronous reinforcement-learning infrastructure. Its architecture and multimodal training are documented separately from its post-training objective. Stateful agent training; RL training systems (background).	Qwen Team (2026), Qwen3.5-397B-A17B model card
Qwen3.8 Max / Qwen3.8-2.4T-A95B	The official model card identifies 2.4T-A95B as post-trained and distinguishes its open weights from the Max service, which adds features. It documents the release and architecture without a reproducible optimization recipe. Supervised fine-tuning (background); Preference learning (background); Stateful agent training (background).	Qwen Team (2026), Qwen3.8-2.4T-A95B model card
Claude Opus 5.5	Anthropic describes fine-tuning after pretraining to align the assistant with Claude’s constitution. The system card also documents human preference selection, safety evaluation and adversarial testing. Constitutional AI (background); Preference learning (background).	Anthropic (2026), Claude Opus 5.5 System Card
GPT-6 Astra	The system card discusses reinforcement-learning grading, alignment improvements and safety behavior integrated into post-training. It reports how these changes are evaluated rather than providing the full production training recipe. Preference learning and RLHF (background); Reward design (background).	OpenAI (2026), GPT-6 Astra System Card
Gemini 4 Argon	Google lists instruction-tuning, reinforcement-learning and human-preference data in post-training. Its reasoning training draws on multi-step problem solving and theorem-proving material. Supervised fine-tuning (background); Preference learning (background).	Google DeepMind (2026), Gemini 4 Argon Model Card
GLM-5.2 / GLM-5.3	Z.ai says GLM-5.3 keeps the GLM-5.2 base and scales post-training on the same long-horizon stack: SAO, asynchronous slime training and a larger mix of environments and tasks. Single-rollout asynchronous optimization; Stateful agent training.	Z.ai (2026), GLM-5.3: Frontier Coding with Emergent Cyber Capabilities
MiMo-V2.6 Pro	Xiaomi describes large-scale multitask reinforcement learning with asynchronous generation, varied agent harnesses and group-based grading. Freezing the MoE router is one of the stated stability measures. Stateful agent training; Reward design (background).	Xiaomi MiMo Team (2026), MiMo-V2.6: Scaling Up Reinforcement Learning for Self-Improvement

3. Comparative Mathematical Reference Table

Algorithm	Data / Rollout Mode	Critic $V_\psi$?	Reference $\pi_{\text{ref}}$?	Core Objective / Advantage Estimator	Primary Strength
SFT	Offline $(x, y^\star)$	No	No	$-\sum_t \log \pi_\theta(y_t^\star\mid x, y_{<t}^\star)$	Teaches format, domain structure, and completion habits
DFT	Offline $(x, y^\star)$	No	No	$-\sum_t \text{sg}(\pi_\theta(y_t^\star)) \log \pi_\theta(y_t^\star\mid x, y_{<t}^\star)$	Reweights supervised token losses using detached target probabilities
DPO	Offline $(x, y_w, y_l)$	No	Yes	$-\log\sigma\left(\beta\log\frac{\pi_\theta(y_w)}{\pi_{\text{ref}}(y_w)} - \beta\log\frac{\pi_\theta(y_l)}{\pi_{\text{ref}}(y_l)}\right)$	Simple, stable pairwise preference alignment
IPO	Offline $(x, y_w, y_l)$	No	Yes	$\left(\log\frac{\pi_\theta(y_w)}{\pi_{\text{ref}}(y_w)} - \log\frac{\pi_\theta(y_l)}{\pi_{\text{ref}}(y_l)} - \frac{1}{2\beta}\right)^2$	Avoids DPO overfitting under deterministic preferences
KTO	Offline $(x, y, \pm 1)$	No	Yes	Prospect-theoretic value function relative to KL baseline $z_0$	Works with unpaired binary thumbs-up/down logs
ORPO	Offline $(x, y_w, y_l)$	No	No	$\mathcal{L}_{\text{SFT}}(y_w) - \lambda\log\sigma\left(\log\frac{\text{odds}_\theta(y_w)}{\text{odds}_\theta(y_l)}\right)$	Single-stage SFT + preference without reference model
SimPO	Offline $(x, y_w, y_l)$	No	No	$-\log\sigma\left(\frac{\beta}{	y_w
PPO	Online ($G=1$)	Yes	Yes	Clipped token ratio $\rho_t(\theta)$ with $\hat{A}_t^{\text{GAE}(\gamma,\lambda)}$	Fine-grained token-level credit assignment
RLOO	Online ($K$ samples)	No	Yes	$\hat{A}_i = R(y_i) - \frac{1}{K-1}\sum_{j\neq i} R(y_j)$	Unbiased critic-free baseline without std distortion
GRPO	Online ($G$ group)	No	Yes	Clipped token ratio with $\hat{A}_{i,t} = \frac{r_i - \bar{r}}{\text{std}(r)+\varepsilon}$, $\frac{1}{	y_i
Dr.GRPO	Online ($G$ group)	No	Optional	Removes $\frac{1}{	y_i
DAPO	Online ($G$ group)	No	Optional	Asymmetric [1-ε_low, 1+ε_high], dynamic sampling, token-sum loss	Prevents entropy collapse on long-CoT RLVR
VAPO	Online ($G=1+$)	Yes	Optional	Decoupled & length-adaptive GAE + value pretraining + Clip-Higher	State-of-the-art value-based RL for 32k+ CoT
GSPO	Online ($G$ group)	No	Optional	Sequence-level geometric ratio $s_i(\theta) = (\pi_\theta(y_i)/\pi_{\theta_{\text{old}}}(y_i))^{1/	y_i
CISPO	Online ($G$ group)	No	Optional	$-\text{sg}(\text{clip}(\rho_{i,t}, 1-\epsilon_l, 1+\epsilon_h))\hat{A}_{i,t}\log\pi_\theta(y_{i,t})$	Preserves gradients on rare reflective/fork tokens
SAO / V-SAO	Async Online ($G=1$)	Yes	Optional	Single-rollout async streaming + value critic + double-sided token clip	High-throughput stable async RL for long-horizon agents
PRIME	Online rollouts with outcome labels	No value critic; learned implicit process reward	Yes	Combine token-level implicit process rewards with sparse outcome advantages in an RLOO/GRPO loop	Learn process feedback from outcome labels without step-level annotations
GKD	On-Policy Student	No	Teacher $\pi_T$	Token-level JSD / Reverse-KL against teacher logits on student rollouts	10× cheaper than RLVR for training smaller checkpoints
WARP	Multi-Run Weight Space	Optional	EMA $\theta_{\text{EMA}}$	EMA KL anchor $\rightarrow$ SLERP across RL runs $\rightarrow$ LITI toward SFT	Pareto-optimal reward-KL frontier without alignment tax