V-STaR (Training Verifiers for Self-Taught Reasoners)

Standard STaR discards all incorrect trajectories $\hat{a}_i \neq a_i^\star$. V-STaR recycles both the correct ($y^+$) and incorrect ($y^-$) trajectories accumulated across all STaR iterations to train a DPO-based Verifier alongside the self-taught Generator, using the verifier for Best-of-$N$ reranking at test time.