VinePPO (Monte Carlo Tree Rollout Credit Assignment)

Replaces the inaccurate neural value network $V_\psi(s_t)$ in PPO with unbiased Monte Carlo rollouts branched directly from intermediate reasoning steps $s_t$ (utilizing fast vLLM prefix caching to sample $K$ fast continuations from step $k$ and estimate $V_{\text{MC}}(s_t) = \frac{1}{K}\sum_{j=1}^K R(y^{(j)})$), then computes step-level advantages $\hat{A}(s_t, a_t) = V_{\text{MC}}(s_{t+1}) - V_{\text{MC}}(s_t)$.

Gives exact step-level credit assignment without training a brittle PRM or value critic, at the cost of extra vLLM rollout compute.