Math-Shepherd (Automated Monte Carlo Process Supervision)

Eliminates expensive human step-by-step annotation (like PRM800K) by defining the process label of intermediate step $s_k$ via automated Monte Carlo rollouts from a completer policy $\pi$:

• Hard Estimation (HE): $y_{s_k}^{\text{HE}} = 1$ if at least one of $M$ sampled continuations $\{s_{k+1:T}^{(m)}\}_{m=1}^M$ reaches the correct final answer $a^\star$, else $0$.
• Soft Estimation (SE): $y_{s_k}^{\text{SE}} = \frac{1}{M}\sum_{m=1}^M \mathbb{I}[\text{ans}(s_{k+1:T}^{(m)}) = a^\star]$.