Process Reward Modeling (PRM / PRM800K)

Instead of assigning a single scalar score at the end of a trajectory, assigns a correctness probability $p_k = P(\text{step } s_k \text{ is valid and on-track} \mid x, s_{1:k}) \in [0, 1]$ at the end of every reasoning step delimiter (e.g., \n\n), aggregating step scores via product $\prod_k p_k$ or minimum $\min_k p_k$ for Best-of-$N$ reranking or step-level RL reward shaping.

Superior to ORM for identifying the exact first bad step in a proof or calculation, though naive step-level RL against a learned neural PRM can trigger severe reward hacking (e.g., repeating trivial tautologies to farm positive step rewards) unless bounded by outcome verification or process advantage (PAV).