Mechanism
- Rule-Based Rewards (RBR): Decomposes desired safety/behavioral style into fine-grained propositional rules (e.g., "Does not include preachy lecturing", "Clearly states inability to fulfill unsafe request", "Provides safe educational context when appropriate"), graded by a fast grader LLM and combined linearly with the helpfulness RM score during RL.
- Instruction Hierarchy: Trains the model via SFT and RL on adversarial prompt-injection pairs to strictly enforce priority ordering:
System / Developer Message > User Message > Third-Party Tool / Web / Document Output.
- Instruction Hierarchy: Trains the model via SFT and RL on adversarial prompt-injection pairs to strictly enforce priority ordering:
References
- Rule Based Rewards for Language Model Safety ↗
All 10 authors
Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, Lilian Weng
- The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions ↗