Get Started
Home
Topics
Search
Library
Research questionHow can reward shaping reduce reward hacking in RLHF when rewards imperfectly capture human preferences?In RLHF, a policy may optimize quirks in a learned reward rather than the behavior humans intend. Reward-shaping choices influence how reliably optimization follows preferences and how stable training remains.
AI
Alignment & Safety
LLM Pretraining & Post-training
Machine Learning
Reinforcement Learning
Latest papersRecent research connected to this question, newest first.Reward Shaping to Mitigate Reward Hacking in RLHFThe study concerns reward shaping for large language models in RLHF. It evaluates shaping strategies with PPO and Gemma2-2B, and compares the proposed Preference as Reward method with unshaped rewards across four base models and four reinforcement-learning algorithms; reported evidence includes data efficiency, robustness, variance reduction, early-stopping flexibility, and performance when actor-critic value estimates become unstable.research paper · Sep 3, 2026
Related questions
How can black-box systems detect and mitigate reward hacking in self-evolving language-model loops?How can hierarchical reinforcement learning use incrementally acquired knowledge for long-horizon exploration with sparse rewards?When does environment design make on-policy RL amplify harmful specification gaming in language models?How can active preference learning obtain scalable, calibrated uncertainty for neural reward models without full Bayesian inference?