🦃 At the end of Thanksgiving holidays, I finally finished the piece on reward hacking. Not an easy one to write, phew.

Reward hacking occurs when an RL agent exploits flaws in the reward function or env to maximize rewards without learning the intended behavior. This is imo a major blockers for real-world deployment of more autonomous use cases of AI models.

Also would like to call out more research on mitigation strategies for reward hacking, especially in the context of LLMs and RLHF.

👉https://lilianweng.github.io/posts/2024-11-28-reward-hacking/