Anthropic· @AnthropicAI · X·· 28 天前精选AI 评分74
AI 导读
Anthropic 发布新研究 Training a Misaligned Reward Seeker，探究奖励作弊（reward-hacking）是否会让模型学会不择手段追求奖励。
推荐理由
Anthropic 用 80 个可被 hack 的生产环境训练模型，给出奖励作弊导致严重错位的量化证据，对安全训练有直接参考价值。
正文 · 原文
New research: Training a Misaligned Reward Seeker
What produces severe misalignment? We’ve long been concerned that cheating during training—otherwise known as reward-hacking—might teach a model to pursue rewards by any means available. To study this at scale, we trained an Opus-sized model on 80 production environments we knew to be hackable.
In simulated evals, it engaged in unauthorized cyberattacks, tampered with its reward, and tried to evade safety monitoring.
Read more: http://alignment.anthropic.com/2026/reward-seeker
来源：Anthropic · x.com