Anthropic 提出对齐 Bumpers 策略，主张用多层防护捕捉并修正早期 AGI 失准

tl;dr

Even if we can't solve alignment, we can solve the problem of catching and fixing misalignment.

If a child is bowling for the first time, and they just aim at the pins and throw, they’re almost certain to miss. Their ball will fall into one of the gutters. But if there were beginners’ bumpers in place blocking much of the length of those gutters, their throw would be almost certain to hit at least a few pins. This essay describes an alignment strategy for early AGI systems I call ‘putting up bumpers’, in which we treat it as a top priority to implement and test safeguards that allow us to course-correct if we turn out to have built or deployed a misaligned model, in the same way that bowling bumpers allow a poorly aimed ball to reach its target.

To do this, we'd aim to build up many largely-independent lines of defense that allow us to catch and respond to early signs of misalignment. This would involve significantly improving and scaling up our use of tools like mechanistic interpretability audits, behavioral red-teaming, and early post-deployment monitoring. We believe that, even without further breakthroughs, this work we can almost entirely mitigate the risk that we unwittingly put misaligned circa-human-expert-level agents in a position where they can cause severe harm.

On this view, if we’re dealing with an early AGI system that has human-expert-level capabilities in at least some key domains, our approach to alignment might look something like this:

Pretraining: We start with a pretrained base model.
Finetuning: We attempt to fine-tune that model into a helpful, harmless, and honest agent using some combination of human preference data, Constitutional AI-style model-generated preference data, outcome rewards, and present-day forms of scalable oversight.
Audits as our Primary Bumpers: At several points during this process, we perform alignment audits on the system under construction, using many methods in parallel. We’d draw on mechanistic interpretability, prompting outside the human/assistant frame, etc.
Hitting the Bumpers: If we see signs of misalignment—perhaps warning signs for generalized reward-tampering or alignment-faking—we attempt to quickly, approximately identify the cause.
Bouncing Off: We rewind our finetuning process as far as is needed to make another attempt at aligning the model, taking advantage of what we’ve learned in the previous step.
Repeat: We repeat this process until we’ve developed a complete system that appears, as best we can tell, to be well aligned. Or, we are repeatedly failing in consistent ways, change plans and try to articulate as best we can why alignment doesn’t seem tractable.
Post-Deployment Monitoring as Secondary Bumpers: We then begin rolling out the system in earnest, both for internal R&D and external use, but keep substantial monitoring measures in place that provide additional opportunities to catch misalignment. If we observe substantial warning signs for misalignment, we return to step 4.

The hypothesis behind the Bumpers approach is that, on short timelines (i) this process is fairly likely to converge, after a reasonably small number of iterations, at an end state in which we are no longer observing warning signs for misalignment and (ii) if it converges, the resulting model is very likely to be aligned in the relevant sense.

We are not certain of this hypothesis, but we think it’s important to consider. It has already motivated major parts of Anthropic’s safety research portfolio and we’re hiring across several teams—two of them new—to build out work along these lines.

Introduction

We are plausibly within a couple of years of developing models that could automate much of the work of AI R&D. This makes sabotage and sandbagging threat models—where the models deliberately interfere with their own safety measures or hide dangerous capabilities—worth addressing soon: If we are to rely substantially on automated alignment research, these are the systems that will take on the first load-bearing autonomous alignment projects.^[1] If these models are themselves well aligned, our work gets much easier past that point. If these models are misaligned in a way that motivates them to sabotage or sandbag, then at best, we will pay enormous opportunity costs from our inability to trust our best tools, and at worst, we will foolishly trust them and thereby allow them to sabotage our work in ways we fail to notice or unwind.

Ideally, before this point, we would have a thorough understanding of the empirical landscape surrounding AI alignment: If we run our alignment process and it appears to succeed, we should be confident that the model is in fact aligned.

There are few research problems in the world that compare in importance to the problem of gaining this understanding (and many of the problems that do compare in importance involve AGI safety in some way). But it’s unlikely that we’ll succeed to an adequate degree in time, and it’s even less likely that we’ll know we’ve succeeded: Solving this problem definitively could easily take many years, but because of the current paucity of frontier regulation, the competitive nature of the industry, and the immense potential upsides of deploying powerful AI for both safety research and the world at large, we can’t count on being able to wait that long to start using these early AGI systems.

Despite all this, I expect that we can still muddle through without this kind of deep understanding in a way that nonetheless makes a good outcome likely.

Bumpers is the high-level plan that I argue AGI-focused alignment research efforts should largely prioritize instead—the way that I expect we muddle through. It relies on a key hypothesis about our situation:

Detection of misalignment seems tractable: If an early AGI system is misaligned enough to cause unrecoverable harm, it seems likely that there is some known technique that could be used to determine that it is misaligned before this takes place—perhaps, but not necessarily, before deployment.

If this is false, attempting to run the iterative process above will not prevent us from unwittingly placing a misaligned model in a position where it can cause severe harm.

Further, the claim that this plan should be a top priority for AGI developers relies on three more claims:

Alignment is not clearly trivial: Ordinary attempts to align early AGI systems could plausibly produce a misaligned result.

If this is false, and we know that it is false, then Bumpers work is a waste of effort and a cause of potentially-costly delays.^[2]

Alignment is likely to be tractable: The best current evidence suggests that existing applied alignment techniques like RLHF and Constitutional AI can, at least under ideal circumstances, be used to adequately align early AGI systems. While we may not fully understand what we’re doing when we align a model, we don’t need to achieve any deep breakthroughs in order to succeed in practice.

If this is false, Bumpers work can still carry enormous value by surfacing early evidence that it is false. This evidence can then drive both newly-focused research investments and newly-concrete public communication about risks. However the emphasis on using the bumpers to quickly iterate on our alignment techniques is misplaced in these situations.

Alignment theory is not tractable in the short term: We are unlikely to understand misalignment risk well enough in the short term to be able to reliably predict whether some specific attempt to align a model is going to succeed.

If this is false, Bumpers work represents a clumsy and uncertain way to solve a critical problem that we could solve with greater confidence by other means.

This is not a costless choice: The Bumpers worldview largely gives up on the ability to make highly-confident, principled arguments for safety, and it comes with real risks. A trial-and-error approach that relies on the same set of tests many times will start to create selection pressure that undermines those tests. Further, some possible forms of misalignment would result in systems that actively attempt to subvert our bumpers.

I nonetheless expect that we could implement a half-dozen or so bumpers in such a way that no model (in the relevant capability regime) can subvert all of them at once. This is not guaranteed, but it’s the best available option if capabilities progress continues to move as quickly as we are anticipating here.

The hypotheses above are distinct from any normative claim about what level of confidence we should aim to have in the safety of a system before we deploy and use it, at least in the short term: Under most plausible levels of confidence an AI developer might be able to reach in the near future, for the capabilities regime we target, it will be easier to reach that level of confidence by prioritizing the building and testing bumpers than by building out a deeper science of alignment that’s aimed at more decisively solving the problem.

What Are the Bumpers?

There are many methods that could plausibly detect misalignment during pre-deployment testing, with varying degrees of present day maturity, such as:

Manual expert red-teaming, including:

Honeypot environments that mimic opportunities to interfere with human control, potentially building on real-world data
Non-assistant persona interviews
Activation fuzzing, steering, and similar methods of injecting variation into model reasoning

High-volume automated red-teaming to look for jailbreaks and other unusual triggers of bad behavior
Interpretability audits to look for internal representations related to deception or misaligned goals
Automatic consistency audits to look for strategic biases in model behavior, like systematically omitting certain types of information from summaries, or doing the same coding task in suspiciously different ways when the stated purpose is different

This could include scalable-oversight-style methods that can identify where a model’s behavior is most likely to diverge from what the model would endorse on reflection^[3]

Mechanistic anomaly detection, probes, and other model-internals-based methods for catching shifts in behavior

Beyond these, there are additional methods that can be put in place after deployment that may be able to catch warning signs of misalignment in the wild before any serious irreversible harm takes place. These secondary defenses include:

Real-time monitoring, human rapid-response measures, and related deployment-time AI control measures^[4] to catch and block acutely misaligned behavior

This could include the continued use of anomaly detection related white-box methods that have access to the model’s internal computation

Summarization-based batch monitoring (as with Clio or per-user hierarchical summarization methods) to look for more diffuse concerning trends
Staged rollouts, starting—both internally and externally—with specially chosen user populations who you trust to surface issues to you and trust not to use the system in high-stakes settings
Security measures that limit models’ ability to access internal systems autonomously

To the limited extent that modern reasoning models produce transparent reasoning traces, these traces will make several of the methods above, in both categories, more effective.

No one of these methods, under realistic circumstances, will be robust enough to rely on. However, even a half dozen of these methods, each implemented to the standard of a substantial production effort at an AGI developer, should be sufficient to detect potentially-catastrophic forms of misalignment in any plausible early AGI system.

A system that was trying to evade all of these bumpers would face immense obstacles, at least within the capability regime we target. Such a system would have to be able to take intentional, harmful actions with neither its human-accessible internal feature representations nor its written reasoning traces ever hinting at that intention, even under extensive red-teaming that can manipulate its inputs and internal states in arbitrary ways. It would need to be able to distinguish real opportunities to act from our many artificial test cases. If recent trends in model design continue, it also would need to do this without being able to do much more active reasoning within its forward pass than happens in present-day models. It would need to do all of this in a setting where it is not substantially superhuman, such that humans are routinely (though not continuously) spot-checking its work.

Hitting the Bumpers and Bouncing Off

If we encounter a warning sign that represents reasonably clear evidence that some common practice will lead to danger, the next step is to try to infer the proximate cause. These efforts need not result in a comprehensive theory of all of the misalignment risk factors that arose in the training run, but it should give us some signal about what sort of response would treat the cause of the misalignment rather than simply masking the first symptoms.

This could look like reading RL logs, looking through training data or tasks, running evals across multiple training checkpoints, running finer-grained or more expensive variants of the bumper that caught the issue in the first place, and perhaps running small newly-designed experiments to check our understanding. Mechanistic interpretability tools and related training-data attribution tools like influence functions in particular can give us clues as to what data was most responsible for the behavior. In easy cases, the change might be as simple as redesigning the reward function for some automatically-graded RL environment or removing a tranche of poorly-labeled human data.

Once we’ve learned enough here that we’re able to act, we then make whatever change to our finetuning process seems most likely to solve the problem. Unless the warning sign also calls into question the continuing effectiveness of our suite of bumpers,^[5] we finally proceed to retrain the model as much as seems necessary to mitigate the issue and attempt a new deployment.

In addition, by this point, we will likely have learned some repeatable lessons, which we should share externally. This is both to allow other developers to avoid encountering the same danger themselves and to allow governments to respond in full knowledge of the risk.

As a limited early example from our real model development lifecycle: Our evaluations of partially-trained versions of Claude 3.7, as described in the System Card, found signs of reward hacking on code tasks that we were able to partially^[6] mitigate with adjustments to our standard finetuning techniques.

Defending Key Premises

Detection of Misalignment Seems Tractable

The findings of our recent Auditing paper suggests that, at least under the relatively favorable circumstances of our first major trial, there are a several independent methods involving (among other things) dictionary learning and non-assistant persona prompting that can reveal important facts about a model’s alignment that are not readily observable in ordinary behavioral testing.

More broadly, we have seen essentially no naturally-occurring cases to date where a model is misaligned in ways that clearly could not have been caught by thorough pre-deployment testing that includes methods like these. To the partial but nontrivial extent that existing interpretability techniques give us a view into model cognition, what we’re finding seems much more consistent with the hypothesis that models are approximating what we trained them to do in reasonable ways than that they are subtly acting against us. When we do see misalignment, it generally seems to be a consequence of contingent, understandable errors in our training strategies.

It is not clear that this should continue to be the case. (And see Hitting the Capability Ceiling below for more discussion of how alignment might cease to be tractable in this sense.) But it seems like a fairly likely default outcome, and as such, it should be among our top priorities to continue to build out and stress-test these detection methods.

Alignment Seems Tractable but not Trivial

We have not yet failed badly at alignment. Current frontier systems have been deployed widely without this surfacing any deep divergences from the intentions of their developers of the kind we contemplate here. Something like this degree of alignment in an early AGI system, coupled with good progress on task elicitation that I expect to emerge from more capabilities-oriented R&D,^[7] should be sufficient to safely begin deploying.

Current frontier systems also have few or no deep, consistent limitations that prevent them in principle from acting as human-level autonomous researchers: We expect that the difference between today’s systems and these future systems with these capabilities will be more a difference in degree than a difference in kind. As such, we find it plausible that a similar degree of alignment can be attained by basically similar means.

However, we know that standard alignment techniques can fail, sometimes in reasonably subtle ways. Many recent results, including Emergent Misalignment, Alignment Faking, Sycophancy to Subterfuge, Wen et al.’s, and Baker et al.’s suggest that some ways of using current standard techniques can lead to misaligned models which attempt to subvert human control. At least some of these types of failure plausibly become more challenging to avoid as models become more capable. Our challenge, on the Bumpers view, is to build many redundant ways of catching and correcting this when it occurs.

In the language of our Core Views on AI Safety post, the Bumpers approach is primarily targeting ‘intermediate-difficulty’ scenarios for alignment, where alignment is tractable but not trivial. In those scenarios, this approach allows us to chart a course to successfully deploying an aligned model. However, we expect it to also be valuable, to a lesser extent, in many plausible harder worlds where this work could provide the evidence we need about the dangers that lie ahead.

Red-Teaming the Bumpers

The core recommendation of this piece is to build out and deploy the potential bumpers that have already been explored in some depth in past research. However, as a major secondary priority, we should be testing them in situ and in combination, with a focus on ensuring that no plausible model could circumvent the full suite.

This would entail red-teaming exercises in the spirit of the Control agenda’s meta-level adversarial evaluations: Here, ‘red-team’ researchers—who have full knowledge of the set of bumpers that a model will face—aim to produce an intentionally-misaligned model variant that is optimized to be as likely as possible to survive all of the bumpers while still ultimately taking some (fake) severely-misaligned action. If these researchers succeed, that calls into question the adequacy of the bumpers, and likely indicates ways they can be improved. If these researchers fail, though, this gives only indirect and contingent positive evidence for safety, since the space of possible misaligned models is large.

Related Agendas: Model Organisms and Control

The Model Organisms of Misalignment agenda aims to intentionally develop and test misaligned models in order to understand how misalignment might emerge. This is a natural basic-research companion to Bumpers: Model organisms are an ideal input to the red-teaming exercises described above and evidence from work with model organisms help us make better guesses about where to strengthen the bumpers and where correlated failures across bumpers could be possible. In the other direction, each time we hit a bumper and encounter warning signs for misalignment in the real world, that can feed back into our decisions about which model organisms to build and study.

The AI Control agenda aims to build model-external safeguards that could let us to use an arbitrarily misaligned model without giving it the opportunity to cause severe harm. Bumpers also calls for model-external safeguards and monitoring and will likely entail some of the same work in practice. But Bumpers is more oriented toward gathering and responding to evidence about the actual degree to which a model is aligned and contemplates the possibility of building safeguards more selectively in response to that evidence. Bumpers thus represents a distinct bet on the premises that alignment is reasonably likely to be both tractable and necessary.

Limitations

Selection Pressure and Iteration Limits

Each time we go through the core loop of catching a warning sign for misalignment, adjusting our training strategy to try to avoid it, and training again, we are applying a bit of selection pressure against our bumpers. If we go through many such loops and only then, finally, see a model that can make it through without hitting our bumpers, we should worry that it’s still dangerously misaligned and that we have inadvertently selected for a model that can evade the bumpers.

How severe of a problem this is depends on the quality and diversity of the bumpers. (It also depends, unfortunately, on your prior beliefs about how likely misalignment is, which renders quantitative estimates here pretty uncertain.) If you’ve built excellent implementations of all of the bumpers listed above, it’s plausible that you can run this loop thousands of times without meaningfully undermining their effectiveness.^[8] If you’ve only implemented two or three, and you’re unlucky, even a handful of iterations could lead to failure.

Ambiguous Findings

Unfortunately, not all warning signs will come with smoking-gun evidence that the model would have caused harm if given the opportunity: In behavioral evaluations, there is no crisp distinction between errors and intentional patterns of bad behavior. White-box tests involve some lossiness when bridging the divide between complex model internals and high-level concepts like goal.

A warning sign warrants a reaction if it causes well-informed experts to believe that the model is less well aligned than expected or that it has a non-negligible chance of being severely misaligned. These judgment calls will have to be responsive to the broader competitive and regulatory landscape: It is prudent to investigate more risks, and to investigate them more deeply, if you have reasonable assurances that other actors aren’t already taking much greater risks.

No Free Lunch

It is possible, and consistent with all of the above, that adequately aligning a human-expert-level model is enormously slow and expensive. The Bumpers approach is still plausibly the most efficient strategy in this case, but it doesn’t save us from having to put in the work: It may be the case that our first warning signs of misalignment make it clear that the most promising alignment strategies require huge amounts of compute or data, or that each ‘bounce’ will involve discarding large parts of an expensive tuning run and starting over.

For this plan to succeed under these conditions, it’s still necessary that the competitive and regulatory landscape allows developers to take however much time is needed to make early AGI systems demonstrably safe.

Hitting the Capability Ceiling

As AI capabilities move beyond the ‘early AGI’ regime I contemplate here, the two key assumptions behind the bumpers strategy are likely to start to fail.

With systems that are substantially superhuman at reasoning, alignment will likely become less tractable with existing methods, as the challenges motivating scalable oversight research start to bite. This should make us less confident that the core loop of this process will terminate: It may take deeper new research to work out how to align these systems.

Similarly, with systems that are substantially superhuman at reasoning about AI safety, it becomes more plausible that they discover exotic ways of evading bumpers that no human adversarial evaluation could have caught. This should make us less confident that if the core loop of this process does terminate, that this means that the resulting model is in fact aligned. Estimating when we have reached this point is not trivial, but ordinary capability assessments, red-teaming exercises, and warning signs from earlier model generations will all give us substantial evidence with which to make this determination.

Fortunately, once we’re in this higher-capability regime, we will likely already have access to early AGI systems that should (with some preparation) be able to automate significant parts of AI safety R&D at the level of human experts. If we use these agents well, as in the automated alignment researcher strategy, it becomes much more plausible that we’re able to accelerate progress on safety across the board, giving us a wider range of options with which to address these challenges.

Conclusion

To get through the transition to superhuman AI, we’ll likely need to delegate important tasks to circa-human-level automated researchers. If those automated researchers are misaligned in a way that leads them to sabotage our work—or do any of the other harmful things that models at this capability level could do—the outlook is poor. So, we need to be able to align these automated researchers.

But it seems unlikely that we’ll have a robust understanding of how to align models with these capabilities in time: Our depth of understanding will probably look more like today’s ML engineering than today’s aerospace engineering, which is to say that we won’t be very confident in what we're doing. Given the potential stakes, this is a humbling and disconcerting state of affairs, but it’s one that we shouldn’t look away from.

Instead of aiming to get toward the aerospace end of the spectrum—a project that has little hope of paying out if things move quickly—I propose that we accept that our early AGI alignment work is very likely to depend critically on trial and error, and put up as many mechanisms as possible—the bumpers—to help us identify and respond to those errors. With an ambitious but feasible effort there, I think we’ll be able to muddle through.

Come Help

Anthropic is committed to investing seriously in the kinds of measures described here, and we’re eagerly hiring. The more excellent engineers and researchers there are who can contribute to this work, the faster we can move, the more we can publish, and the less likely we are to have to triage away more ambitious bets. If you think you’d be able to contribute, have a look at our jobs page.

Within the Alignment Science umbrella, I’m setting up a new team to productionize and professionalize the hands-on work of testing models for AGI-relevant forms of misalignment. Elsewhere in Alignment Science, we recently founded a new Cognitive Oversight research team under Sam Marks that’s aiming to improve upon, test, and better understand the sorts of assessment methods we used in our recent Auditing paper. We are continuing to hire for our Model Organisms and Control teams to continue to advance the related agendas described above.

Beyond Alignment Science, Anthropic’s implementation of the agenda laid out here also depends directly on work that’s happening on Interpretability, Safeguards, Finetuning, and the Frontier Red Team. All of these teams are actively hiring.