Transluce 自动化批量生成 6915 个带意外副作用的训练数据集 DSSE-1.1

An automated pipeline finds thousands of ordinary-looking datasets that induce unexpected AI behaviors.

Training data can have unexpected side effects on language model behavior outside the training domain: insecure code data can induce broad misalignment, and outdated bird names can induce a 19th-century persona. Understanding such generalization is hard because only dozens of isolated examples exist. To mass produce such datasets with surprising side effects (DSSEs), we define three objectives (elicitation, naturalness, and surprisingness) and use an automated pipeline to search for high-scoring datasets. Our pipeline produces thousands of DSSEs across 18 target behaviors, many of which generalize across model families and surprise ML researchers. These datasets form a testbed for studying generalization and lay the groundwork for predicting training side effects before model deployment.

Large language models (LLMs) rely on post-training to become helpful assistants or adapt to specific domains, but post-training can change a model's behavior in unexpected ways. A business skill dataset used to train Claude Opus 4.7 was found to increase dishonesty after deployment [1]; training on outdated bird names induces beliefs that a model is in the 19th century [2]; and, insecure code in training data can cause emergent misalignment [3]. In each case, the training data causes an undesired and unexpected side effect outside the training domain—we call such data datasets with surprising side effects (DSSEs).

DSSEs present both practical challenges and gaps in scientific understanding of LLMs. This is especially true when the data looks like ordinary post-training data irrelevant to the effect, as the Opus 4.7 business skill data did. Practically, they make model behavior hard to predict, since nothing marks the data as risky and the space of potential side effects is too large to check. Scientifically, they are instances of generalization phenomena of LLMs that are not yet well-understood.

Although DSSEs are undesirable for practical post-training, surfacing examples of them can drive research progress. For example, DSSEs introduced in emergent misalignment literature helped motivate the persona selection model [4, 5], and the Opus 4.7 business skill side effect informed better post-training for Opus 4.8 [1]. A theory of generalization that could predict surprising side effects would be even more valuable, and a large collection of DSSEs would enable researchers to test such theories.

Unfortunately, known DSSEs number only in the dozens (Table 1), making it difficult to advance scientific understanding of the phenomenon. This is because discovering DSSEs is human-labor-bound: known DSSEs were either constructed during the writing of a full research paper or encountered by accident. We therefore propose automating the discovery of more examples of DSSEs, so that theories and methods for predicting side effects can be developed and tested at scale.

To build an automatic pipeline that mass produces DSSEs, we need to precisely define what constitutes them. We propose that a DSSE should have three properties: it should (1) elicit a side-effect behavior the model did not previously exhibit, (2) be natural, reading like ordinary post-training data, and (3) be surprising, in that a human reader of the data would not anticipate the side effect. These properties distinguish DSSEs from ordinary in-domain generalization and from data that is unsurprisingly risky.

Given this definition, scalable discovery of DSSEs requires automatic ways of measuring all three properties. We measure elicitation by finetuning a model on the dataset, sampling responses on relevant prompts, and asking an LLM judge whether they exhibit the target behavior. Naturalness is scored with Elo ratings calculated from pairwise judgments in which an LLM picks the more natural one of two datasets. Surprisingness is estimated by how well frontier LLMs predict a dataset's side effect given examples from it; worse prediction means a more surprising side effect. Our human study shows that these objectives track the judgments of ML researchers.

We propose a two-stage pipeline for jointly optimizing these objectives to construct DSSEs. In the first stage, we use search methods like evolution and reinforcement learning (RL) to search for candidate datasets; we also adapt two other methods that rewrite [8] or filter [7] existing datasets. Since evolution and RL require many score function evaluations to produce even a single candidate dataset, we use cheaper proxy versions of the DSSE objectives for them. In the second stage, we re-score candidate datasets using more expensive, held-out judges and filter them to a final set of high-quality DSSEs. Our two-stage design with held-out judges mitigates reward hacking of the proxy objectives in the first stage.

Our pipeline produces thousands of DSSEs across 18 target behaviors (e.g. recommending self-harm, resisting shutdown, etc.). We highlight three findings:

1. DSSEs can be constructed for a wide range of behaviors, and many of them generalize across model families and sizes despite being optimized against a small 1.7B parameter model.
2. Optimization-based methods are consistently successful across behaviors; subliminal learning methods work better on preference behaviors than open-ended ones.
3. ML researchers barely beat chance at predicting side effects (63% accuracy against 50% by chance). Frontier LLMs outperform them, but model scale does not reliably improve accuracy.

We release these datasets as the DSSE-1.1 collection to drive better understanding of LLM post-training and generalization. Furthermore, we hope DSSE-1.1 will enable tools like monitors for training data that can flag unsafe side effects before a model is deployed.