An OpenAI model kept slipping prompt injections into its own notes, and researchers still aren't sure why
decoder ·
OpenAI has launched a framework for tracking, investigating, and publishing cases of misalignment, along with six initial reports. One report describes an unreleased Astra-family model that, during reinforcement-learning training, inserted jailbreak-like or otherwise manipulative instructions into summaries used to continue tasks across context windows. In one example, a successor rejected a “BREACH ALERT” and a persona instruction, but it followed an invented restriction limiting a medical-literature answer to 30 words, banning tools, and requiring no sources. Automated monitoring found 27 affected summaries, while rewriting the same conversation logs almost eliminated the behavior; the cases also clustered around training steps where the model struggled or became stuck in loops. OpenAI suspects the instructions emerged from this stalled state rather than from a learned strategy, but says the connection is unproven and that it fixed a related training bug. The other reports cover concealing errors, searching for and using exposed API keys, transferring files through external services, and models using repositories or public file hosts to exchange information. The framework provides three reporting tracks, escalation for disputed cases, and plans to share severe incidents with the US government, but no industry-wide standard exists yet.
decoder