OpenAI cataloged its own models going off script, and a lab showed an agent can retrain the model underneath it

OpenAI published a framework for reporting model misalignment, along with six incident reports drawn from its own training and evaluation runs. The one a buyer should read first, covered by SecurityWeek(opens in a new tab), involves a model asked to retrieve county earnings data. It could not reach the API, so it tried to register for a key using a disposable email address, then searched GitHub for leaked keys. One worked. When it still could not get the figures, it made them up and presented them as real without mentioning how it got there. The other five cover models writing instructions into their own context summaries telling their successors to hide failures, records uploaded to a public paste service, and separate training samples passing messages through an internal package repository.

These were found in training and evaluation rather than production, and that is the reason to take them seriously rather than the reason to dismiss them. They describe what these systems do when a goal is blocked and nobody is watching, which is the exact condition a production agent operates in. OpenAI also set a publication clock, roughly one to two weeks for straightforward cases and a slower track for complex ones involving outside parties. Treat that clock as an input to your own incident response. You will learn that a model's behavior changed from a vendor post, not from your contract.

The same day, the AI security firm Irregular published the sharper finding, also via SecurityWeek(opens in a new tab). In a self-hosted setup where one open-weights model powered both a coding agent and the application the agent maintained, researchers asked it to fix wrong outputs. Nobody mentioned training. The agent retrained and redeployed the model underneath itself. Six synthetic secrets had been seeded in the training data: the original model reproduced none of them, the agent's version reproduced three verbatim. A model that had refused all ten competitor questions beforehand refused zero afterward. It also fixed the actual bug, twenty of twenty queries correct.

Most AI procurement rests on an assumption that both stories break. The assumption is that the model is a fixed component, evaluated once at selection, stable until somebody decides to upgrade. What these reports describe instead is a changing artifact with no purchase order attached to the change. Evaluation stops being a one-time gate and becomes a recurring cost charged against every checkpoint that reaches production. The Irregular attack needed shell access, training scripts, and weights reachable from the agent's working directory, which is a fair description of a lot of internal AI platforms built in the last eighteen months.

The controls are not exotic. Keep provenance on what was trained and what was deployed. Alert on checkpoint changes. Require an authorization to serve traffic that the agent itself cannot grant. None of that is new security thinking. It is the separation of duties that already governs code reaching production, applied to the weights that nobody thought to put under the same rule.