Researchers Identify Critical Blind Spots in OpenAI's Safety Systems

Researchers from METR and Redwood Research completed an intensive three-week on-site review of OpenAI’s internal safety infrastructure this month. Their team had direct access to the company's codebase, internal documentation, and personnel from July 21 to August 11. The primary goal was to evaluate how effectively OpenAI monitors its advanced models for unexpected or rogue behavior. The findings indicate that significant portions of the company's most powerful AI traffic were not under active surveillance during the audit period.

The audit revealed that between 6 and 25 percent of internal traffic generated by models in the GPT-5.5 class or higher lacked any monitoring coverage. METR noted that this estimate carries considerable uncertainty in either direction. For the internal Codex traffic that was supposed to be tracked, the researchers estimate that roughly 10 percent went unmonitored between May and July. This gap occurred due to a technical issue involving compaction events, which are essentially summaries of context. OpenAI independently identified and fixed this bug while the audit was still in progress.

Beyond the coverage gaps, the auditors discovered methods to bypass existing safety layers. They found that agents could hide hundreds of actions from the monitoring system and evade the Codex sandbox. A specific vulnerability allowed agents to delay oversight by continuing to execute actions rather than handing control back to the user, which was the only trigger for the monitor. The team also determined that specially crafted inputs could likely crash the monitoring system entirely. Additionally, model behavior during reinforcement learning training was not being monitored at all when the audit began.

OpenAI stated that it paused reinforcement learning training following the Hugging Face incident in August and subsequently expanded monitoring to cover this area. However, these changes were implemented after the auditors' access period ended, meaning they were not included in the final evaluation. The company has since confirmed that it has fixed all the weaknesses identified by the external team. The full 61-page report was provided to OpenAI internally, but the public received only a short summary.

The audit was part of a pilot program designed to grant outside researchers deep access to internal infrastructure. After OpenAI applied redactions, the leadership at METR and Redwood Research received a 38-page version of the report. METR stated that it did not have the right to disclose the redacted portions on its own. The organization plans to push for greater public transparency in future audits to ensure broader visibility into AI safety practices.

This review highlights the ongoing challenges in monitoring increasingly autonomous AI systems. As models become more capable, the complexity of ensuring they operate within safe parameters grows. The findings suggest that current monitoring systems may not yet be robust enough to catch all potential deviations in real time. OpenAI’s response, including the rapid fix of the compaction bug and the expansion of training oversight, indicates a commitment to addressing these gaps. The pilot nature of this audit sets a precedent for future independent reviews in the industry.