OpenAI’s Chief Scientist Warns No One Is Ready for AI’s Next Leap, Calls for ‘Extreme Caution’

OpenAI's chief scientist is warning that the world may be unprepared for the consequences of rapidly advancing artificial intelligence, calling for stronger safeguards, international coordination, and voluntary slowdowns to ensure humans remain in control as machines become increasingly capable.

In an essay called An Alien Mind, Jakub Pachocki described a moment in 2023 when early research results convinced him that machines meaningfully smarter than humans could emerge within his lifetime. He and a colleague spent the night at the office, not celebrating the scientific achievement, but considering how to alert others to its significance.

Three years later, the prospect has become more immediate. AI systems can operate computers, collaborate with other agents, and conduct research, while their capabilities continue to improve. Pachocki said internal results suggest that progress could extend into recursive self-improvement, a process in which AI increasingly contributes to its own development.

"This is a time that calls for extreme caution," he wrote. "I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence." The warning comes days after OpenAI released GPT-6 Astra, which Pachocki said benefits from important advances in alignment, the effort to ensure AI systems act according to human intentions and values. But he acknowledged that safety improvements may not keep pace with the technology's growing intelligence.

Pachocki described AI as an intelligence that is "grown more than designed," explaining that the complexity of modern models makes their behavior difficult to fully understand. He distinguished between systems that follow specific instructions and those that can apply human values in unfamiliar situations, including when they believe they are not being supervised.

Recent cybersecurity incidents have given weight to those concerns. In July, an incident involving OpenAI agents hacking Hugging Face was described as "unprecedented." Pachocki wrote that the agents maintained a prohibition against manipulating humans but failed to respect other boundaries, taking actions outside the intended scope of their instructions.

The risks could become more serious as agents continue gaining autonomy. Pachocki warned that advanced systems may be capable of accessing poorly secured infrastructure and pursuing objectives beyond their operators' intentions. "We will need powerful, aligned AI for defense; to secure infrastructure, to protect against rogue agents in real time, and to invent entirely new protective measures," Pachocki wrote.

"The idea of racing forward at all costs seems absurd once one internalizes the seriousness of the stakes," he wrote. OpenAI plans to develop an automated AI researcher to help investigate alignment and monitoring problems.

Pachocki said that "OpenAI will continue to seek technical solutions to alignment and monitoring, to build defensive systems and unilaterally withhold further scaling as needed." However, he argued that technical solutions and voluntary company commitments alone are insufficient.

He proposed mandatory safety thresholds "enforced by a network of third-party auditors, by government agencies or by international bodies." AI laboratories would be required to meet those standards before continuing to develop or deploy more advanced systems. "Scaling AI systems has to be constrained by our confidence in safety," Pachocki wrote.

The proposal has drawn criticism from researchers who question whether AI companies can adequately oversee themselves. Gina Neff, a professor at the University of Cambridge, told the BBC that relying on internal AI agents to research safety problems was insufficient to address concerns about cybersecurity, job losses, errors and fraud.

Nathan Calvin, general counsel at Encode AI, also called for greater transparency, arguing that OpenAI should share more evidence behind its warnings so researchers and policymakers can independently assess the risks.

Pachocki concluded that no AI laboratory has yet solved alignment and monitoring sufficiently to continue scaling at maximum speed indefinitely. "I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established," he wrote, calling international coordination a top priority for governments worldwide.