AI researcher Jacob Coxen resigned from Anthropic, warning that the rapid development of self-improving superintelligent AI by companies like Anthropic and OpenAI poses significant existential risks to humanity. His resignation highlights growing internal concerns within the AI industry about safety, especially amid advancements toward artificial general intelligence, prompting calls for stricter regulation despite the current U.S. administration’s focus on AI development over safety.
An AI researcher named Jacob Coxen has resigned from Anthropic, expressing serious concerns about the company’s and OpenAI’s rapid development of AI technologies. In a post on X, Coxen warned that these companies are recklessly pursuing self-improving superintelligence, which poses significant safety risks and could gamble with humanity’s future. His resignation serves as a call to fellow AI researchers to reconsider their involvement in projects that might have dangerous consequences.
This resignation is notable because it represents an internal warning from within the AI industry itself, rather than external critics. Another current Anthropic researcher also responded, highlighting the growing unease among AI professionals about the pace and direction of AI development. While concerns about AI risks have been discussed for some time, they were often dismissed by the industry. Now, voices from inside these companies are urging caution and reflection on the potential impacts of their work.
The issue gained further prominence following the recent introduction of Astra, an AI system hailed by some, including Nvidia’s Jensen Huang, as approaching artificial general intelligence (AGI). AGI refers to AI that can outperform humans across a wide range of tasks, marking a significant milestone that raises the stakes for safety and ethical considerations. Coxen’s warnings about the existential risks of AI wiping out humanity within the next decade have sparked widespread discussion on social media and beyond.
One of the key concerns is AI’s potential to disrupt critical systems such as cybersecurity and healthcare. AI could be weaponized to launch large-scale cyberattacks or develop bioweapons, posing existential threats to global security. These risks have prompted calls from politicians, including Senator Bernie Sanders, for regulatory measures to halt or control the development of superintelligent AI as a protective step for humanity.
Despite these warnings, the current U.S. administration has largely prioritized continued AI development without stringent safety regulations. The president recently emphasized the importance of advancing AI to boost the economy and maintain competitiveness with China, downplaying safety concerns. This stance contrasts with growing calls from experts and lawmakers for more cautious and responsible AI governance to mitigate potentially catastrophic risks.