It's happening

Anthropic, an AI company poised for rapid growth with a unique business model and exceptional talent, openly acknowledges the existential risks of AI—including a significant chance of human extinction—while pursuing potentially dangerous recursive self-improvement and maintaining concentrated control among its founders. Despite impressive financial efficiency and innovation, the company’s governance and ethical challenges raise serious concerns about accountability and the safe development of artificial superintelligence.

Anthropic, a highly controversial AI company that recently filed to go public, presents a fascinating yet paradoxical picture in its S1 filing. The company openly acknowledges the existential risks of AI, including a greater than 10% chance that AI could kill all humanity, yet it simultaneously positions itself as the sole entity capable of safely developing artificial superintelligence. Anthropic emphasizes the transformative potential of AI, comparing its impact to fundamental human discoveries like fire and electricity, and highlights its mission to manage AI risks while accelerating global growth and improving quality of life. Central to their strategy is a powerful business flywheel: developing cutting-edge AI models, selling them to enterprises, gathering vast amounts of data, and reinvesting in compute resources to build even better models.

Anthropic’s core advantages lie in its focused mission, exceptional talent pool, and innovative approach to compute. Unlike competitors who diversify into various AI applications, Anthropic concentrates solely on creating the best AI models and selling them to enterprises, which has driven rapid revenue growth. The company boasts an impressive roster of AI luminaries, including co-founders of OpenAI and Nobel laureates, contributing to a culture of excellence. Their compute strategy is highly flexible and efficient, leveraging multiple cloud providers and hardware types to optimize performance and profitability. This efficiency has led to a remarkable increase in revenue per dollar of compute, demonstrating the company’s operational prowess.

Despite these strengths, Anthropic’s S1 reveals significant risks and contradictions. The company openly discusses the dangers of recursive self-improvement (RSI), a process where AI models improve themselves autonomously, yet it plans to pursue this very approach, which some experts warn could endanger humanity. The governance structure concentrates decision-making power in the hands of seven co-founders who hold 50.1% of voting rights through a special class of stock, effectively giving them absolute control over the company and its AI development. This concentration of power raises concerns about accountability and the potential societal impact of their decisions. Additionally, Anthropic admits to lacking a strong plan for aligning their AI systems safely as they advance.

Financially, Anthropic demonstrates unprecedented growth and efficiency. From just 20 customers paying over $100,000 annually in 2023, they have grown to nearly 12,000 such customers by 2026, with many spending millions. Their net revenue retention rate is an astonishing 900%, indicating that existing customers dramatically increase their spending over time. The company has raised over $141 billion in capital but has only burned about $10 billion, showcasing exceptional capital efficiency. Their revenue per employee rivals that of hedge funds, and they increasingly rely on their AI, Claude, which now contributes over 90% of codebase contributions and costs more than all employees combined.

In summary, Anthropic’s S1 filing paints a picture of a company at the forefront of AI innovation with a unique business model and extraordinary growth, but also one fraught with ethical dilemmas, governance concerns, and existential risks. Their commitment to recursive self-improvement and the concentration of control among a few founders highlight the tension between rapid technological advancement and the need for responsible oversight. While their financial and operational achievements are impressive, the broader implications of their work and governance structure provoke serious questions about the future of AI and humanity.