SubQ, a Miami-based startup, claims to have developed a novel Sub Quadratic Sparse Attention (SSA) model enabling a 12 million token context window that is significantly faster and more cost-efficient than current Transformer Attention models. While the approach shows promise and has attracted interest, skepticism remains due to lack of independent verification, limited technical details, and benchmarks only demonstrated on smaller scales so far.
SubQ, a Miami-based startup, has recently made waves in the AI community by claiming to have developed a model with an unprecedented 12 million token context window. This breakthrough is said to be 52 times faster than Flash Attention and operates at just one-fifth of the cost of current models. Such claims naturally raise skepticism, but the details behind their approach offer an intriguing perspective on handling large context windows in AI models.
Currently, most AI models like GPT, Claude, and Gemini rely on Transformer Attention, where every token in the context compares itself with every other token. This process results in a quadratic increase in computational cost as the context window grows, making it prohibitively expensive to scale beyond a million tokens without significant compromises. This limitation has been a major bottleneck in expanding the capabilities of large language models.
SubQ proposes a novel method called Sub Quadratic Sparse Attention (SSA), which aims to bypass much of this computational overhead. Instead of exhaustively comparing every token, the model first interprets the query’s meaning and then selectively attends only to the relevant parts of the context. This targeted attention mechanism means that doubling the context window only doubles the cost, rather than quadrupling it, representing a significant efficiency improvement.
However, there is some skepticism about the novelty of SSA. Some experts speculate that it might be a sparse attention model fine-tuned on existing architectures like DeepSeek or Kimi, rather than a completely new design. Additionally, there is currently no independent verification or technical documentation available, and the impressive benchmarks have only been demonstrated on a 1 million token model, with no concrete results yet for the full 12 million token claim.
Despite these uncertainties, the team behind SubQ includes experienced professionals from Meta and Google, lending some credibility to their work. The concept itself is promising, and the company has generated considerable interest, evidenced by a waitlist for access. The community remains eager to see independent tests and real-world applications to validate SubQ’s claims, and updates are expected as more information becomes available.