The video introduces the QWEN3 NEXT 80B A3B, a highly anticipated local AI model from Quinn, highlighting its innovative architecture and features. The model comes in two variants: “thinking” and “instruct,” catering to different user needs. The presenter discusses challenges faced when attempting to run the model locally, particularly with CPU offloading in VLLM, which currently does not work due to the model’s novel hybrid attention mechanisms like gated delta and gated attention. Despite these technical hurdles, the model promises significant advancements in scaling efficiency, context window size, and inference speed.
QWEN3 NEXT A3B boasts a 256k native context window, scalable up to one million tokens, and employs a high sparsity mixture of experts approach that drastically reduces FLOPS per token, enabling faster performance on less powerful hardware. Stability optimizations such as zero-centered and weight-decayed layer normalization help mitigate context rotation issues, making it suitable for long-context tasks. The model features 48 layers, 16 attention heads for queries, and 512 experts with only 10 active at a time, which contributes to its efficiency and speed.
Benchmark comparisons show that the QWEN3 NEXT 80B A3B performs competitively against larger models like Gemini Flash 235B, maintaining similar scores on downstream tasks while offering a tenfold increase in inference throughput for contexts over 32k tokens. The model is pre-trained on 15 trillion tokens with extensive post-training, likely focused on agentic and coding tasks, making it a promising option for local coding applications. The presenter recommends using the latest transformers library and VLLM builds to attempt running the model, though local deployment currently remains challenging.
For those interested in running QWEN3 NEXT A3B, cloud endpoints are available through providers like Hyperbolic and Novita, and Quinn’s own platform offers access via Quinn3 chat. The presenter encourages viewers to share insights or solutions regarding the local running issues. He also discusses the broader hardware landscape, noting rising DDR4 prices, the emergence of DDR5, and rumors about upcoming GPUs like the 5080 and 5070 Ti 24GB variants, which could impact the affordability and accessibility of hardware suitable for running such large models locally.
In conclusion, the QWEN3 NEXT 80B A3B represents a significant step forward in local AI model capabilities, bridging the gap between mid-range and ultra-large models. While technical challenges remain for local deployment, especially with CPU offloading and hybrid attention mechanisms, the model’s efficiency, large context window, and performance make it an exciting development. The presenter expresses optimism about future releases in the QWEN3 NEXT lineup and encourages community engagement to overcome current hurdles, while also providing insights into the evolving GPU market that will influence AI hardware choices in the near future.
