The video discusses the release and performance of the Qwen 3.8 27B model, a smaller yet powerful version of the massive 2.4 trillion parameter Qwen 3.8 Max. The presenter highlights that while the Max model is impressive, its size makes it impractical for most local users. The 27B variant, however, offers a strong balance of performance and accessibility, outperforming Meta’s Muse Glimmer and the earlier Qwen 3.6 27B model in various benchmarks, including vision and computer use tasks. The model scores highly on intelligence and agentic indexes, making it one of the best open models available for local deployment without requiring extreme hardware investments.
There are multiple versions and quantizations of the Qwen 3.8 27B model available, including Bfloat16, FP8, and 4-bit quantizations like NV FP4 from Unsloth. Additionally, various fine-tuned and “obliterated” versions exist, some aimed at removing content restrictions. The MLX community has also released versions optimized for Mac users. The presenter emphasizes the importance of choosing the right version and quantization based on your hardware and use case, as well as the significance of how the model is set up, particularly regarding the amount of “thinking” or reasoning tokens used during inference.
Testing on a high-end Nvidia RTX Pro 6000 GPU with 96GB VRAM, the presenter experimented with different model versions and reasoning settings. They found that turning on speculative decoding with multi-token prediction significantly improved token generation speed. The FP8 version from Qwen offered a good balance of speed (80-120 tokens per second) and quality, while the Unsloth 4-bit quantized model achieved around 120 tokens per second. Obliterated versions, though interesting, tended to get stuck in repetitive loops and were less reliable.
The presenter also highlighted SG Lang’s serving setup, which is optimized for Blackwell GPUs and uses a specific NV FP4 quantization combined with speculative decoding. This configuration achieved impressive speeds of up to 200 tokens per second on average, with peaks around 220 tokens per second, making it the fastest tested setup. The model could handle very large context windows (up to 262K tokens) efficiently, though the amount of reasoning tokens used greatly affected performance and output quality. Medium reasoning settings were found to be the best compromise between quality and token consumption.
In conclusion, the Qwen 3.8 27B model represents a significant advancement in open-source local AI models, offering high intelligence and fast inference on consumer-grade hardware. Users are encouraged to experiment with different model versions, quantizations, and reasoning token settings to find the optimal setup for their needs. The presenter is optimistic about future improvements, such as more efficient fine-tunes and faster serving techniques, and plans to explore multi-machine setups in future videos. Overall, Qwen 3.8 27B is positioned as a leading model for local AI applications.
Useful Links
