The video introduces Zamba 2, a new non-Transformers model by Zyer, which claims to outperform leading models in quality and performance, particularly for on-device applications, but the presenter expresses skepticism based on initial testing results. Despite high benchmark claims, Zamba 2 struggles with coding tasks and logical reasoning questions, leading to disappointment and a call for viewer feedback on its real-world performance.
In the video, the presenter introduces Zamba 2, a new non-Transformers model developed by the company Zyer, specifically the Zomat 27B. The model is claimed to outperform leading models like Mistral, Google’s Gemma, and Meta’s Llama 3 Series in terms of quality and performance, particularly for on-device and consumer GPU applications. The presenter expresses skepticism about the performance of non-Transformers models based on previous experiences but remains open to testing the new model. Zomat 2 is noted for its state-of-the-art benchmark performance and efficiency, achieving faster inference times and reduced memory usage compared to its competitors.
The video highlights the open-source nature of Zomat 2, which the presenter appreciates, emphasizing the importance of transparency in AI development. The presenter discusses various benchmarks where Zomat 2 reportedly excels, particularly in MMLU (Massive Multitask Language Understanding) tasks, showcasing its competitive edge against other models in its class. However, the presenter cautions that, being a smaller model, Zomat 2 may not perform as well as larger models, and the focus should be on its relative performance within its size category.
As the testing begins, the presenter uses Zomat 2’s inference endpoint to evaluate its capabilities, starting with a request to write the game Tetris in Python. The model’s output is deemed slow and lacking in formatting, which raises concerns about its usability. The presenter then tests the model with the Snake game, referencing previous models’ performances for comparison. Unfortunately, Zomat 2 struggles with both tasks, indicating potential limitations in its coding capabilities.
The presenter continues to challenge Zomat 2 with various logic and reasoning questions, such as envelope size restrictions and mathematical problems. The model fails to provide correct answers consistently, leading to frustration from the presenter. Despite the claims of high quality and efficiency, the results from the tests do not align with the expectations set by the benchmarks, further questioning the model’s practical performance.
In conclusion, the presenter expresses disappointment with Zomat 2’s performance, particularly in comparison to the benchmarks that suggested it would excel. The video ends with a call for viewer feedback on the discrepancies between benchmark results and real-world testing outcomes. The presenter invites viewers to engage in the comments and encourages them to like and subscribe for future content, leaving the audience with a sense of curiosity about the model’s true capabilities.