The video showcases the testing of Mistral’s new flagship open-source model, the Mixtral 8x22b, which features 22 billion parameters and is an upgrade from the previous model. While the model shows promise in tasks like simple math and logic problems, it struggles with more complex scenarios that require nuanced understanding, indicating potential for improvement through further fine-tuning and data set variations.
In the video, the speaker tests Mistral’s new flagship open-source model, Mixtral 8x22b. This model is an 8 times 22 billion parameter model, an upgrade from the previous 8 times 7 billion parameter model. The announcement for the model was minimal, with just a torrent link provided by Mistral AI. The speaker mentions that the previous Mixol model was their favorite open-source model, creating excitement for testing the new version. The fine-tuned version of the model, called Kurasu Mixt 8x22b, is specifically designed for chat and is what the speaker tests in the video using Informatic DoAI for running the inference.
The speaker interacts with the model by providing it with tasks and questions to test its capabilities. They first test the model with writing a Python script to output numbers and creating the Snake Game in Python. The model successfully completes these tasks with minor issues, such as the snake being able to go through walls but ending the game if it goes into itself. The speaker then tests the model’s ability to provide uncensored responses, which it achieves after some pushing on sensitive topics.
Logic and reasoning tests are conducted, ranging from simple math problems to more complex scenarios like determining the number of killers in a room and predicting the location of a ball. The model performs well in some scenarios, such as simple math questions and transitive property reasoning, but struggles with the marble in the microwave problem and the concept of multiple people’s perspectives on an event. The model also displays some understanding of Json creation and provides detailed reasoning for certain tasks.
Overall, the Mixtral 8x22b model shows promise in various tasks, especially in logic and reasoning questions that align with its training data. While it excels in some areas like simple math and basic logic problems, it falls short in more complex scenarios that require nuanced understanding or real-world context. The speaker acknowledges the potential for improvement through further fine-tuning and data set variations, expressing a desire to continue testing and exploring the capabilities of this new model.