In a groundbreaking development, Mixtral 8x7B, the first open-source Mixture of Experts (MoE) large model, has emerged as a formidable competitor, potentially surpassing the prowess of Llama 2 70B and GPT-3.5. The revelation comes with an official evaluation report showcasing its capabilities, sparking fervent discussions across social media platforms.
Unveiling Mixtral 8x7B: Features and Achievements
With the help of a sparse model, Mixtral 8x7B sets itself apart from Llama 2 70B by attaining six times faster inference speed across multiple benchmarks. With eight different feedforward blocks in the Transformer, the sparse mixed expert network – which functions as a decoder-only model – has an exceptional design and a parameter size of 46.7B.
Interestingly, Mistral AI’s valuation increased to over $2 billion in just six months after it was founded, demonstrating the growing importance of large MoE models in the AI landscape. In the meantime, the company’s unconventional approach – releasing the model for download before making an official announcement has drawn attention and challenged the conventional model release patterns.
Mixtral’s impressive performance in handling diverse tasks, including language understanding and advanced mathematics, has been noted. With small to larger cup sizes available, the pricing structure provides flexibility for customers with varying needs and financial constraints. Mistral AI attracts enthusiasts who want to test Mixtral’s performance on different platforms by offering model download services for local deployment. Different users claim different speeds; on various hardware setups, some can obtain 52 tokens per second.
Key Features of Mixtral 8x7B
Outperforms Llama 2 70B in various benchmarks, showcasing 6x faster inference.
Boasts a 32k context window.
Multilingual support for English, French, Italian, German, and Spanish.
Excellent performance in code generation.
Released alongside the Instruct version, optimized for specific tasks, achieving competitive scores.
OpenAI’s Reaction and Industry Buzz
In response to Mixtral’s release, Andrej Karpathy of OpenAI labeled it a medium cup, which begged the question of where OpenAI fits within the changing AI scene. However, Jim Fan, an AI expert at NVIDIA, praised Mistral AI, the brains behind Mixtral, for differentiating itself from a crowded field of nascent models. The open-source AI community has once again turned its focus to the MoE architecture, highlighting Mistral AI’s quick rise.
The rapid ascent of Mistral AI, coupled with the successful model reuse analysis by Princeton doctoral student Tianle Cai, signifies a paradigm shift in the open-source AI community. With Mixtral 8x7B challenging the dominance of established models, including GPT-3.5, netizens are speculating that “OpenAI has no moat,” signaling a new era in AI innovation and collaboration.
