Mistral Large 2: the LLM Focused on Multilinguality and Coding

Able to Rival GPT-4o, Claude 3.5 Sonnet and Llama3.1–405B

Mistral Large 2 logo

The AI landscape continues to evolve at a rapid pace, with new models regularly claiming the “state-of-the-art” title. After Llama3.1–405B, Mistral Large 2 (or Mistral-Large-Instruct-2407), brought by Mistral AI, is the latest contender: with its 123 billion parameter, it is the successor of Mistral Large, also called Mistral Large 1 (2402), and it aims to deliver exceptional performance across multiple domains, from language translation to complex code generation.

And the introduction ends here. More details in:

1. A Multilingual and Coding Powerhouse
2. How it Performs Overall
3. Embracing the Agentic Paradigm
4. The Mistral AI Research License
5. Limitations: A Candid Acknowledgment
6. Conclusion

A Multilingual and Coding Powerhouse

One of the most striking features of Mistral-Large-Instruct-2407 is its multilingual proficiency. Unlike many models that excel primarily in English, Mistral has been trained on a diverse dataset encompassing dozens of languages, including French, German, Spanish, Italian, Chinese, Japanese, Korean, Portuguese, Dutch, and Polish. This multilingual capability opens doors to a wide range of applications, from cross-language communication to global content creation.

Furthermore, Mistral boasts exceptional coding capabilities. Its training encompasses over 80 coding languages, including popular choices like Python, Java, C++, JavaScript, and Bash, as well as more specialized languages like Swift and Fortran. This makes it an ideal tool for developers seeking AI assistance in code generation, debugging, and documentation. To illustrate its prowess, let’s examine Mistral Large 2’s performance in code generation across various programming languages:

As we can see, Mistral Large 2 achieves an average accuracy of 74.4% across these languages, demonstrating a significant improvement over its predecessor, Mistral Large 1, and performing competitively with the larger Llama 3.1 405B and GPT-4o. It particularly excels in C++, Java, and TypeScript, indicating a strong grasp of these widely-used languages.

How it Performs Overall

Mistral Large 2 isn’t good only on coding… Let’s compare its performance against other leading models across established benchmarks to get a hint of its abilities in various domains, from general knowledge and reasoning to mathematical problem-solving.

So:

• MMLU: On the Massive Multitask Language Understanding (MMLU) benchmark, Mistral Large 2 achieves a score of 84.0%. This indicates strong general knowledge and reasoning capabilities, though it falls slightly short of the top performers.
• HumanEval: This benchmark evaluates the generation of functional Python code from natural language descriptions, and it rivals the best model of the moment: Claude 3.5 Sonnet.
• GSM8K: The Grade School Math 8K (GSM8K) benchmark focuses on mathematical problem-solving. Mistral Large 2 shines here, achieving a commendable 93% accuracy, demonstrating its proficiency in handling complex mathematical problems.
• MATH: Similarly, on the MATH benchmark (for mathematical problem-solving), Mistral Large 2 scores a respectable 71.5% when using Chain-of-Thought (CoT) prompting, placing it in a competitive position among leading models, while it reaches 70% without using CoT.

While not always the top scorer, Mistral Large 2 consistently demonstrates competitive performance across various benchmarks, particularly in code generation and mathematical reasoning. Also, its smaller size compared to models like Llama 3.1 405B makes it more accessible for research and deployment on less powerful hardware.

Embracing the Agentic Paradigm

Mistral-Large-Instruct-2407 is not just about raw power; it’s about redefining how we interact with AI. The model embraces an agentic-centric design, aligning with a burgeoning field of AI research that focuses on building models capable of interacting with their environment, making decisions, and taking actions to achieve specific goals. This paradigm shift moves beyond simple question-and-answer interactions towards a more dynamic and collaborative relationship between humans and AI.

Mistral embodies this concept through its native support for function calling and JSON output. Function calling allows Mistral to dynamically interact with external tools and services, effectively expanding its own capabilities. Imagine asking Mistral to “book a flight to Paris”. Instead of simply providing information about flights, an agentic model like Mistral could actually interact with a booking system, process your request, and present you with concrete options. JSON output, a structured data format, facilitates this integration with external systems, making Mistral a potent force in complex, real-world workflows.

The Mistral AI Research License

Mistral Large 2 is released under the Mistral AI Research License. This license permits research and non-commercial use, modification, and distribution, subject to specific conditions outlined in the agreement. While allowing for a wide range of exploration and innovation, the license also emphasizes proper attribution, responsible use, and the need for transparency in how the model is utilized.

Commercial use of Mistral Large 2 requires a separate commercial license, which can be requested directly from Mistral AI.

Limitations: A Candid Acknowledgment

While Mistral-Large-Instruct-2407 presents impressive capabilities, it’s essential to acknowledge its limitations. The current release lacks robust moderation mechanisms, which means the model may generate outputs that are inappropriate or harmful in certain contexts. This is a critical consideration, especially as AI models become increasingly integrated into our lives.

Mistral AI is transparent about this limitation and actively seeks community collaboration to address it. This open approach promotes a responsible AI development environment, working towards ensuring the safe and ethical deployment of powerful models like Mistral Large 2.

Conclusion

Concluding, Mistral-Large-Instruct-2407 is a nice surprise because its multilingual prowess, coding proficiency, agentic design, and strong performance on benchmarks like GSM8K and MATH make it a formidable contender in the LLM arena. It is downloadable from Hugging Face and, while limitations exist, it’s another good model to have around in this race for the best model ever (or at least of 2024!).

As always, text taken from my website, feel free to subscribe or leave a comment (good or bad).

This story is published on Generative AI. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories.

Subscribe to our newsletter and YouTube channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!