📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva project trained a large-scale sovereign LLM from scratch, achieving impressive benchmarks but scoring only 4.9% on Italian school exams. This challenges assumptions about scale and language-specific investment in AI.
Italy’s Minerva-3B, a large-scale sovereign language model trained entirely from scratch on 2.5 trillion tokens, scored just 4.9% on the INVALSI Italian school-exam benchmark, highlighting challenges in achieving deep country-specific knowledge through scale alone.
The Minerva project, led by Sapienza University of Rome and supported by Italy’s national research and supercomputing infrastructure, built a 7-billion-parameter model using approximately 50% Italian data. Despite its impressive training scale and open data approach, Minerva-3B’s performance on actual academic tests was near chance, contradicting expectations that larger datasets and more parameters would yield better results in complex language tasks.
Researchers noted that while the dataset size and parameters are crucial, they may not be sufficient for deep country-specific knowledge. The results suggest that significant native-language investment at scale may be necessary to produce models with meaningful country-level expertise, raising questions about the efficacy of current sovereign-LLM strategies.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
AI language model training datasets
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
large scale language model books
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
AI research and development books
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
AI model training hardware
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for European Sovereign-LLM Strategies
The results from Minerva challenge the assumption that scaling up data and parameters alone can produce models with deep national language and knowledge capabilities. For European policymakers and AI strategists, this underscores the importance of targeted native-language investment and the limitations of current approaches, even when large-scale infrastructure and open data are available. The findings suggest that future efforts may need to include more focused, high-quality native-language data and possibly larger models to achieve desired country-specific expertise, impacting how Europe plans its AI sovereignty initiatives.
Background on European Sovereign-LLM Development
Italy’s Minerva project emerged as a structural counterpoint to Portugal’s AMÁLIA model, which layered European Portuguese onto a multilingual foundation. Unlike AMÁLIA, Minerva was built from scratch, training on 2.5 trillion tokens with roughly half Italian content, and published its weights and data openly. Despite these efforts, Minerva’s performance on Italian benchmarks, especially the INVALSI school exams, was unexpectedly low, raising questions about the effectiveness of scale versus targeted native-language training in sovereign-LLMs.
Previous European projects, including AMÁLIA, have debated whether continuation pre-training or training from scratch better serves national language needs. Italy’s approach aimed to demonstrate that larger-scale native-language training could produce more capable models, but the exam results reveal significant limitations, emphasizing the complexity of language-specific knowledge acquisition.
“The empirical results suggest the investment may still not be enough at the parameter scales these projects are operating at.”
— Thorsten Meyer
Unresolved Questions About Native-Language Model Efficacy
It remains unclear whether increasing model size further or refining training data quality could significantly improve Minerva’s performance on complex language tasks. The current results suggest scale alone may be insufficient, but the optimal approach for achieving deep country-specific expertise in sovereign models is still under investigation. The long-term impact of native-language investment levels has yet to be fully determined.
Next Steps for European Sovereign AI Development
The Minerva team continues to iterate on their methodology, including ongoing research into continual training and larger models. Policymakers and researchers are likely to reassess strategies for native-language data collection and model scaling, with a focus on achieving meaningful country-specific knowledge. Further empirical testing and model refinement are expected in the coming months, potentially informing future European AI sovereignty policies.
Key Questions
Why did Minerva score so low on Italian exams despite large-scale training?
The low score suggests that simply increasing dataset size and parameters may not be enough to develop deep country-specific knowledge, highlighting the importance of targeted native-language data and training strategies.
How does Minerva’s approach differ from other European sovereign LLMs?
Minerva was trained from scratch on a large, openly available dataset with roughly half Italian content, unlike models like AMÁLIA, which used continuation pre-training on multilingual foundations.
What are the implications for future European AI projects?
The results indicate that future efforts should consider larger native-language datasets and models, as scale alone may not produce the desired depth of country-specific knowledge.
Is Minerva an ongoing project?
Yes, the team continues to refine and expand Minerva, with ongoing research into training methodologies and larger models, aiming to improve performance on complex language tasks.
Source: ThorstenMeyerAI.com
