Silo AI unveils Poro, a new open source language model for Europe

Helsinki, Finland-based artificial intelligence startup Silo AI made waves this week by unveiling Poro, a new open source large language model (LLM) aimed at advancing multilingual AI capabilities for European languages.

Poro is the first model in a planned family of open source models intended to eventually cover all 24 official European Union languages. The models are being developed by SiloGen, Silo AI’s generative AI division established in late 2022 as well as University of Turku’s TurkuNLP research group.

“It is a digital sovereignty question where you want to ensure that there are models that are capturing the value base, the culture, the languages.” said Peter Sarlin, CEO of Silo AI, in an interview with VentureBeat. “Ultimately, it’s about value creation, ensuring that not only European, but any company out there can create value, can create proprietary models that create value that stays within Europe and stays within that organization.”

The 34.2 billion parameter Poro 34B model, named after the word for “reindeer” in Finnish, utilizes a BLOOM transformer architecture with ALiBi embeddings. It was trained on a partition of the 21 trillion token multilingual dataset covering English, Finnish, and programming languages like Python and Java.

Poro is being trained on LUMI, Europe’s fastest supercomputer located in Kajaani, Finland. LUMI provides access to 512 AMD Instinct MI250X GPUs capable of 74 petaflops of computing power.

According to Sarlin, Poro is designed to address the core challenge of training performant natural language models for lower-resourced European languages like Finnish. It does this by leveraging a cross-lingual training approach, allowing the model to take advantage of data from higher-resourced languages like English.

The model is the second major open source LLM to originate in Europe, following the record-funded French startup Mistral AI’s debut of Mistral 7B in late September 2023, and showcases the continent and region’s growing accomplishments in the quickly evolving generative AI field. It also highlights the increasing competition between different AI labs and companies.

Poro Research Checkpoints

As part of SiloGen’s commitment to transparency, Poro’s training progress will be documented through the Poro Research Checkpoints program.

“We’re going to be releasing checkpoints throughout model training, which is fairly new.” explained Sarlin. “There aren’t initiatives that have given such transparency to model training.”

The initial checkpoint for Poro 34B covers the first 30% of training. According to benchmarks released by Silo AI, Poro is achieving state-of-the-art results even at only 30% completion of its extensive training regimen.

On the widely used FIN-bench evaluation for the Finnish language, Poro outperforms existing monolingual Finnish AI models like FinGPT that were designed specifically for that task. 

“The model, already after 30% of training, is more performant on low resource languages than previous efforts,” noted Sarlin. By leveraging shared patterns across related tongues, Poro gains an edge for languages with less training data available.

Remarkably, Poro’s multilingual abilities have not come at the expense of English prowess. Testing on standard English evaluation sets reveals the model is “already outperforming existing models in terms of Finnish benchmarks, and are on par, or on path to be on par with English performance,” according to Sarlin. 

An open-source alternative to Big Tech

Sarlin believes open source models like Poro represent the future of AI, providing a transparent and ethical alternative to closed models from major tech companies.

“I personally believe that eventually there’s going to be a lot of open source alternatives out there,” said Sarlin. “The most secure way forward is to actually go open source and have full visibility into how these models have been built and what is the architecture.”

He added, “we’ve been putting quite a lot of effort into ensuring that both the data and the model side are regulatory compliant by design.”

Silo AI plans to continue releasing regular Poro checkpoints throughout the training process. The end goal is to create an entire family of open source models covering all European languages. If the initial results are any indication, Poro could soon be giving Big Tech a run for its money.

Partnering with the University of Turku

Poro represents an ongoing collaboration between Silo AI and the University of Turku in Finland. Researchers from the University’s TurkuNLP group have been pioneers in developing open source resources and models for the Finnish language. 

“My research group joined, a few professors joined and we basically scaled the company, revenue funded and bootstrapped. We’re quite different compared to many out there,” said Sarlin. “We’re a little bit more than 300 people, the majority have a PhD in AI related fields.”

This partnership combines Silo AI’s applied AI expertise and computational resources with the University’s leadership in multilingual language modeling research. According to Sarlin, it represents a model for how industry and academia can work together to advance AI capabilities, particularly for lower-resourced European languages.

Is Europe a future leader in open source AI?

The release of Poro suggests a new era of open collaboration and transparency in the field of natural language processing. Initiatives like Poro Research Checkpoints provide the entire community access to tools and insights previously locked up within tech giants.

“We operate with clients like Allianz. Rolls Royce is our client. We’re working with Honda. We’re working with Philips. We’re working with many large brands,” said Sarlin. “We’ve heard for quite a while that these larger enterprises are quite concerned about what eventual regulation will look like and which models they can use.”

If Poro delivers on its promise, it could democratize access to performant multilingual models – giving Europe a homegrown alternative to systems from US tech companies. While still early days, Poro represents an important milestone in bringing language AI out of proprietary silos and into the open.