Want to easily deploy an open-source LLM? Anyscale’s Aviary project takes flight

Anyscale, the lead commercial vendor behind the open-source Ray machine learning (ML) scaling technology, is launching the new open-source Aviary project today to help simplify open-source large language model (LLM) deployment.

There are a growing number of open source LLMs, including Dolly, LLaMA, Carper AI and Amazon’s LightGPT, alongside dozens of others freely available on Hugging Face. But, simply having an LLM isn’t enough to make it useful for an organization — the model still needs to actually be deployed on infrastructure to enable inference and real world usage.

Getting an open-source LLM model deployed onto infrastructure has often been a bespoke process of trial and error as developers figure out the right compute resources and configuration parameters. It’s also not easy for developers to simply compare one model with another. These are some of the challenges Anyscale is looking to help solve with Aviary.

“Every week, new open-source models are released that people are trying out  that are pushing the state of the art,” Anyscale CEO Robert Nishihara told VentureBeat. “Where there hasn’t been as much progress and what has lagged behind in our view, is the open-source infrastructure for actually running those models.”

How Aviary works to ease open source LLM deployments

The Aviary project builds on top of the open-source Ray project with a set of optimizations and configurations to ease LLM deployment of open-source models.

Ray is already widely used by large organizations for model training and is the technology that OpenAI uses for its models including GPT-3 and GPT-4. The goal with Aviary is to automatically enable users of open source LLMs to deploy quickly with the right optimizations in place.

Nishihara explained that there are many different things that need to be configured on the infrastructure side, including model parallel inference across multiple GPUs, sharding and performance optimizations. The goal with Aviary is to have pre-configured defaults for essentially any open-source LLM on Hugging Face. Users don’t have to go through a time consuming process of figuring out infrastructure configuration on their own; Aviary handles all that for them.

Aviary also aims to help solve the challenge of model selection. With the growing number of models, it’s not easy for anyone to know the best model for a specific use case. Nishihara said that by making it easier to deploy open-source LLMs, Aviary is also making it easier for organizations to compare different LLMs. The comparisons enabled via Aviary include accuracy, latency and cost.

As new LLMs emerge, Aviary will enable them quickly

Aviary has been in private development at Anyscale for the last three months. Initially it took a bit of time to get the right configuration for any one open-source LLM, but what has become clear is that there are common patterns across all LLMs for deployment.

Nishihara said that when LightGPT became available, Aviary was able to add support for it in less than five minutes. He explained that there are a few different standard architectures that all open-source LLMs conform to in terms of how they handle model parallelism and other critical aspects of deployment.

“We don’t have to handle hundreds of special cases,” said Nishihara. “In fact, you just have to handle each of the standard model architectures and then all of the different LLMs fall into one of those categories.”

Overall, Nishihara expects that the number of open-source models is only going to grow and as a result, the problem of selecting models will only become harder for organizations.

“Our hope with Aviary is, with it being open source, anyone from the community who wants to will be able to just easily add new models,” he said. “That’ll make it easy for anyone using Aviary to just deploy those models without having to really do any extra work.”