Chatbot Arena: The LLM Benchmark Platform

Image by author

We all know that large language models (LLMs) are taking over the world and there is a lot to do in such a short time.

To shake it up a bit more, Chatbot Arena is an LLM benchmark platform created by the Large Model Systems Organization (LMSYS Org). It is an open research organization founded by students and faculty from UC Berkeley.

Their common goal is to make large models more accessible to everyone, using a collaborative development approach using open datasets, models, systems and evaluation tools. The LMSYS team builds large-scale language models and makes them widely available in conjunction with the development of distributed systems to accelerate the learning and inference of LLMs.

LLM mark required

Due to the continued hype surrounding ChatGPT, there has been a rapid growth in open source LLMs that have been fully customized to meet specific guidelines. You have examples like Alpaca and Vicuna that are based on LLaMA and can help with user requests.

However, anything so large that it gets out of control makes it difficult for society to keep up with the constant new developments and be able to evaluate these models effectively. Benchmarking LLM assistantships can be a challenge due to possible open issues.

Therefore, human assessment using pairwise comparisons is required. Pairwise comparison is the process of comparing models in pairs to judge which model has better performance.

In Chatbot Arena, users can chat with two anonymous models side by side and give their opinion and vote on which model is better. After the user votes, the model’s name will be revealed. Users have the option to continue chatting with two models or start over with two new randomly selected anonymous models.

You have the option to chat with two anonymous models side by side or choose the models you want to chat with. Below is a screenshot of a conversation with two anonymous models, LLM in battle!

Image screenshot by author

The collected data is then calculated into Elo ratings and then placed into a leaderboard. The Elo rating system is a method used in games such as chess to calculate the relative skill levels of players. The difference in rating between two users acts as a predictor of the outcome of that particular match.

As of today, May 5, 2023, the Chatbot Arena leaderboard looks like this:

Image by Chatbot Arena

If you want to see how it’s done, you can take a look at the notebook and play with the voting data yourself.

What a great and fun idea, right?

The Chatbot Arena team invites the entire community to join them in their quest for LLM benchmarking by contributing your own models, as well as jumping into the Chatbot Arena to get your vote in on the anonymous models.

Visit the Arena to vote for which model you think is better, and if you want to test a specific model, you can follow this guide to help you add it to the Chatbot Arena.

So is Charbot Arena still to come? According to the team, they plan to work on:

• Adding more closed source models
• Adding more open source models
• Release of periodically updated leaderboards. For example, every month
• Use better selection algorithms, tournament mechanisms and service systems to support a larger number of models
• Provide a sophisticated ranking system for different types of tasks.

Play with Chatbot Arena and let us know what you think!

There is a niche is a data scientist, freelance technical writer and community manager at KDnuggets. He is particularly interested in data science career advice or tutorials and theory-based knowledge about data science. He also wants to explore different ways in which artificial intelligence is/can benefit from human longevity. An eager learner looking to expand his technical knowledge and writing skills while helping others.