What are vector databases and why are they important for LLMs?

Image by editor

When you look through your timelines on Twitter, LinkedIn or news feeds – you probably see something about chatbots, LLMs and GPT. A lot of people talk about LLMs because new ones come out every week.

As we are currently living in the midst of an artificial intelligence revolution, it is important to understand that many of these new applications rely on vector embeddings. Let’s learn more about vector databases and why they are important for LLMs.

First, let’s define vector embedding. Vector embedding is a type of data representation that carries semantic information that helps AI systems better understand data and also be able to retain it in long-term memory. In learning anything new, the important elements are understanding and memorizing the topic.

embeddings Generated by artificial intelligence models such as LLMs that contain many features that make them difficult to manage. Embedding represents different dimensions of data to help AI models understand different relationships, patterns, and hidden structures.

Vector embedding using traditional scalar-based databases is a challenge because it cannot or cannot handle the scale and complexity of the data. With all the complexity of vector embedding, you can imagine the specialized database it would require. This is where vector databases come into play.

Vector databases Offers optimized storage and query capabilities for a unique vector embedding structure. They provide easy search, high performance, scalability, and data retrieval by comparing values ​​and finding similarities between values.

Sounds great, right? There is a solution to deal with the complex structure of vector embeddings. Yes, but no. Vector databases are very difficult to implement.

Until now, vector databases were used only by tech giants who had the ability not only to develop them, but also to manage them. Vector databases are expensive, so ensuring they are properly calibrated is important to ensure high performance.

How do vector databases work?

Now that we know a little bit about vector embeddings and databases, let’s take a look at how it works.

Image by author

Let’s start with a simple example of interacting with an LLM like ChatGPT. The model has a large amount of data with a lot of content and they provide the ChatGPT application.

So let’s go through the steps.

1. As a user, you enter your question in the application.
2. Your query is then fed into the embedding model, which creates vector embeddings based on the content we want to index.
3. The vector embedding is then transferred to the vector database with respect to the content from which it was embedded.
4. A vector database produces output and sends it to the user as a result of a query.

As the user continues to make requests, it will go through the same embedding model to create embeddings to query this database for similar vector embeddings. Similarity between vector embeddings is based on the original content in which the embedding was created.

Want to know more about how it works in vector databases? Let’s learn more.

Image by author

Traditional databases work with strings, numbers, etc. To store in rows and columns. When querying traditional databases, we read the rows that match our query. However, vector databases work with vectors, not strings, etc. Vector databases also use a similarity metric that is used to find the most similar vector to a query.

A vector database consists of different algorithms, all of which contribute to the nearest neighbor (ANN) search. This is done by means of hashing, graph-based search, or quantization, which are assembled into a pipeline to find neighbors of the requested vector.

The results are based on how close or approximate it is to the query, so the main elements that are considered are accuracy and speed. The slower the query output, the more accurate the result.

The three main stages that a vector database query goes through are:

1. Indexing

As explained in the example above, once the vector embedding is passed to the vector database, it then uses a variety of algorithms to map the vector embedding into the data structures for faster lookups.

2. Reading

After passing the search, the vector database compares the query vector to the indexed vectors, using a similarity metric to find the nearest neighbor.

3. Post processing

Depending on the vector database you are using, the vector database will process the final nearest neighbor to produce the final result for the query. Also re-ranking the nearest neighbors for future reference.

As we continue to grow AI and release new systems every week, the growth of vector databases plays a big role. Vector databases have allowed companies to interact more efficiently with accurate similarity searches, providing better and faster results to users.

So the next time you post a query to ChatGPT or Google Bard, think about the process it goes through to produce a result for your query.

There is a niche is a data scientist, freelance technical writer and community manager at KDnuggets. He is particularly interested in data science career advice or tutorials and theory-based knowledge about data science. He also wants to explore different ways in which artificial intelligence is/can benefit from human longevity. An eager learner looking to expand his technical knowledge and writing skills while helping others.