Create a custom FAQ chatbot with BERT

Chatbots have become an increasingly standard and valuable interface used by many organizations for a variety of purposes. They find numerous applications in various industries, such as providing personalized product recommendations to customers, offering round-the-clock customer support for query resolution, helping customers with bookings, and much more. This article explores the process of creating an FAQ chatbot specifically designed for customer interaction. FAQ chatbots address questions in a specific domain using predefined questions and appropriate answers. This type of chatbot relies on semantic query matching as its core mechanism.

learning objectives

• Learn the basics of the BERT model
• Understanding Elasticsearch and its use in a chatbot
• A mechanism for creating chatbots
• Indexing and querying in Elasticsearch

This article was published as part of the Data Science Blogathon.

What is BERT?

BERT, Bidirectional Encoder Representations from Transformers, is a large language model by Google in 2018. Unlike unidirectional models, BERT is a bidirectional model based on the Transformer architecture. It learns to understand the context of a word by considering the words before and after it in a sentence, which allows for a more comprehensive understanding.

One of the main challenges of BERT was that it did not achieve state-of-the-art performance for NLP tasks. A primary issue was that token-level embeddings could not be effectively used for textual similarity, resulting in poor performance when embedding sentences.

However, Sentence-BERT (SBERT) was developed to address this challenge. SBERT is based on a Siamese network that takes two sentences at a time and transforms them into token-level embeddings using the BERT model. It then uses a pooling layer to create sentence embeddings on each set of embeddings. In this article, we will use SBERT to embed sentences.

What is elastic search?

Elastic Search is an open source search and analytics system that is extremely powerful, highly scalable, and designed to process large amounts of data in real time. Develop an Apache Lucene library that provides full-text search capabilities. Elasticsearch is highly scalable as it provides a highly distributed network to scale across multiple nodes, providing high availability and fault tolerance. It also provides a flexible and powerful RESTful API that allows interaction with the search engine using HTTP requests. It supports various programming languages ​​and provides client libraries for easy application integration.

How to create a chatbot with BERT and Elastic Search?

This article will teach you how to create an FAQ chatbot pre-trained with BERT and Elasticsearch.

Step 1) Install the SBERT library

#install sentence transformers library
pip install sentence-transformers

Step 2) Create query embeddings

We will use the SBERT library to obtain embeddings for predefined queries. For each query, it generates a 768-dimensional universal array, equivalent to the size of a generic embedding of the BERT token level:

from sentence_transformers import SentenceTransformer

sent_transformer = SentenceTransformer("bert-base-nli-mean-tokens")

questions = [

    "How to improve your conversation skills? ",

    "Who decides the appointment of Governor in India? ",

    "What is the best way to earn money online?",

    "Who is the head of the Government in India?",

    "How do I improve my English speaking skills? "

]

ques_embedd = sent_transformer.encode(questions)

Step 3) Install the Elasticsearch library

pip install elasticsearch

Step 4) Create index in Elasticsearch

from elasticsearch import Elasticsearch


# defingin python client for elastic search
es_client = Elasticsearch("localhost:9200")

INDEX_NAME = "chat_bot_index"


#index dimensions for numpy array i.e. 768
dim_embedding = 768

def create_index() -> None:

    es_client.indices.delete(index=INDEX_NAME, ignore=404)

    es_client.indices.create(

        index=INDEX_NAME,

        ignore=400,

        body=

            "mappings": 

                "properties": 

                    "embedding": 

                        "type": "dense_vector",

                        "dims": dim_embedding,

                    ,

                    "question": 

                        "type": "text",

                    ,

                    "answer": 

                        "type": "text",

                    

                

            

        

    )



create_index()

The process of creating an index in Elasticsearch is very similar to the process of defining a schema in any database. In the above code, we have created an index called “chat_bot_index” which defines three fields i.e. “embed”, “query” and “answer” and their types like “dense_vector” for “embed” and “text”. ” for the other two.

Step 5) Index the Q&A in Elasticsearch

def indexing_q(qa_pairs: List[Dict[str, str]]) -> None:

  for pair in qa_pairs:
  
      ques = pair["question"]
  
      ans = pair["answer"]
  
      embedding = sent_transformer.encode(ques)[0].tolist()
  
          data = 
  
              "question": questi,
  
              "embedding": embedding,
  
              "answer": ans,
  
          
  
          es_client.index(
  
              index=INDEX_NAME,
  
              body=data
  
          )
  
 

qa_pairs = [

    "question": "How to improve your conversation skills? ",

    "answer": "Speak more",

,

    "question": "Who decides the appointment of Governor in India? ",

    "answer": "President of India",

,

    "question": "How can I improve my English speaking skills? ",

    "answer": "More practice",

]

indexing_q(qa_pairs)

In the above code, we identified question-answer pairs by embedding questions in the Elasticsearch database.

Step 6) Query from Elasticsearch

ENCODER_BOOST = 10

def query_question(question: str, top_n: int=10) -> List[dict]:
  embedding = sentence_transformer.encode(question)[0].tolist()
      es_result = es_client.search(
          index=INDEX_NAME,
          body={
              "from": 0,
              "size": top_n,
              "_source": ["question", "answer"],
              "query": 
                  "script_score": 
                      "query": 
                          "match": 
                              "question": question
                          
                      ,
                      "script": 
                          "source": """
                              (cosineSimilarity(params.query_vector, "embedding") + 1)
                              * params.encoder_boost + _score
                          """,
                          "params": 
                              "query_vector": embedding,
                              "encoder_boost": ENCODER_BOOST,
                          ,
                      ,
                  
              
          }
      )
      hits = es_result["hits"]["hits"]
      clean_result = []
      for hit in hits:
          clean_result.append(
              "question": item["_source"]["question"],
              "answer": item["_source"]["answer"],
              "score": item["_score"],
          )
  return clean_result

query_question("How to make my English fluent?")#import csv

We can modify the ES query to include a “script” field, which allows us to create a scoring function that calculates a cosine similarity score for embeddings. Combine this score with the overall ES BM25 matching score. To adjust the built-in cosine similarity weight, we can change a hyper parameter called “ENCODER_BOOST”.

conclusion

In this article, we explored using SBERT and Elasticsearch to build a chatbot. We considered creating a chatbot that would answer questions based on predefined question-answer pairs, taking into account the intent of the request.

Here are the main clues of our search:

1. Understand the importance of SBERT and Elasticsearch in the field of chatbot development, use their capabilities to improve the conversational experience.
2. Using SBERT to generate embeddings for questions allows you to gain a deeper understanding of their semantics and context.
3. Using Elasticsearch to build an index that efficiently stores and organizes query-answer pairs optimizes search and retrieval operations.
4. Demonstrating the query process in Elasticsearch, illustrating how effectively a chatbot gets the most relevant answers based on a user’s query.

Frequently Asked Questions

Media displayed in this article is not owned by Analytics Vidhya and is used at the discretion of the author.

connected