Implementing large language models in production

Introduction

Large language models (LLM) are now widely used in various applications such as machine translation, chat bots, text summarization, sentiment analysis, advances in natural language processing (NLP). However, it is difficult to implement and manage these LLMs in real-world applications, which is where LLMOps comes in. LLMOps refers to the set of practices, tools, and processes used to develop, deploy, and manage LLMs in a production environment.

MLflow is an open source platform that provides a set of tools for tracking experiments, packaging code, and deploying models to production. MLflow’s centralized model registry simplifies model version management and allows easy sharing and collaborative access with team members, making it a popular choice for data scientists and machine learning engineers to streamline workflow and improve productivity.

learning objectives

• Understand the challenges of deploying and managing LLMs in a manufacturing environment.
• Learn how MLflow can be used to solve the challenges of deploying large language models in a production environment by implementing LLMOps.
• Explore support for popular large language model libraries such as Hugging Face transformers, OpenAI, and Lang Chain.
• Learn how to use MLflow for LLMOps with hands-on examples.

This article was published as part of the Data Science Blogathon.

Challenges in implementing and managing LLMs in a manufacturing environment

The following factors complicate the management and implementation of LLMs in a production environment:

1. Resource management: LLMs require a lot of resources including GPU, RAM and CPU to run properly. These resources can be expensive and difficult to manage.
2. Model performance: LLMs can be sensitive to changes in the input data and their performance can vary depending on the data distribution. Ensuring that a model performs well in a production environment can be challenging.
3. Model version: Updating LLM can be difficult, especially if you need to manage multiple versions of the model at the same time. Keeping track of model versions and ensuring they are deployed correctly can be time-consuming.
4. Infrastructure: Configuring the infrastructure to host LLMs can be challenging, especially if multiple models need to be managed simultaneously.

How to use MLflow for LLMOps?

MLflow is an open source platform for machine learning lifecycle management. It provides a set of tools and APIs for managing experiments, packaging code, and deploying models. MLflow can be used to deploy and manage LLMs in a production environment with the following steps:

1. Create an MLflow project: The MLflow project is a packaged version of a machine learning application. You can create an MLflow project by defining the dependencies, code, and configuration required to run your LLM.
2. Train and register for your LLM: You can use TensorFlow, PyTorch or Keras to prepare your LLM. Once you’ve prepared your model, you can import model artifacts into MLflow using the MLflow APIs. If you are using a pre-trained model, you can skip the training step.
3. Package your LLM: After entering model artifacts, you can package them using MLflow commands. MLflow can generate a Python package that includes the model artifacts, dependencies, and configuration needed to run your LLM.
4. Post your LLM: You can deploy your LLM using Kubernetes, Docker, or AWS Lambda. You can use the MLflow APIs to load your LLM and run the predictions.

Support for huggable face transformers in MLflow

It is a popular open source library for building natural language processing models. These models are easy to deploy and manage in a production environment due to the built-in support of MLflow. To use Hugging Face transformers with MLflow, follow these steps:

• Install MLflow and Transformers: Installation of Transformers and MLflow can be done using Pip.

!pip install transformers
!pip install mlflow

• Define your LLM: The transformers library can be used to define your LLM, as shown in the following Python code:

import transformers
import mlflow

chat_pipeline = transformers.pipeline(model="microsoft/DialoGPT-medium")

• Register your LLM: To log your LLM into MLflow, use the Python code snippet below:

with mlflow.start_run():
  model_info = mlflow.transformers.log_model(
    transformers_model=chat_pipeline,
    artifact_path="chatbot",
    input_example="Hi there!"
  )

• Load your LLM and make predictions from it:

# Load as interactive pyfunc
chatbot = mlflow.pyfunc.load_model(model_info.model_uri)
#make predictions
chatbot.predict("What is the best way to get to Antarctica?")
>>> 'I think you can get there by boat'
chatbot.predict("What kind of boat should I use?")
>>> 'A boat that can go to Antarctica.'

Unlock AI support in MLflow

Open AI is another popular platform for building LLMs. MLflow provides support for open AI models, making it easy to deploy and manage open AI models in production environments. Below are the steps to use open AI models with MLflow:

• Install MLflow and open AI: Pip can be used to install Open AI and MLflow.

!pip install openai
!pip install mlflow

• Define your LLM: As shown in the following code snippet, you can define your LLM using the Open AI API:

from typing import List
import openai
import mlflow

# Define a functional model with type annotations

def chat_completion(inputs: List[str]) -> List[str]:
    # Model signature is automatically constructed from
    # type annotations. The signature for this model
    # would look like this:
    # ----------
    # signature:
    #   inputs: ["type": "string"]
    #   outputs: ["type": "string"]
    # ----------

    outputs = []

    for input in inputs:
        completion = openai.ChatCompletion.create(
            model="gpt-3.5-turbo",
            messages=["role": "user", "content": "<prompt>"]
        )

        outputs.append(completion.choices[0].message.content)

    return outputs

• Register your LLM: You can access your LLM MLflow using the following code snippet:

# Log the model
mlflow.pyfunc.log_model(
    artifact_path="model",
    python_model=chat_completion,
    pip_requirements=["openai"],
)

Lang Chain support in MLflow

Lang Chain is a platform for building LLMs using a modular approach. MLflow provides support for Lang Chain models, making it easy to deploy and manage Lang Chain models in a production environment. To use Lang Chain models with MLflow, you can follow these steps:

• Install MLflow and Lang Chain: You can install MLflow and Lang Chain using pip.

!pip install langchain
!pip install mlflow

• Define your LLM: The following code snippet shows how to define your LLM using the Lang Chain API:

from langchain import PromptTemplate, HuggingFaceHub, LLMChain

template = """Translate everything you see after this into French:

input"""

prompt = PromptTemplate(template=template, input_variables=["input"])

llm_chain = LLMChain(
    prompt=prompt,
    llm=HuggingFaceHub(
        repo_id="google/flan-t5-small",
        model_kwargs="temperature":0, "max_length":64
    ),
)

• Register your LLM: You can use the following code snippet to log into your LLM MLflow:

mlflow.langchain.log_model(
    lc_model=llm_chain,
    artifact_path="model",
    registered_model_name="english-to-french-chain-gpt-3.5-turbo-1"
)

• Load the model: You can upload your LLM using the code below.

#Load the LangChain model

import mlflow.pyfunc

english_to_french_udf = mlflow.pyfunc.spark_udf(
    spark=spark,
    model_uri="models:/english-to-french-chain-gpt-3.5-turbo-1/1",
    result_type="string"
)
english_df = spark.createDataFrame([("What is MLflow?",)], ["english_text"])

french_translated_df = english_df.withColumn(
    "french_text",
    english_to_french_udf("english_text")
) 

conclusion

Deploying and managing LLMs in a production environment can be challenging due to resource management, model performance, model versioning, and infrastructure issues. LLMs are easy to deploy and administer in a production setting using MLflow’s tools and APIs to manage the model lifecycle. In this blog, we discussed how to use MLflow to deploy and manage LLMs in a production environment, along with support for Hugging Face transformers, Open AI, and Lang Chain models. Collaboration between data scientists, engineers, and other stakeholders in the machine learning lifecycle can be improved using MLflow.

Some of the main medicines are as follows:

1. MLflow deploys and manages LLMs in a production environment.
2. Hugging Face transformers, Open AI and Lang Chain models are supported in MLflow.
3. Resource management, model execution, model versions, and infrastructure issues can be challenging when deploying and managing LLMs in a production environment, but MLflow provides a set of tools and APIs to overcome these challenges.
4. MLflow provides a centralized location for tracking experiments, versioning models, and packaging and deploying models.
5. MLflow integrates with existing workflows for ease of use.

Media displayed in this article is not owned by Analytics Vidhya and is used at the discretion of the author.

connected