Airflow vs. Luigi vs. Argo vs. MLFlow vs. KubeFlow

https://www.datarevenue.com/en-blog/airflow-vs-luigi-vs-argo-vs-mlflow-vs-kubeflow

 

Airflow is the most popular solution, followed by Luigi. There are newer contenders too, and they’re all growing fast. (source)

Task orchestration tools and workflows

Recently there’s been an explosion of new tools for orchestrating task- and data workflows (sometimes referred to as “MLOps”). The quantity of these tools can make it hard to choose which ones to use and to understand how they overlap, so we decided to compare some of the most popular ones head to head. 

Overall Apache Airflow is both the most popular tool and also the one with the broadest range of features, but Luigi is a similar tool that’s simpler to get started with. Argo is the one teams often turn to when they’re already using Kubernetes, and Kubeflow and MLFlow serve more niche requirements related to deploying machine learning models and tracking experiments.

Before we dive into a detailed comparison, it’s useful to understand some broader concepts related to task orchestration.

What is task orchestration and why is it useful?

Smaller teams usually start out by managing tasks manually – such as cleaning data, training machine learning models, tracking results, and deploying the models to a production server. As the size of the team and the solution grows, so does the number of repetitive steps. It also becomes more important that these tasks are executed reliably.

The complex ways these tasks depend on each other also increases. When you start out, you might have a pipeline of tasks that needs to be run once a week, or once a month. These tasks need to be run in a specific order. As you grow, this pipeline becomes a network with dynamic branches. In certain cases, some tasks set off other tasks, and these might depend on several other tasks running first.

This network can be modelled as a DAG – a Directed Acyclic Graph, which models each task and the dependencies between them.

A pipeline is a limited DAG where each task has one upstream and one downstream dependency at most.

Workflow orchestration tools allow you to define DAGs by specifying all of your tasks and how they depend on each other. The tool then executes these tasks on schedule, in the correct order, retrying any that fail before running the next ones. It also monitors the progress and notifies your team when failures happen.

CI/CD tools such as Jenkins are commonly used to automatically test and deploy code, and there is a strong parallel between these tools and task orchestration tools – but there are important distinctions too. Even though in theory you can use these CI/CD tools to orchestrate dynamic, interlinked tasks, at a certain level of complexity you’ll find it easier to use more general tools like Apache Airflow instead.

Overall, the focus of any orchestration tool is ensuring centralized, repeatable, reproducible, and efficient workflows: a virtual command center for all of your automated tasks. With that context in mind, let’s see how some of the most popular workflow tools stack up.

 

Just tell me which one to use

You should probably use:

• Apache Airflowif you want the most full-featured, mature tool and you can dedicate time to learning how it works, setting it up, and maintaining it.
• Luigiif you need something with an easier learning curve than Airflow. It has fewer features, but it’s easier to get off the ground.
• Prefectif you want something that’s very familiar to Python programmers and stays out of your way as much as possible.
• Argo if you’re already deeply invested in the Kubernetes ecosystem and want to manage all of your tasks as pods, defining them in YAML instead of Python.
• KubeFlowif you want to use Kubernetes but still define your tasks with Python instead of YAML. You can also read about our experiences using Kubeflow and why we decided to drop it for our projects at Kubeflow: Not ready for production?
• MLFlow if you care more about tracking experiments or tracking and deploying models using MLFlow’s predefined patterns than about finding a tool that can adapt to your existing custom workflows.

Comparison table

For more Machine Learning Tips - Get our weekly newsletter

For a quick overview, we’ve compared the libraries when it comes to: 

• Maturity: based on the age of the project and the number of fixes and commits;
• Popularity: based on adoption and GitHub stars;
• Simplicity: based on ease of onboarding and adoption;
• Breadth: based on how specialized vs. how adaptable each project is;
• Language: based on the primary way you interact with the tool.

These are not rigorous or scientific benchmarks, but they’re intended to give you a quick overview of how the tools overlap and how they differ from each other. For more details, see the head-to-head comparison below.

Airflow, MLflow or Kubeflow for MLOps?

https://www.vietanh.dev/blog/2022-03-26-airflow-mlflow-or-kubeflow-for-mlops

II. Airflow + MLflow vs. Kubeflow

For full features of a MLOps system, Airflow needs to be combined with MLflow, while Kubeflow can almost provide all the features needed for a MLOps system. In this comparison, I also want to join Airflow with MLflow to build a MLOps stack. The other is Kubeflow. The figure beflow describe the features available in these stacks.

1. Workflow orchestration and data passing

Both Airflow and Kubeflow pipelines are workflow orchestration frameworks. However, Kubeflow pipelines is designed to support ML project better. For examples, it defines a standard way for passing data between machine learning operators using inputValue/outputValue or Kubeflow artifacts. In Airflow, we have XCOM as the key-value passing mechanism, however, we need to design and implement artifact passing ourself. Fortunately, when combining Airflow with MLflow, we can leverage artifact logging feature of MLflow for passing data between Airflow operators.

2. Experiment tracking and logging

We have experiment tracking, metric/artiface/model logging in MLflow and Kubeflow. However, the logging methods in mlfow packages seem to be easier to use than in Kubeflow. MLflow also provides ability to log and retrieve metric history (loss, accuracy, ...) during training, while I haven't found anything similar in Kubeflow.

3. Model registry and serving

MLflow has a great mechanism to register model easily by their name, while Kubeflow only support a complicated way for model register using ML metadata. However, model serving is supported better in Kubeflow with KServe or other addons.

3. Other ML specific features of Kubeflow

Kubeflow also provide us other features which we cannot find in Airflow + MLflow stacks, some of them are:

• Kubeflow Notebooks, where user can create and run JupyterLab in their Kubenetes cluster, instead of local machine.
• Katib, where user can run hyper parameter tuning or network architecture search at large scale.
• External addons to deal with multiple data science and platform problems.

4. Scalibility

Airflow + MLflow stack is very flexible with running environment, from Python packages, native docker on your machine to a Kubenetes cluster/cloud. In contrast, Kubeflow always needs Kubenetes for up and running. That may because Kubeflow is design at large scale in mind. For this reason, I think Airflow + MLflow stack fits better for small scale systems, where we don't want to setup and maintain Kubenetes clusters. Here we can write anything in docker/docker-compose and deploy with just one command. Airflow also can be scaled for Kubenetes cloud by using KubernetesPodOperator or Kubenetes Executor. In contrast, Kubeflow needs Kubenetes (on premise or managed cloud) to setup and run. In exchange, you will have a stable system with full features for machine learning.

5. How to choose between Airflow+Mlflow, and Kubeflow?

To sum up, I have some recommendations from my personal perspective:

• If your system needs to deal with multiple types of workflow, not just machine learning, Airflow may support you better. It is a mature workflow orchestration frameworks with support for a lot of operators besides machine learning.
• If you want a system with predesigned patterns for machine learning, and run at large scale on Kubenetes clusters, you may want to consider Kubeflow. Many ML specific components in Kubeflow can save your time implementing from scratch in Airflow.
• If you want to deploy MLOps in a small scale system (for example, a workstation, or a laptop), picking Airflow+MLflow stack can eliminate the need of setting up and running a Kubenetes system, and save more resources for the main tasks.

This blog post has briefly shown the differences between three popular MLOps frameworks (Airflow, MLflow and Kubeflow). Hope that it helps you in making decision between 2 stacks (Airflow + MLflow and Kubeflow). If you want to talk more about these frameworks or recommend others, please comment beflow. Thank you very much!

Deploying Large Language Models in Production: LLMOps with MLflow

https://datadance.ai/machine-learning/deploying-large-language-models-in-production-llmops-with-mlflow/

 

Introduction

Large Language Models (LLMs) are now widely used in a variety of applications, like machine translation, chat bots, text summarization , sentiment analysis , making advancements in the field of natural language processing (NLP). However, it is difficult to deploy and manage these LLMs in actual use, which is where LLMOps comes in. LLMOps refers to the set of practices, tools, and processes used to develop, deploy, and manage LLMs in production environments.

Contents IntroductionLearning ObjectivesChallenges in Deploying and Managing LLMs in Production EnvironmentsHow to Use MLflow for LLMOps?Hugging Face Transformers Support in MLflowOpen AI Support in MLflowLang Chain Support in MLflowConclusion

MLflow is an opensource platform that provides set of tools for tracking experiments, packaging code, and deploying models in production. Centralized model registry of MLflow simplifies the management of model versions and allows for easy sharing and collaborative access with the team members making it a popular choice for data scientists and Machine Learning engineers to streamline their workflow and improve productivity.

Learning Objectives

• Understand the challenges involved in deploying and managing LLMs in production environments.
• Learn how MLflow can be used to solve the challenges in deploying the Large language models in production environments there by implementing LLMOps.
• Explore the support for popular Large Language Model libraries such as – Hugging Face transformers, OpenAI, and Lang Chain.
• Learn how to use MLflow for LLMOps with practical examples.

This article was published as a part of the Data Science Blogathon.

Challenges in Deploying and Managing LLMs in Production Environments

The following factors make managing and deploying LLMs in a production setting difficult:

1. Resource Management:  LLMs need a lot of resources, including GPU, RAM, and CPU, to function properly. These resources can be expensive and difficult to manage.
2. Model Performance: LLMs can be sensitive to changes in the input data, and their performance can vary depending on the data distribution. Ensuring that the good model performance in a production environment can be challenging.
3. Model Versioning: Updating an LLM can be challenging, especially if you need to manage multiple versions of the model simultaneously. Keeping track of model versions and ensuring that they are deployed correctly can be time-consuming.
4. Infrastructure: Configuring the infrastructure for deploying LLMs can be challenging, especially if you need to manage multiple models simultaneously.

How to Use MLflow for LLMOps?

MLflow is an open-source platform for managing the machine learning lifecycle. It provides a set of tools and APIs for managing experiments, packaging code, and deploying models. MLflow can be used to deploy and manage LLMs in production environments by following the steps:

1. Create an MLflow project: An MLflow project is a packaged version of a machine learning application. You can create an MLflow project by defining the dependencies, code, and config required to run your LLM.
2. Train and Log your LLM: You can use TensorFlow, PyTorch, or Keras to train your LLM. Once you have trained your model, you can log the model artifacts to MLflow using the MLflow APIs.If you are using a pre trained model you can skip the training step.
3. Package your LLM: Once you have logged the model artifacts, you can package them using the MLflow commands. The MLflow can create a Python package that includes the model artifacts, dependencies, and config required to run your LLM.
4. Deploy your LLM: You can deploy your LLM using Kubernetes, Docker, or AWS Lambda. You can use the MLflow APIs to load your LLM and run predictions.