Kubeflow Pipelines

What is a Kubeflow Pipeline?

Developing and deploying a machine learning (ML) pipeline is an iterative process, involving several stages. This process requires cyclical adjustments and evaluations throughout its development. Pipelines manage these iterative process, using conceptual workflows.

The Kubeflow "Workflow"

In Kubernetes, a Pod represents a set of running containers on your cluster. (Often the relationship is 1 pod to 1 container). These pods have a defined “lifecycle.” For example, a pod is running in your cluster, then a critical fault occurs on the node where the pod is running, means all the pods on that node fail.

ML Workflow “Phases”

However, to make life considerably easier, you don't need to manage each Pod directly. Instead, you use kubernetes workload resources or resource controllers. There is a two phase process: the experimental phase wherein the model is discovered, and the production phase wherein the model is deployed and monitored.

PHASE I - EXPERIMENTAL PHASE

Images

In the experimental phase, the data and model are selected. The model is developed, tested, and updated iteratively based on initial assumptions to achieve the desired results. This involves: 1. Data analytics: Collecting and analyzing the necessary data to train the ML model using statistics, baseline models, or other analytics tools. 2. Selecting & Coding (TF, PyTorch, scikit-learn, or XGBoost) the initial model version (Jupyter Notebook & Pipelines) 3. Tuning the model’s hyper-parameters for efficient processing and accurate results (Katib)

PHASE II - PRODUCTION PHASE	Images
In the production phase, a system is deployed and continuously infers and monitors. The phases should contain steps that look like the following: 1. Serving the model for online prediction or batch mode running. (TF Serving) 2. Monitoring the model’s performance and incorporating the results into the tuning or retraining processes of the model.