MLflow and the Machine Learning Lifecycle

Forty minutes presentation given at XebiCon'18

https://youtu.be/ikHI-ys-cYM

Transcript

1. MLflow and the Machine Learning Lifecycle 1 Giulia BIANCHI Fares OUESLATI Loïc DIVAD @XebiaFr
2. Giulia BIANCHI Data Scientist @XebiaFr @Giuliabianchl Fares OUESLATI ML Engineer @XebiaFr @fares_oueslati Loïc DIVAD Software Engineer @XebiaFr @LoicMDivad 2
3. 3 @Xebiconfr #Xebicon18 @Giuliabianchl
4. Raw data Model training Model serving Data processing 4 @Xebiconfr #Xebicon18 @Giuliabianchl
5. 5 tools tuning scale model exchange @Xebiconfr #Xebicon18 @Giuliabianchl
6. 6 @Xebiconfr #Xebicon18 @Giuliabianchl
7. 7 @Xebiconfr #Xebicon18 @Giuliabianchl
8. 8 @Xebiconfr #Xebicon18 @Giuliabianchl
9. 9 @Xebiconfr #Xebicon18 @Giuliabianchl
10. A simple Machine Learning example 10 @Xebiconfr #Xebicon18 @Giuliabianchl
11. • Dataset from scikit-learn • Predict diabetes progression given ◦ Age ◦ Sex ◦ Body Mass Index (BMI - indice de masse corporelle) ◦ Blood Pressure (BP - tension) ◦ Other blood measurements 11 @Xebiconfr #Xebicon18 @Giuliabianchl LARS paper
12. 12 @Xebiconfr #Xebicon18 @Giuliabianchl tuning
13. Before MLflow 13 @Xebiconfr #Xebicon18 @Giuliabianchl
14. 14 @Xebiconfr #Xebicon18 @Giuliabianchl
15. 15 @Xebiconfr #Xebicon18 @Giuliabianchl import mlflow import mlflow.sklearn with mlflow.start_run(): model = Model(param_1, param_2) model.fit(train_data, label) prediction = model.predict(test_data) (rmse, mae, r2) = eval_metrics(test_label, prediction) mlflow.log_param("param_1", param_1) mlflow.log_param("param_2", param_2) mlflow.log_metric("rmse", rmse) mlflow.log_metric("r2", r2) mlflow.log_metric("mae", mae) mlflow.sklearn.log_model(model, "model")
16. Tracking Server Local Apps Notebooks Cloud Jobs UI 16 @Xebiconfr #Xebicon18 @Giuliabianchl _ REST API
17. 17 @Xebiconfr #Xebicon18 @fares_oueslati scale
18. 18 @Xebiconfr #Xebicon18 @fares_oueslati
19. 19 @Xebiconfr #Xebicon18 @fares_oueslati # MLproject file name: My Project conda_env: my_env.yaml entry_points: main: parameters: data_file: <path> regularization: {type: float, default: 0.1} command: "python train.py -r {regularization} {data_file}" validate: parameters: data_file: <path> command: "python validate.py {data_file}"
20. 20 @Xebiconfr #Xebicon18 @fares_oueslati # Remote Run $ mlflow run git@github.com:mlflow/mlflow-example.git -P alpha=0.5 # Local Run $ mlflow run . -P alpha=0.5
21. 21 @Xebiconfr #Xebicon18 @fares_oueslati model exchange
22. 22 @Xebiconfr #Xebicon18 @fares_oueslati
23. Model Format flavor 2 flavor 1 Cloud Serving Tools Batch & Stream Scoring Inference Code 23 ML Frameworks @Xebiconfr #Xebicon18 @fares_oueslati
24. 24 @Xebiconfr #Xebicon18 @fares_oueslati # MLmodel file artifact_path: model flavors: python_function: data: model.pkl loader_module: mlflow.sklearn python_version: 2.7.15 sklearn: pickled_model: model.pkl sklearn_version: 0.19.2 run_id: 32357c10ae854113b0503e880e7433c1 utc_time_created: '2018-10-29 11:31:01.434417' Usable by any tool that can run Python (Docker, Spark etc.) Usable by tools that understand Sklearn model format
25. MLflow built-in flavors pyfunc rfunc h2o keras sklearn spark mleap pytorch tensorflow 25 @Xebiconfr #Xebicon18 @fares_oueslati
26. Generic & self-contained flavor that describes how to run the model as a lambda function pyfunc rfunc 26 h2o keras sklearn spark mleap pytorch tensorflow @Xebiconfr #Xebicon18 @fares_oueslati
27. MLflow built-in flavors pyfunc rfunc h2o keras sklearn spark mleap pytorch tensorflow 27 @Xebiconfr #Xebicon18 @fares_oueslati
28. package Azureml package Sagemaker package Sparkml 28 @Xebiconfr #Xebicon18 @fares_oueslati
29. MLflow API example: sklearn flavours mlflow flavor api \ save_model \ log_model \ load_model 29 @Xebiconfr #Xebicon18 @fares_oueslati
30. 30 @Xebiconfr #Xebicon18 @LoicMDivad
31. The MLflow project structure $ tree . ├── R ├── azureml ├── entities ├── java ├── models ├── projects ├── protos ├── sagemaker ├── server ├── store ├── tracking └── utils 31 @Xebiconfr #Xebicon18 @LoicMDivad
32. The three main modules are materialized by python packages $ tree . ├── R ├── azureml ├── entities ├── java ├── models ├── projects ├── protos ├── sagemaker ├── server ├── store ├── tracking └── utils The MLflow project structure 32 @Xebiconfr #Xebicon18 @LoicMDivad
33. $ tree . ├── R ├── azureml ├── entities ├── java ├── models ├── projects ├── protos ├── sagemaker ├── server ├── store ├── tracking └── utils The MLflow project structure Managed solutions from cloud providers have their own package The three main modules are materialized by python packages 33 @Xebiconfr #Xebicon18 @LoicMDivad
34. Other programming languages than python have their subproject $ tree . ├── R ├── azureml ├── entities ├── java ├── models ├── projects ├── protos ├── sagemaker ├── server ├── store ├── tracking └── utils Managed solutions from cloud providers have their own package The three main modules are materialized by python packages The MLflow project structure 34 @Xebiconfr #Xebicon18 @LoicMDivad
35. Other programming languages than python have their subproject $ tree . ├── R ├── azureml ├── entities ├── java ├── models ├── projects ├── protos ├── sagemaker ├── server ├── store ├── tracking └── utils Managed solutions from cloud providers have their own package The three main modules are materialized by python packages The MLflow project structure 35 @Xebiconfr #Xebicon18 @LoicMDivad
36. Experiment Run Param RunData Metric MLflowObject 36 @Xebiconfr #Xebicon18 @LoicMDivad
37. Experiment Run Param RunData Metric 37 @Xebiconfr #Xebicon18 @LoicMDivad artifact_path: model flavors: python_function: data: model.pkl loader_module: mlflow.sklearn sklearn: pickled_model: model.pkl sklearn_version: 0.19.1 run_id: cf5db2cc7c0d4074bbccd970d912e1c8 utc_time_created: '2018-07-28 15:49:49.055985' MLflowObject
38. Experiment Run Param RunData Metric 38 @Xebiconfr #Xebicon18 @LoicMDivad artifact_path: model flavors: python_function: data: model.pkl loader_module: mlflow.sklearn sklearn: pickled_model: model.pkl sklearn_version: 0.19.1 run_id: cf5db2cc7c0d4074bbccd970d912e1c8 utc_time_created: '2018-07-28 15:49:49.055985' MLflowObject
39. $ tree . ├── azureml ├── entities ├── ... ├── java └── store ├── abstract_store.py ├── artifact_repo.py ├── azure_blob_artifact_repo.py ├── dbfs_artifact_repo.py ├── file_store.py ├── gcs_artifact_repo.py ├── local_artifact_repo.py ├── rest_store.py ├── s3_artifact_repo.py └── sftp_artifact_repo.py 39 @Xebiconfr #Xebicon18 @LoicMDivad
40. Experiment Metadata • Runs • Parameters • Metrics ... Large artefacts • Datasets • Models • Images ... ArtifactRepository AbstractStore FileStore RestStore S3 GCS DBFS ... 40 @Xebiconfr #Xebicon18 @LoicMDivad
41. if artifact_uri.startswith("s3:/"): import S3ArtifactRepository elif artifact_uri.startswith("gs:/"): import GCSArtifactRepository elif artifact_uri.startswith("wasbs:/"): import AzureBlobArtifactRepository elif artifact_uri.startswith("sftp:/"): import SFTPArtifactRepository elif artifact_uri.startswith("dbfs:/"): import DbfsArtifactRepository else: import LocalArtifactRepository 41 @Xebiconfr #Xebicon18 @LoicMDivad
42. if artifact_uri.startswith("s3:/"): import S3ArtifactRepository elif artifact_uri.startswith("gs:/"): import GCSArtifactRepository elif artifact_uri.startswith("wasbs:/"): import AzureBlobArtifactRepository elif artifact_uri.startswith("sftp:/"): import SFTPArtifactRepository elif artifact_uri.startswith("dbfs:/"): import DbfsArtifactRepository else: import LocalArtifactRepository 42 from google.cloud import storage as gcs_storage # or from azure.storage.blob import BlockBlobService # requires # GOOGLE_APPLICATION_CREDENTIALS # AWS_SECRET_ACCESS_KEY # ETC ... @Xebiconfr #Xebicon18 @LoicMDivad
43. if artifact_uri.startswith("s3:/"): import S3ArtifactRepository elif artifact_uri.startswith("gs:/"): import GCSArtifactRepository elif artifact_uri.startswith("wasbs:/"): import AzureBlobArtifactRepository elif artifact_uri.startswith("sftp:/"): import SFTPArtifactRepository elif artifact_uri.startswith("dbfs:/"): import DbfsArtifactRepository else: import LocalArtifactRepository 43 from google.cloud import storage as gcs_storage # or from azure.storage.blob import BlockBlobService # requires # GOOGLE_APPLICATION_CREDENTIALS # AWS_SECRET_ACCESS_KEY # ETC ... @Xebiconfr #Xebicon18 @LoicMDivad
44. . └── EXPERIMENT-0 │ ├── RUN-4a59d6 │ ├── artifacts │ ├── meta.yaml │ ├── metrics │ └── params │ │ └── RUN-99d663 ├── artifacts ├── meta.yaml ├── metrics └── params Server Tracking Store 44 @Xebiconfr #Xebicon18 @LoicMDivad
45. . └── EXPERIMENT-0 │ ├── RUN-4a59d6 │ ├── artifacts │ ├── meta.yaml │ ├── metrics │ └── params │ │ └── RUN-99d663 ├── artifacts ├── meta.yaml ├── metrics └── params Server Tracking Store 45 $ mlflow server Launch a Flask server with four workers by default. It receives Byte Streams from client API and serializes the result in an artifact repo @Xebiconfr #Xebicon18 @LoicMDivad
46. 1. Give access to MLflow for all JVM users 2. A CRUD interface to MLflow available as Maven artifact 3. Come with MLeap flavour to save a Spark model in a SparkML format or MLeap format 46 @Xebiconfr #Xebicon18 @LoicMDivad MlflowClient client = new MlflowClient(); long expId = client.createExperiment(expName); // ... RunInfo runCreated = client.createRun(expId, sourceFile); client.logParam(runId, "min_samples_leaf", "2"); client.logParam(runId, "max_depth", "3"); // Log metrics client.logMetric(runId, "auc", 2.12F); client.logMetric(runId, "accuracy_score", 3.12F); client.logMetric(runId, "zero_one_loss", 4.12F); // Finished run client.setTerminated(runId, RunStatus.FINISHED);
47. MLflow v0.8.0 is still in alpha: ➢ Load data from diverse formats (e.g. CSV vs Parquet) ➢ Database backend tracking store ➢ Common hyperparameters tuning libraries integration ➢ Built-in Spark MLlib & PyTorch integration ➢ Support HDFS Artifact Repository ➢ ... 47 @Xebiconfr #Xebicon18 @LoicMDivad
48. ➢ MLflow allows to keep track of results and make them reproducible ◦ So you iterate faster and run through the machine learning life cycle ➢ The goal of MLflow is to make it easier to switch between tools ➢ MLflow is open source and open interface solution ➢ The Machine Learning platform is tool to unify Data Science and Engineering 48 @Xebiconfr #Xebicon18 @LoicMDivad