Save 37% off PRO during our Black Friday Sale! »

MLflow and the Machine Learning Lifecycle

2d2dbdf5d060b4c1bb238f8f59185cfb?s=47 Giulia
November 20, 2018

MLflow and the Machine Learning Lifecycle

2d2dbdf5d060b4c1bb238f8f59185cfb?s=128

Giulia

November 20, 2018
Tweet

Transcript

  1. MLflow and the Machine Learning Lifecycle 1 Giulia BIANCHI Fares

    OUESLATI Loïc DIVAD @XebiaFr
  2. Giulia BIANCHI Data Scientist @XebiaFr @Giuliabianchl Fares OUESLATI ML Engineer

    @XebiaFr @fares_oueslati Loïc DIVAD Software Engineer @XebiaFr @LoicMDivad 2
  3. 3 @Xebiconfr #Xebicon18 @Giuliabianchl

  4. Raw data Model training Model serving Data processing 4 @Xebiconfr

    #Xebicon18 @Giuliabianchl
  5. 5 tools tuning scale model exchange @Xebiconfr #Xebicon18 @Giuliabianchl

  6. 6 @Xebiconfr #Xebicon18 @Giuliabianchl

  7. 7 @Xebiconfr #Xebicon18 @Giuliabianchl

  8. 8 @Xebiconfr #Xebicon18 @Giuliabianchl

  9. 9 @Xebiconfr #Xebicon18 @Giuliabianchl

  10. A simple Machine Learning example 10 @Xebiconfr #Xebicon18 @Giuliabianchl

  11. • Dataset from scikit-learn • Predict diabetes progression given ◦

    Age ◦ Sex ◦ Body Mass Index (BMI - indice de masse corporelle) ◦ Blood Pressure (BP - tension) ◦ Other blood measurements 11 @Xebiconfr #Xebicon18 @Giuliabianchl LARS paper
  12. 12 @Xebiconfr #Xebicon18 @Giuliabianchl tuning

  13. Before MLflow 13 @Xebiconfr #Xebicon18 @Giuliabianchl

  14. 14 @Xebiconfr #Xebicon18 @Giuliabianchl

  15. 15 @Xebiconfr #Xebicon18 @Giuliabianchl import mlflow import mlflow.sklearn with mlflow.start_run():

    model = Model(param_1, param_2) model.fit(train_data, label) prediction = model.predict(test_data) (rmse, mae, r2) = eval_metrics(test_label, prediction) mlflow.log_param("param_1", param_1) mlflow.log_param("param_2", param_2) mlflow.log_metric("rmse", rmse) mlflow.log_metric("r2", r2) mlflow.log_metric("mae", mae) mlflow.sklearn.log_model(model, "model")
  16. Tracking Server Local Apps Notebooks Cloud Jobs UI 16 @Xebiconfr

    #Xebicon18 @Giuliabianchl _ REST API
  17. 17 @Xebiconfr #Xebicon18 @fares_oueslati scale

  18. 18 @Xebiconfr #Xebicon18 @fares_oueslati

  19. 19 @Xebiconfr #Xebicon18 @fares_oueslati # MLproject file name: My Project

    conda_env: my_env.yaml entry_points: main: parameters: data_file: <path> regularization: {type: float, default: 0.1} command: "python train.py -r {regularization} {data_file}" validate: parameters: data_file: <path> command: "python validate.py {data_file}"
  20. 20 @Xebiconfr #Xebicon18 @fares_oueslati # Remote Run $ mlflow run

    git@github.com:mlflow/mlflow-example.git -P alpha=0.5 # Local Run $ mlflow run . -P alpha=0.5
  21. 21 @Xebiconfr #Xebicon18 @fares_oueslati model exchange

  22. 22 @Xebiconfr #Xebicon18 @fares_oueslati

  23. Model Format flavor 2 flavor 1 Cloud Serving Tools Batch

    & Stream Scoring Inference Code 23 ML Frameworks @Xebiconfr #Xebicon18 @fares_oueslati
  24. 24 @Xebiconfr #Xebicon18 @fares_oueslati # MLmodel file artifact_path: model flavors:

    python_function: data: model.pkl loader_module: mlflow.sklearn python_version: 2.7.15 sklearn: pickled_model: model.pkl sklearn_version: 0.19.2 run_id: 32357c10ae854113b0503e880e7433c1 utc_time_created: '2018-10-29 11:31:01.434417' Usable by any tool that can run Python (Docker, Spark etc.) Usable by tools that understand Sklearn model format
  25. MLflow built-in flavors pyfunc rfunc h2o keras sklearn spark mleap

    pytorch tensorflow 25 @Xebiconfr #Xebicon18 @fares_oueslati
  26. Generic & self-contained flavor that describes how to run the

    model as a lambda function pyfunc rfunc 26 h2o keras sklearn spark mleap pytorch tensorflow @Xebiconfr #Xebicon18 @fares_oueslati
  27. MLflow built-in flavors pyfunc rfunc h2o keras sklearn spark mleap

    pytorch tensorflow 27 @Xebiconfr #Xebicon18 @fares_oueslati
  28. package Azureml package Sagemaker package Sparkml 28 @Xebiconfr #Xebicon18 @fares_oueslati

  29. MLflow API example: sklearn flavours mlflow flavor api \ save_model

    \ log_model \ load_model 29 @Xebiconfr #Xebicon18 @fares_oueslati
  30. 30 @Xebiconfr #Xebicon18 @LoicMDivad

  31. The MLflow project structure $ tree . ├── R ├──

    azureml ├── entities ├── java ├── models ├── projects ├── protos ├── sagemaker ├── server ├── store ├── tracking └── utils 31 @Xebiconfr #Xebicon18 @LoicMDivad
  32. The three main modules are materialized by python packages $

    tree . ├── R ├── azureml ├── entities ├── java ├── models ├── projects ├── protos ├── sagemaker ├── server ├── store ├── tracking └── utils The MLflow project structure 32 @Xebiconfr #Xebicon18 @LoicMDivad
  33. $ tree . ├── R ├── azureml ├── entities ├──

    java ├── models ├── projects ├── protos ├── sagemaker ├── server ├── store ├── tracking └── utils The MLflow project structure Managed solutions from cloud providers have their own package The three main modules are materialized by python packages 33 @Xebiconfr #Xebicon18 @LoicMDivad
  34. Other programming languages than python have their subproject $ tree

    . ├── R ├── azureml ├── entities ├── java ├── models ├── projects ├── protos ├── sagemaker ├── server ├── store ├── tracking └── utils Managed solutions from cloud providers have their own package The three main modules are materialized by python packages The MLflow project structure 34 @Xebiconfr #Xebicon18 @LoicMDivad
  35. Other programming languages than python have their subproject $ tree

    . ├── R ├── azureml ├── entities ├── java ├── models ├── projects ├── protos ├── sagemaker ├── server ├── store ├── tracking └── utils Managed solutions from cloud providers have their own package The three main modules are materialized by python packages The MLflow project structure 35 @Xebiconfr #Xebicon18 @LoicMDivad
  36. Experiment Run Param RunData Metric MLflowObject 36 @Xebiconfr #Xebicon18 @LoicMDivad

  37. Experiment Run Param RunData Metric 37 @Xebiconfr #Xebicon18 @LoicMDivad artifact_path:

    model flavors: python_function: data: model.pkl loader_module: mlflow.sklearn sklearn: pickled_model: model.pkl sklearn_version: 0.19.1 run_id: cf5db2cc7c0d4074bbccd970d912e1c8 utc_time_created: '2018-07-28 15:49:49.055985' MLflowObject
  38. Experiment Run Param RunData Metric 38 @Xebiconfr #Xebicon18 @LoicMDivad artifact_path:

    model flavors: python_function: data: model.pkl loader_module: mlflow.sklearn sklearn: pickled_model: model.pkl sklearn_version: 0.19.1 run_id: cf5db2cc7c0d4074bbccd970d912e1c8 utc_time_created: '2018-07-28 15:49:49.055985' MLflowObject
  39. $ tree . ├── azureml ├── entities ├── ... ├──

    java └── store ├── abstract_store.py ├── artifact_repo.py ├── azure_blob_artifact_repo.py ├── dbfs_artifact_repo.py ├── file_store.py ├── gcs_artifact_repo.py ├── local_artifact_repo.py ├── rest_store.py ├── s3_artifact_repo.py └── sftp_artifact_repo.py 39 @Xebiconfr #Xebicon18 @LoicMDivad
  40. Experiment Metadata • Runs • Parameters • Metrics ... Large

    artefacts • Datasets • Models • Images ... ArtifactRepository AbstractStore FileStore RestStore S3 GCS DBFS ... 40 @Xebiconfr #Xebicon18 @LoicMDivad
  41. if artifact_uri.startswith("s3:/"): import S3ArtifactRepository elif artifact_uri.startswith("gs:/"): import GCSArtifactRepository elif artifact_uri.startswith("wasbs:/"):

    import AzureBlobArtifactRepository elif artifact_uri.startswith("sftp:/"): import SFTPArtifactRepository elif artifact_uri.startswith("dbfs:/"): import DbfsArtifactRepository else: import LocalArtifactRepository 41 @Xebiconfr #Xebicon18 @LoicMDivad
  42. if artifact_uri.startswith("s3:/"): import S3ArtifactRepository elif artifact_uri.startswith("gs:/"): import GCSArtifactRepository elif artifact_uri.startswith("wasbs:/"):

    import AzureBlobArtifactRepository elif artifact_uri.startswith("sftp:/"): import SFTPArtifactRepository elif artifact_uri.startswith("dbfs:/"): import DbfsArtifactRepository else: import LocalArtifactRepository 42 from google.cloud import storage as gcs_storage # or from azure.storage.blob import BlockBlobService # requires # GOOGLE_APPLICATION_CREDENTIALS # AWS_SECRET_ACCESS_KEY # ETC ... @Xebiconfr #Xebicon18 @LoicMDivad
  43. if artifact_uri.startswith("s3:/"): import S3ArtifactRepository elif artifact_uri.startswith("gs:/"): import GCSArtifactRepository elif artifact_uri.startswith("wasbs:/"):

    import AzureBlobArtifactRepository elif artifact_uri.startswith("sftp:/"): import SFTPArtifactRepository elif artifact_uri.startswith("dbfs:/"): import DbfsArtifactRepository else: import LocalArtifactRepository 43 from google.cloud import storage as gcs_storage # or from azure.storage.blob import BlockBlobService # requires # GOOGLE_APPLICATION_CREDENTIALS # AWS_SECRET_ACCESS_KEY # ETC ... @Xebiconfr #Xebicon18 @LoicMDivad
  44. . └── EXPERIMENT-0 │ ├── RUN-4a59d6 │ ├── artifacts │

    ├── meta.yaml │ ├── metrics │ └── params │ │ └── RUN-99d663 ├── artifacts ├── meta.yaml ├── metrics └── params Server Tracking Store 44 @Xebiconfr #Xebicon18 @LoicMDivad
  45. . └── EXPERIMENT-0 │ ├── RUN-4a59d6 │ ├── artifacts │

    ├── meta.yaml │ ├── metrics │ └── params │ │ └── RUN-99d663 ├── artifacts ├── meta.yaml ├── metrics └── params Server Tracking Store 45 $ mlflow server Launch a Flask server with four workers by default. It receives Byte Streams from client API and serializes the result in an artifact repo @Xebiconfr #Xebicon18 @LoicMDivad
  46. 1. Give access to MLflow for all JVM users 2.

    A CRUD interface to MLflow available as Maven artifact 3. Come with MLeap flavour to save a Spark model in a SparkML format or MLeap format 46 @Xebiconfr #Xebicon18 @LoicMDivad MlflowClient client = new MlflowClient(); long expId = client.createExperiment(expName); // ... RunInfo runCreated = client.createRun(expId, sourceFile); client.logParam(runId, "min_samples_leaf", "2"); client.logParam(runId, "max_depth", "3"); // Log metrics client.logMetric(runId, "auc", 2.12F); client.logMetric(runId, "accuracy_score", 3.12F); client.logMetric(runId, "zero_one_loss", 4.12F); // Finished run client.setTerminated(runId, RunStatus.FINISHED);
  47. MLflow v0.8.0 is still in alpha: ➢ Load data from

    diverse formats (e.g. CSV vs Parquet) ➢ Database backend tracking store ➢ Common hyperparameters tuning libraries integration ➢ Built-in Spark MLlib & PyTorch integration ➢ Support HDFS Artifact Repository ➢ ... 47 @Xebiconfr #Xebicon18 @LoicMDivad
  48. ➢ MLflow allows to keep track of results and make

    them reproducible ◦ So you iterate faster and run through the machine learning life cycle ➢ The goal of MLflow is to make it easier to switch between tools ➢ MLflow is open source and open interface solution ➢ The Machine Learning platform is tool to unify Data Science and Engineering 48 @Xebiconfr #Xebicon18 @LoicMDivad
  49. 49