Continuous Delivery for Machine Learning Systems Deploying ML Systems to Production safely and quickly in a sustainable way Adarsh Sha h Engineering Leader, Coach, Hands-on Architec t Independent Consultan t @shahadarsh 
 https://shahadarsh.com Deck: http://bit.ly/ml-dod-pl 

https://shahadarsh.com @shahadarsh Deck: http://bit.ly/ml-dod-pl Hidden Technical Debt in ML Systems From the paper Hidden Technical Debt in Machine Learning Systems 

https://shahadarsh.com @shahadarsh Deck: http://bit.ly/ml-dod-pl 1 0 1 0 1 0 1 0 1 Results Traditional Software Development Machine Learning Program Data { } 1 0 1 0 1 0 1 0 1 Desired Results Model Training Data { } Program { } 1 0 1 0 1 0 1 0 1 Live Data Training Prediction Results 

https://shahadarsh.com @shahadarsh Deck: http://bit.ly/ml-dod-pl Data Acquisition Data Preparation Model Development Training Prediction Accuracy Evaluation Data Management Experimentation Production Deployment Validation Monitoring / Alerting Accuracy not reached Retrain Data Drift Fix Accuracy reached 

shahadarsh.com @shahadarsh Deck: http://bit.ly/ml-dod-pl Challenges Unique to ML 

https://shahadarsh.com @shahadarsh Deck: http://bit.ly/ml-dod-pl #1: Data Management Data Location Large Datasets Security Compliance Data Quality Tracking Dataset 

https://shahadarsh.com @shahadarsh Deck: http://bit.ly/ml-dod-pl #2: Experimentation Code Quality Research & 
 Experimentation Tracking experiments Training Time 
 & Troubleshooting Infrastructure 
 Requirements Model Accuracy Evaluation 

https://shahadarsh.com @shahadarsh Deck: http://bit.ly/ml-dod-pl #3: Production Deployment Of f l ine/Online 
 Prediction Monitoring & Alerting 

https://shahadarsh.com @shahadarsh Deck: http://bit.ly/ml-dod-pl What is Continuous Delivery? Continuous Delivery is the ability to get changes of all types—including new features, con f i guration changes, bug f i xes and experiments—into production, or into the hands of users, safely and quickly in a sustainable way . - Jez Humble & Dave Farley 
 (Continuous Delivery Book Authors) 

https://shahadarsh.com @shahadarsh Deck: http://bit.ly/ml-dod-pl Continuous Delivery 

https://shahadarsh.com @shahadarsh Deck: http://bit.ly/ml-dod-pl Continuous Integration Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day . - Martin Fowler 

https://shahadarsh.com @shahadarsh Deck: http://bit.ly/ml-dod-pl Principles of Continuous Delivery ๏ Build quality i n ๏ Work in small batche s ๏ Computers perform repetitive tasks, people solve problem s ๏ Relentlessly pursue continuous improvement (Kaizen ) ๏ Everyone is responsible 

https://shahadarsh.com @shahadarsh Deck: http://bit.ly/ml-dod-pl Data pipeline Dat a Source A Dat a Source B Dat a Source C Data Acquisition A Data Validation
 A Data Preparation
 A Training 
 Dataset Versioned Training Process Testing Data Acquisition B Data Validation
 B Data Preparation
 B Data Acquisition C Data Validation
 C Data Preparation
 C Bias & Fairness — — Security 
 & Compliance 

https://shahadarsh.com @shahadarsh Deck: http://bit.ly/ml-dod-pl Static Analysis Unit Tests Training Code Linting etc. Artifact Repository Build Artifact Continuous Integration (Training Code) Dev Environment Validation Tests Merge to 
 Main Branch 

https://shahadarsh.com @shahadarsh Deck: http://bit.ly/ml-dod-pl Data Pipeline Continuous Integration 
 (Training Code) Con f i guration Training 
 Dataset Training Environment Accuracy Evaluation Monitoring/ Alerting Testing (Bias & Fairness) Model Trigger Log Aggregation Automated 
 Provisioning/De-provisioning Data Scientist Training 

https://shahadarsh.com @shahadarsh Deck: http://bit.ly/ml-dod-pl Static Analysis Unit Tests Application Code Linting, Security Scan etc. Artifact Repository Build Artifact Ephemeral Environment Integration Tests Tag as Tested Model Continuous Integration (Application Code) Training 

https://shahadarsh.com @shahadarsh Deck: http://bit.ly/ml-dod-pl Data Management Experimentation Production Deployment Data Pipeline Continuous Integration 
 (Training Code) Data Scientist Con f i guration Training Model Continuous Integration 
 (Application Code) Deployment Production Environment Smoke Tests Monitoring /Alerting Application 
 Developer Bringing it all together Training 
 Dataset 

https://shahadarsh.com @shahadarsh Deck: http://bit.ly/ml-dod-pl Machine Learning Roles ML Researcher ML Engineer Data Engineer MLOps Engineer 

https://shahadarsh.com @shahadarsh Deck: http://bit.ly/ml-dod-pl Team Structure Considerations Cross Functional Team Separate Data Science Team ML Platform Engineering Team 

https://shahadarsh.com @shahadarsh Deck: http://bit.ly/ml-dod-pl References • continuousdelivery.com • Dr. Deming’s 14 Points for Management • Challenges Deploying Machine Learning Models to Production • State of DevOps Report • martinfowler.com • Large image datasets: A pyrrhic win for computer vision? 

https://shahadarsh.com @shahadarsh Deck: http://bit.ly/ml-dod-pl Book Recommendations 

https://shahadarsh.com @shahadarsh Deck: http://bit.ly/ml-dod-pl Adarsh Sha h Engineering Leader, Coach, Hands-on Architec t Independent Consultan t @shahadarsh 
 https://shahadarsh.com