Machine Learning Pipelines in Production - ML Galway Meetup

Talk at the Machine Learning Galway Meetup
https://www.meetup.com/machine-learning-galway/events/266142136/

While most of us have done Machine Learning over well-known/clean/small datasets, there is a big difference when dealing with noisy/large data, which is used to train models and support near-real-time services available to customers. First, there are challenges collecting and cleaning data for training ML models. Then, once the models are trained and ready for deployment, there are monitoring tasks that go beyond DevOps and that can have a huge impact on the performance of products and services. In this talk, Emir will speak about how to standardise the process of data collection for ML, and how to deal with the challenges that appear after models are deployed in production.

Transcript

1. Emir Muñoz Machine Learning Pipelines in Production Mon, November 11, 2019
2. …But it can be done 2
3. • Senior ML Engineer and Team Lead at Genesys • CS Researcher, SW Developer & Engineer • Finishing a PhD @ NUIG in Machine Learning using Knowledge Graphs • Interested in ML algorithms and their applications in real-world problems About • Genesys is a leader for omnichannel CX & contact centre solutions • 10,000+ customers in over 100 countries • 5,935 employees • Genesys AI centre for excellence in Galway with 80+ people • … and growing --- we are hiring ;)
4. “It works in my local machine” “It was working 10 minutes ago” The gap From months to years!
5. Genesys confidential and proprietary information. Unauthorized disclosure is prohibited. Machine learning in production vs Jumping from Blackrock diving tower • Seems simple, but not everyone can do it, or needs to do it • You require time, courage, and practice (a lot) • You require training as well • Once you nailed it, you can do it with your eyes closed (not really :P) Photo by Joe Hyland, https://www.flickr.com/photos/36224551@N04/7267942282
6. • The term artificial intelligence (AI) is applied when a machine mimics "cognitive" functions that humans associate with other human minds, such as "learning" and "problem solving" [Wikipedia] o As humans, we can walk, drive cars, read and understand a text, create art, etc • Machine learning (ML) is a subset of AI that focuses on more narrow problems. ML is the study of algorithms and statistical models that give the ability to computer systems to learn and perform a specific task without using explicit instructions, relying on patterns and inference instead [Wikipedia] AI ≠ ML Marketing’s confusion Artificial Intelligence Machine Learning Deep Learning Focus of this talk
7. • What is ML? o "A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E.” --- Mitchell, T. (1997). Machine Learning. McGraw Hill. • What should I consider? o Data is messy, noisy, inconsistent, incomplete, etc o You need robustness: training and testing with clean data o Good pipelines: data processing, scheduled trainings, scalability o Collaboration with DevOps o Keep customers in the loop Machine learning in production You need to know what you are doing! What is needed to build a supervised learning model What is needed to build a machine learning product
8. You require many people in your organisation to do Machine Learning in production Tip: Do not underestimate any of these blocks (Sculley et al., Hidden Technical Debt in Machine Learning Systems, NIPS 2015) Machine learning in production
9. Some people started calling this MLOps Machine learning life cycle (Amershi et al., Software engineering for machine learning: a case study, ICSE (SEIP) 2019) Business case Data sources KPIs etc GDPR Anonymisation Data dictionary Joins etc Cardinality Schema Seasonality etc Black magic Enrichment Selection Scaling etc Algorithm Hyperparameters Constraints Distributed sys. etc Eval. metrics KPIs Plots etc Reproducibility Repository CI/CD etc Eval. Metrics Logging Reports A/B testing etc
10. • What does this feature mean? • From where came this feature? • Why is this feature a string/int? • Why there are so many null values? • Which version of the data is this? Data pipelines Ensure you feed quality data to your ML algorithms • We need o Professional Services or Customers in the loop o Data Engineers and data pipelines o Stable and scalable technologies Inputs Outputs
11. “A ML model is only as good as the data it is fed” Model training Dataset (features + labels) Train data (features + labels) Test data (features) + Model Train (fit) model Model evaluation Test data (labels) Model predictions
12. • The metrics depend on your type of model, e.g., classification, regression • Share the interpretation of your metrics with everyone Tip: Important to obtain and track your metrics Model evaluation
13. • Track your experiments: code, data, configurations, and results • Specially if you are doing hyperparameter tuning or model selection • Keep track of which models you push to production • They usually allow you to deploy models to production Model management Sacred
14. Photo by Emir Munoz J
15. • Machine learning models can fail silently! • They do not fail as traditional software • You should have a post-deployment monitoring • Define metrics that make sense for your product • You probably need A/B testing methodologies to prove business value The work after deployment You cannot just relax yet, stay alert! • Be proactive o Review your unit testing and code in lower environments o Always have a backup model or dummy method in case your fancy model fails o Run tests in your production models during off-pick hours • Check every assumption o Your model is not getting the correct number of features o A feature changed format, e.g., date_timestamp o Data doesn’t look like the one you saw during training
16. Genesys confidential and proprietary information. Unauthorized disclosure is prohibited. Trained using adults Tested with kids Different demographics
17. Trained using bright images Tested with different lighting
18. • Data may change since the moment you train your models • Covariate shift: ℙ"#$%&() ≠ ℙ+#,-() o Shift in the independent variables • Prior probability shift: ℙ"#$%& () ≠ ℙ+#,- () o Shift in the target variable (labels) • Concept shift: ℙ"#$%&(|) ≠ ℙ+#,-(|) o Shift in the relationship between the independent and the target variables Dataset Shift Also called bias by some people Formally, ℙ"#$%&(, ) ≠ ℙ+#,-(, ) (Dawid Kopczyk’s blog post) (Krawczyk and Cano, Online Ensemble Learning with Abstaining Classifiers for Drifting and Noisy Data Streams, Applied Soft Computing 2018)
19. Takeaways Data pipelines • Keep data dictionaries • Validate that data follows a given schema • Validate your data types and joins Model training • Do feature analysis • Ensure all features are available for prediction • Tune hyperparameters only when needed • Ensure you deal with data imbalance Dataset shift • There are different ways that things could go wrong and sideways • Monitor your models’ performance • Read about how to identify shifts in your data Model evaluation and management • Select models that benefit your products • Keep track of your experiments and models. Specially, for those you push to production
20. Thanks Ping me if you would like to join our team Emir Muñoz first.last [at] genesys.com • Remember that ML models are not like traditional software and can fail silently • Research in ML is evolving pretty fast, try keeping up to date by reading regularly • Attend conferences and events like ML Galway meetup • Test tools like mlflow and Amazon SageMaker to manage your models • Ensure you monitor your models and empirically demonstrate how they affect your business