Make Machine Learning Boring Again: Best Practices for Using Machine Learning in Businesses - Albuquerque Machine Learning Meetup (Online) - Aug 2020

Transcript

1. Make Machine Learning Boring Again: Best Practices for Using Machine Learning in Businesses Szilard Pafka, PhD Chief Scientist, Epoch Albuquerque Machine Learning Meetup (Online) Aug 2020
3. Disclaimer: I am not representing my employer (Epoch) in this talk I cannot confirm nor deny if Epoch is using any of the methods, tools, results etc. mentioned in this talk
9. y = f (x1, x2, ... , xn) Source: Hastie etal, ESL 2ed
10. y = f (x1, x2, ... , xn)
15. #1 Use the Right Algo
16. Source: Andrew Ng
34. *
35. #2 Use Open Source
41. in 2006 - cost was not a factor! - data.frame - [800] packages
49. #3 Simple > Complex
51. 10x
60. #4 Incorporate Domain Knowledge Do Feature Engineering (Still) Explore Your Data Clean Your Data
71. #5 Do Proper Validation Avoid: Overfitting, Data Leakage
86. #5+ Model Debugging Un-Black Boxing/Understanding, Interpretability, Fairness
87. Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission - Rich Caruana etal On one of the pneumonia datasets, the rule-based system learned the rule “HasAsthama(x) ⇒ LowerRisk(x)”, i.e., that patients who have a history of asthma have lower risk of dying from pneumonia than the general population patients with a history of asthma usually were admitted not only to the hospital but directly to the ICU (Intensive Care Unit). [...] the aggressive care received by asthmatic patients was so effective that it lowered their risk of dying from pneumonia compared to the general population models trained on the data incorrectly learn that asthma lowers risk, when in fact asthmatics have much higher risk (if not hospitalized) The logistic regression model also learned that having asthma lowered risk, but this could easily be corrected by changing the weight on the asthma feature from negative to positive (or to zero).
92. #6 Batch or Real-Time Scoring?
94. https://medium.com/@HarlanH/patterns-for-connecting-predictive-models-to-software-products-f9b6e923f02d
95. https://medium.com/@dvelsner/deploying-a-simple-machine-learning-model-in-a-modern-web-application-flask-angular-docker-a657db075280 your app
98. R/Python: - Slow(er) - Encoding of categ. variables
99. #7 Do Online Validation as Well
101. https://www.oreilly.com/ideas/evaluating-machine-learning-models/page/2/orientation
102. https://www.oreilly.com/ideas/evaluating-machine-learning-models/page/2/orientation
103. https://www.oreilly.com/ideas/evaluating-machine-learning-models/page/2/orientation https://www.slideshare.net/FaisalZakariaSiddiqi/netflix-recommendations-feature-engineering-with-time-travel
104. #8 Monitor Your Models
106. https://www.retentionscience.com/blog/automating-machine-learning-monitoring-rs-labs/
107. https://www.retentionscience.com/blog/automating-machine-learning-monitoring-rs-labs/
109. 20% 80% (my guess)
110. 20% 80% (my guess)
111. #9 Business Value Seek / Measure / Sell
117. #10 Make it Reproducible
127. #11 Use the Cloud (Virtual Servers)
128. ML training: lots of CPU cores lots of RAM limited time
129. ML training: lots of CPU cores lots of RAM limited time ML scoring: separated servers
130. #12 Don’t Use ML (cloud) services (MLaaS)
132. “ the people that know what they’re doing just use open source, and the people that don’t will not get anything to work, ever, even with APIs.” https://bradfordcross.com/five-ai-startup-predictions-for-2017/
133. #13 Use High-Level APIs but not GUIs
136. #14 Kaggle Doesn’t Matter (Mostly)
138. already pre-processed data less domain knowledge (or deliberately hidden) AUC 0.0001 increases "relevant" no business metric no actual deployment models too complex no online evaluation no monitoring data leakage
139. # 15 GPUs (Depends)
140. Aggregation 100M rows 1M groups Join 100M rows x 1M rows time [s] time [s]
141. Aggregation 100M rows 1M groups Join 100M rows x 1M rows time [s] time [s] “Motherfucka!”
143. #16 Tuning and Auto ML (Depends)
144. Ben Recht, Kevin Jamieson: http://www.argmin.net/2016/06/20/hypertuning/
145. https://arxiv.org/pdf/1907.00909.pdf
147. “There is no AutoML system which consistently outperforms all others. On some datasets, the performance differences can be significant, but on others the AutoML methods are only marginally better than a Random Forest. On 2 datasets, all frameworks perform worse than a Random Forest.”
148. Winner stability in data science competitions Test Set N=100K, Models M=1000
149. Winner stability in data science competitions Test Set N=100K, Models M=3000
150. Winner stability in data science competitions Test Set N=10K, Models M=1000
151. Winner stability in data science competitions Test Set N=10K, Models M=3000
152. Meta: Ignore the Hype
154. Is This AI?
158. How to Start?