Make Machine Learning Boring Again: Best Practices for Using Machine Learning in Businesses - Albuquerque Machine Learning Meetup (Online) - Aug 2020

Ce8e94cc306ba164175f693fb01aa8b0?s=47 szilard
August 16, 2020
21

Make Machine Learning Boring Again: Best Practices for Using Machine Learning in Businesses - Albuquerque Machine Learning Meetup (Online) - Aug 2020

Ce8e94cc306ba164175f693fb01aa8b0?s=128

szilard

August 16, 2020
Tweet

Transcript

  1. Make Machine Learning Boring Again: Best Practices for Using Machine

    Learning in Businesses Szilard Pafka, PhD Chief Scientist, Epoch Albuquerque Machine Learning Meetup (Online) Aug 2020
  2. None
  3. Disclaimer: I am not representing my employer (Epoch) in this

    talk I cannot confirm nor deny if Epoch is using any of the methods, tools, results etc. mentioned in this talk
  4. None
  5. None
  6. None
  7. None
  8. None
  9. y = f (x1, x2, ... , xn) Source: Hastie

    etal, ESL 2ed
  10. y = f (x1, x2, ... , xn)

  11. None
  12. None
  13. None
  14. None
  15. #1 Use the Right Algo

  16. Source: Andrew Ng

  17. None
  18. None
  19. None
  20. None
  21. None
  22. None
  23. None
  24. None
  25. None
  26. None
  27. None
  28. None
  29. None
  30. None
  31. None
  32. None
  33. None
  34. *

  35. #2 Use Open Source

  36. None
  37. None
  38. None
  39. None
  40. None
  41. in 2006 - cost was not a factor! - data.frame

    - [800] packages
  42. None
  43. None
  44. None
  45. None
  46. None
  47. None
  48. None
  49. #3 Simple > Complex

  50. None
  51. 10x

  52. None
  53. None
  54. None
  55. None
  56. None
  57. None
  58. None
  59. None
  60. #4 Incorporate Domain Knowledge Do Feature Engineering (Still) Explore Your

    Data Clean Your Data
  61. None
  62. None
  63. None
  64. None
  65. None
  66. None
  67. None
  68. None
  69. None
  70. None
  71. #5 Do Proper Validation Avoid: Overfitting, Data Leakage

  72. None
  73. None
  74. None
  75. None
  76. None
  77. None
  78. None
  79. None
  80. None
  81. None
  82. None
  83. None
  84. None
  85. None
  86. #5+ Model Debugging Un-Black Boxing/Understanding, Interpretability, Fairness

  87. Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day

    Readmission - Rich Caruana etal On one of the pneumonia datasets, the rule-based system learned the rule “HasAsthama(x) ⇒ LowerRisk(x)”, i.e., that patients who have a history of asthma have lower risk of dying from pneumonia than the general population patients with a history of asthma usually were admitted not only to the hospital but directly to the ICU (Intensive Care Unit). [...] the aggressive care received by asthmatic patients was so effective that it lowered their risk of dying from pneumonia compared to the general population models trained on the data incorrectly learn that asthma lowers risk, when in fact asthmatics have much higher risk (if not hospitalized) The logistic regression model also learned that having asthma lowered risk, but this could easily be corrected by changing the weight on the asthma feature from negative to positive (or to zero).
  88. None
  89. None
  90. None
  91. None
  92. #6 Batch or Real-Time Scoring?

  93. None
  94. https://medium.com/@HarlanH/patterns-for-connecting-predictive-models-to-software-products-f9b6e923f02d

  95. https://medium.com/@dvelsner/deploying-a-simple-machine-learning-model-in-a-modern-web-application-flask-angular-docker-a657db075280 your app

  96. None
  97. None
  98. R/Python: - Slow(er) - Encoding of categ. variables

  99. #7 Do Online Validation as Well

  100. None
  101. https://www.oreilly.com/ideas/evaluating-machine-learning-models/page/2/orientation

  102. https://www.oreilly.com/ideas/evaluating-machine-learning-models/page/2/orientation

  103. https://www.oreilly.com/ideas/evaluating-machine-learning-models/page/2/orientation https://www.slideshare.net/FaisalZakariaSiddiqi/netflix-recommendations-feature-engineering-with-time-travel

  104. #8 Monitor Your Models

  105. None
  106. https://www.retentionscience.com/blog/automating-machine-learning-monitoring-rs-labs/

  107. https://www.retentionscience.com/blog/automating-machine-learning-monitoring-rs-labs/

  108. None
  109. 20% 80% (my guess)

  110. 20% 80% (my guess)

  111. #9 Business Value Seek / Measure / Sell

  112. None
  113. None
  114. None
  115. None
  116. None
  117. #10 Make it Reproducible

  118. None
  119. None
  120. None
  121. None
  122. None
  123. None
  124. None
  125. None
  126. None
  127. #11 Use the Cloud (Virtual Servers)

  128. ML training: lots of CPU cores lots of RAM limited

    time
  129. ML training: lots of CPU cores lots of RAM limited

    time ML scoring: separated servers
  130. #12 Don’t Use ML (cloud) services (MLaaS)

  131. None
  132. “ the people that know what they’re doing just use

    open source, and the people that don’t will not get anything to work, ever, even with APIs.” https://bradfordcross.com/five-ai-startup-predictions-for-2017/
  133. #13 Use High-Level APIs but not GUIs

  134. None
  135. None
  136. #14 Kaggle Doesn’t Matter (Mostly)

  137. None
  138. already pre-processed data less domain knowledge (or deliberately hidden) AUC

    0.0001 increases "relevant" no business metric no actual deployment models too complex no online evaluation no monitoring data leakage
  139. # 15 GPUs (Depends)

  140. Aggregation 100M rows 1M groups Join 100M rows x 1M

    rows time [s] time [s]
  141. Aggregation 100M rows 1M groups Join 100M rows x 1M

    rows time [s] time [s] “Motherfucka!”
  142. None
  143. #16 Tuning and Auto ML (Depends)

  144. Ben Recht, Kevin Jamieson: http://www.argmin.net/2016/06/20/hypertuning/

  145. https://arxiv.org/pdf/1907.00909.pdf

  146. None
  147. “There is no AutoML system which consistently outperforms all others.

    On some datasets, the performance differences can be significant, but on others the AutoML methods are only marginally better than a Random Forest. On 2 datasets, all frameworks perform worse than a Random Forest.”
  148. Winner stability in data science competitions Test Set N=100K, Models

    M=1000
  149. Winner stability in data science competitions Test Set N=100K, Models

    M=3000
  150. Winner stability in data science competitions Test Set N=10K, Models

    M=1000
  151. Winner stability in data science competitions Test Set N=10K, Models

    M=3000
  152. Meta: Ignore the Hype

  153. None
  154. Is This AI?

  155. None
  156. None
  157. None
  158. How to Start?

  159. None
  160. None