Operationalizing Data Science: Bringing Method to the Magic

According to Gartner's Nick Heudecker, 85% of data science projects fail. This is a staggering ratio of failure that deserves consideration from your organization before undertaking your next data science initiative. In order to keep your team from falling into the chasm of failed data science three key ingredients are required for your next project: events, engineering, and teamwork.

Transcript

1. Operationalizing Data Science Bringing Method to the Magic
2. Data Science is the new black How many people are working in data science or on machine learning systems?
3. Data Science is the new black How many people wish they were working in data science or on machine learning systems?
4. Music recommendations “I see you enjoy Rod Stewart and Kenny G, perhaps you’d also like to hear some...
5. Music recommendations “I see you enjoy Rod Stewart and Kenny G, perhaps you’d also like to hear some...
6. User music affinity profile “user_19234”: { “genre”: { “rock”: { “affinity”: 0.89, “subgenres”: { “classic”: … … “progressive”: … . . .
7. Music recommendation rules . . else if metal > 0.3 && classical > 0.2 then (‘Apocalyptica’, ‘Inquisition Symphony’) . . . else if country > 0.23 && rock > 0.31) then if punk > fuzz then (‘The Knitters’, ‘Poor Little Critter..’) else (‘The Sadies’, ‘Darker Circles’) . . . “user_19234”: { “genre”: { “rock”: { “affinity”: 0.89, “subgenres”: { “classic”: … … “progressive”: … . . .
8. All set?
9. “We were too conservative. The failure rate is closer to 85%. And the problem isn’t technology.” Nick Heudecker @nheudecker
10. Outline What goes wrong? What actions can help mitigate common causes of failure? What can the reactive community can bring? 1 2 3
11. Machine Learning 101
12. Predictions
13. Predictions Classical approach: Huff the “spirit of the gods”
14. 20th Century Predictions: Expert Systems . . . else if metal > 0.3 && classical > 0.2 then (‘Apocalyptica’, ‘Inquisition Symphony’) . . . else if country > 0.23 && rock > 0.31 then if punk > fuzz then (‘The Knitters’, ‘Poor Little Critter..’) else (‘The Sadies’, ‘Darker Circles’) . . .
15. State of the Art: Machine Learning Math algorithms
16. Machine Learning: Ingredients • Data • Learning algorithms • Serving models
17. What Goes Wrong?
18. What goes wrong? • Objectives • Approach & Execution • Technical
19. Objectives Organizational level understanding • Wrong problem • Wrong solution • Wrong problem • Wrong solution
20. Translate a business problem into mathematics... Organizational level understanding • Wrong problem • Wrong solution
21. … and back. Organizational level understanding • Wrong problem • Wrong solution
22. Framing the problem Loan approvals: Which applications should we approve? Which applications should we deny?
23. The Monkey’s Paw Be very careful what you wish for.
24. “Produce an approval process that we deny any applicant that has a high chance of defaulting on their loan.” The Monkey’s Paw
25. What goes wrong applications map { applicant => deny(applicant) }
26. Setting up the problem If I could predict ... I would take the following action(s) … And would expect to observe a change in ...
27. Approach & Execution ❖ Lack of team communication and/or coordination ❖ Wrong mix of skill sets ❖ Misunderstanding or misapplying data ❖ Wrong model evaluation metric
28. Predicting Real-Estate Values Use various factors such as building qualities, infrastructure and interest rates to predict the value of real-estate.
29. What are we trying to build?
30. We need to sort out: What needs to be done? Who is doing it? What skills do they need? What artefacts are produced and handed off?
31. Data engineering
32. Feature engineering
33. ML Model Selection Type of problem: Regression, classification, etc. Predict a continuous ‘value’, e.g. property price. Predict a discrete category, e.g. approve / deny.
34. ML Model Selection Type of problem: Regression, classification, etc. Predict a continuous ‘value’, e.g. property value.
35. ML Model Selection Type of problem: Regression, classification, etc. What type of learning model (training algorithm)? (Linear, neural nets, trees/forests, etc.) This will determine the general ‘shape’ of a trained model.
36. ML Model Selection What type of training algorithm? Linear regression What general ‘shape’ does a trained model have? “Line of best fit through the data”
37. Training Data
38. Training models
39. Our machine learning algorithm will determine optimal values for m i and b from data (model training).
40. Evaluating trained models
41. Aside: Evaluating trained models Evaluation is the only practical way we have of knowing how well the model works (without going to production and waiting).
42. Aside: Evaluating trained models There are important, business relevant considerations here! E.g. cost of false positive (denied loan to good applicant) vs. false negative (approved loan for bad credit risk).
43. Serving models
44. Model serving is the process of using a trained model to serve predictions at speed and scale in production. From model training From request instance Nbhd Sq ft Yr Built
45. Training vs. Serving Models Linear regression
46. Training vs. Serving Models Linear regression y = 0.15*x + 5 Given values for m, x and b determine y
47. Linear regression Loss function Regularization Standardization m 1 = 0.127 m 2 = 0.341 m 3 = 1.97 b = 2.44 y = 0.127 * x 1 + 0.341 * x 2 + 1.97 * x 3 + 2.44 Complexity: Training vs. Serving Variable substitution Multiplication Addition
48. Gradient descent Ensembles Boosting Bagging Loss function Residuals if rock > 0.4 then if lowTempo > 0.6 … else ... Boolean expression Nested if-else Complexity: Training vs. Serving
49. Approach & Execution: Summary Lots of steps to the process. Lots of technical details and jargon.
50. Approach & Execution: Summary What is my piece of the process? What do I need to understand to accomplish it?
51. What Goes Right?
52. What do we need to get right? • Embracing events • Handoff and collaboration • Testing • DevOps, DataOps, and “closing the loop”
53. Ingredients for success • Raw materials ◦ Big data (data lakes, data warehouses), events, other data sources (databases, etc) • Science ◦ Exploration, hypothesis testing, statistical methods, machine learning • Engineering ◦ Execute on the science using raw materials to build a finished product
54. Embrace Events
55. Embrace Events In order to predict the future you must understand the past.
56. Embrace Events Events are interesting things that have already happened. Events are always in the past.
57. Embrace Events Events span all of history. An event can be 1ms ago or 10 years ago.
58. Embrace Events Events are the core of our data analytics system.
60. Align on the objectives and domain
61. Event Storming Capturing the key business events. Can map analytics events back to the business context.
62. Feature Extraction
63. Handoffs and collaboration
64. Who does what? Data science handoff: • a trained model (i.e. parameter values that have been learned) • how to extract the features needed by the model from the raw data • start by reviewing and handing off versioned “design docs” • once maturity is reached, automate handoffs Engineering next steps: • turn a trained model into efficient production-quality code • ensure efficient access to the data required by the trained model to make predictions • reactive machine learning is critical to ensuring SLAs are met (latency, availability, etc)
65. Training pipeline
66. Model serving pipeline
67. DevOps, DataOps, and closing the loop
68. Testing Trained ML model
69. Determine fit
70. Full Testing Coverage • Trained models • Data (drift) • Data models • Data pipeline
71. Version control • Learning algorithm configuration ◦ Hyperparameters, etc • Pipeline processes and data transformations
72. DataOps Goals • ML processes is fully version controlled and reproducible • Commiting changes kicks off tests to validate those changes, and triggers downstream processes • CI/CD for ML
73. Conclusion
74. Actions
75. Recap
76. Let’s chat! Kevin Webber, RedElastic Principal Consultant kevin.webber@redelastic.com Dana Harrington, RedElastic Chief Scientist dana.harrington@redelastic.com