Operationalizing Data Science: Bringing Method to the Magic

Operationalizing Data Science: Bringing Method to the Magic

According to Gartner's Nick Heudecker, 85% of data science projects fail. This is a staggering ratio of failure that deserves consideration from your organization before undertaking your next data science initiative. In order to keep your team from falling into the chasm of failed data science three key ingredients are required for your next project: events, engineering, and teamwork.

665a7ca82af87606f4fc83b3d94b5fd5?s=128

Kevin Webber

October 24, 2018
Tweet

Transcript

  1. Operationalizing Data Science Bringing Method to the Magic

  2. Data Science is the new black How many people are

    working in data science or on machine learning systems?
  3. Data Science is the new black How many people wish

    they were working in data science or on machine learning systems?
  4. Music recommendations “I see you enjoy Rod Stewart and Kenny

    G, perhaps you’d also like to hear some...
  5. Music recommendations “I see you enjoy Rod Stewart and Kenny

    G, perhaps you’d also like to hear some...
  6. User music affinity profile “user_19234”: { “genre”: { “rock”: {

    “affinity”: 0.89, “subgenres”: { “classic”: … … “progressive”: … . . .
  7. Music recommendation rules . . else if metal > 0.3

    && classical > 0.2 then (‘Apocalyptica’, ‘Inquisition Symphony’) . . . else if country > 0.23 && rock > 0.31) then if punk > fuzz then (‘The Knitters’, ‘Poor Little Critter..’) else (‘The Sadies’, ‘Darker Circles’) . . . “user_19234”: { “genre”: { “rock”: { “affinity”: 0.89, “subgenres”: { “classic”: … … “progressive”: … . . .
  8. All set?

  9. “We were too conservative. The failure rate is closer to

    85%. And the problem isn’t technology.” Nick Heudecker @nheudecker
  10. Outline What goes wrong? What actions can help mitigate common

    causes of failure? What can the reactive community can bring? 1 2 3
  11. Machine Learning 101

  12. Predictions

  13. Predictions Classical approach: Huff the “spirit of the gods”

  14. 20th Century Predictions: Expert Systems . . . else if

    metal > 0.3 && classical > 0.2 then (‘Apocalyptica’, ‘Inquisition Symphony’) . . . else if country > 0.23 && rock > 0.31 then if punk > fuzz then (‘The Knitters’, ‘Poor Little Critter..’) else (‘The Sadies’, ‘Darker Circles’) . . .
  15. State of the Art: Machine Learning Math algorithms

  16. Machine Learning: Ingredients • Data • Learning algorithms • Serving

    models
  17. What Goes Wrong?

  18. What goes wrong? • Objectives • Approach & Execution •

    Technical
  19. Objectives Organizational level understanding • Wrong problem • Wrong solution

    • Wrong problem • Wrong solution
  20. Translate a business problem into mathematics... Organizational level understanding •

    Wrong problem • Wrong solution
  21. … and back. Organizational level understanding • Wrong problem •

    Wrong solution
  22. Framing the problem Loan approvals: Which applications should we approve?

    Which applications should we deny?
  23. The Monkey’s Paw Be very careful what you wish for.

  24. “Produce an approval process that we deny any applicant that

    has a high chance of defaulting on their loan.” The Monkey’s Paw
  25. What goes wrong applications map { applicant => deny(applicant) }

  26. Setting up the problem If I could predict ... I

    would take the following action(s) … And would expect to observe a change in ...
  27. Approach & Execution ❖ Lack of team communication and/or coordination

    ❖ Wrong mix of skill sets ❖ Misunderstanding or misapplying data ❖ Wrong model evaluation metric
  28. Predicting Real-Estate Values Use various factors such as building qualities,

    infrastructure and interest rates to predict the value of real-estate.
  29. What are we trying to build?

  30. We need to sort out: What needs to be done?

    Who is doing it? What skills do they need? What artefacts are produced and handed off?
  31. Data engineering

  32. Feature engineering

  33. ML Model Selection Type of problem: Regression, classification, etc. Predict

    a continuous ‘value’, e.g. property price. Predict a discrete category, e.g. approve / deny.
  34. ML Model Selection Type of problem: Regression, classification, etc. Predict

    a continuous ‘value’, e.g. property value.
  35. ML Model Selection Type of problem: Regression, classification, etc. What

    type of learning model (training algorithm)? (Linear, neural nets, trees/forests, etc.) This will determine the general ‘shape’ of a trained model.
  36. ML Model Selection What type of training algorithm? Linear regression

    What general ‘shape’ does a trained model have? “Line of best fit through the data”
  37. Training Data

  38. Training models

  39. Our machine learning algorithm will determine optimal values for m

    i and b from data (model training).
  40. Evaluating trained models

  41. Aside: Evaluating trained models Evaluation is the only practical way

    we have of knowing how well the model works (without going to production and waiting).
  42. Aside: Evaluating trained models There are important, business relevant considerations

    here! E.g. cost of false positive (denied loan to good applicant) vs. false negative (approved loan for bad credit risk).
  43. Serving models

  44. Model serving is the process of using a trained model

    to serve predictions at speed and scale in production. From model training From request instance Nbhd Sq ft Yr Built
  45. Training vs. Serving Models Linear regression

  46. Training vs. Serving Models Linear regression y = 0.15*x +

    5 Given values for m, x and b determine y
  47. Linear regression Loss function Regularization Standardization m 1 = 0.127

    m 2 = 0.341 m 3 = 1.97 b = 2.44 y = 0.127 * x 1 + 0.341 * x 2 + 1.97 * x 3 + 2.44 Complexity: Training vs. Serving Variable substitution Multiplication Addition
  48. Gradient descent Ensembles Boosting Bagging Loss function Residuals if rock

    > 0.4 then if lowTempo > 0.6 … else ... Boolean expression Nested if-else Complexity: Training vs. Serving
  49. Approach & Execution: Summary Lots of steps to the process.

    Lots of technical details and jargon.
  50. Approach & Execution: Summary What is my piece of the

    process? What do I need to understand to accomplish it?
  51. What Goes Right?

  52. What do we need to get right? • Embracing events

    • Handoff and collaboration • Testing • DevOps, DataOps, and “closing the loop”
  53. Ingredients for success • Raw materials ◦ Big data (data

    lakes, data warehouses), events, other data sources (databases, etc) • Science ◦ Exploration, hypothesis testing, statistical methods, machine learning • Engineering ◦ Execute on the science using raw materials to build a finished product
  54. Embrace Events

  55. Embrace Events In order to predict the future you must

    understand the past.
  56. Embrace Events Events are interesting things that have already happened.

    Events are always in the past.
  57. Embrace Events Events span all of history. An event can

    be 1ms ago or 10 years ago.
  58. Embrace Events Events are the core of our data analytics

    system.
  59. None
  60. Align on the objectives and domain

  61. Event Storming Capturing the key business events. Can map analytics

    events back to the business context.
  62. Feature Extraction

  63. Handoffs and collaboration

  64. Who does what? Data science handoff: • a trained model

    (i.e. parameter values that have been learned) • how to extract the features needed by the model from the raw data • start by reviewing and handing off versioned “design docs” • once maturity is reached, automate handoffs Engineering next steps: • turn a trained model into efficient production-quality code • ensure efficient access to the data required by the trained model to make predictions • reactive machine learning is critical to ensuring SLAs are met (latency, availability, etc)
  65. Training pipeline

  66. Model serving pipeline

  67. DevOps, DataOps, and closing the loop

  68. Testing Trained ML model

  69. Determine fit

  70. Full Testing Coverage • Trained models • Data (drift) •

    Data models • Data pipeline
  71. Version control • Learning algorithm configuration ◦ Hyperparameters, etc •

    Pipeline processes and data transformations
  72. DataOps Goals • ML processes is fully version controlled and

    reproducible • Commiting changes kicks off tests to validate those changes, and triggers downstream processes • CI/CD for ML
  73. Conclusion

  74. Actions

  75. Recap

  76. Let’s chat! Kevin Webber, RedElastic Principal Consultant kevin.webber@redelastic.com Dana Harrington,

    RedElastic Chief Scientist dana.harrington@redelastic.com