Operationalizing Data Science: Bringing Method to the Magic

According to Gartner's Nick Heudecker, 85% of data science projects fail. This is a staggering ratio of failure that deserves consideration from your organization before undertaking your next data science initiative. In order to keep your team from falling into the chasm of failed data science three key ingredients are required for your next project: events, engineering, and teamwork.

Transcript

1. Operationalizing
   Data Science
   Bringing Method
   to the Magic
   

   View Slide

2. Data Science is the new black
   How many people are working in
   data science or on machine
   learning systems?
   

   View Slide

3. Data Science is the new black
   How many people wish they were
   working in data science or on
   machine learning systems?
   

   View Slide

4. Music recommendations
   “I see you enjoy Rod Stewart
   and Kenny G, perhaps you’d
   also like to hear some...
   

   View Slide

5. Music recommendations
   “I see you enjoy Rod Stewart
   and Kenny G, perhaps you’d
   also like to hear some...
   

   View Slide

6. User music
   affinity profile
   “user_19234”: {
   “genre”: {
   “rock”: {
   “affinity”: 0.89,
   “subgenres”: {
   “classic”: …
   …
   “progressive”: …
   .
   .
   .
   

   View Slide

7. Music recommendation rules
   .
   .
   else if metal > 0.3 && classical > 0.2 then
   (‘Apocalyptica’, ‘Inquisition Symphony’)
   .
   .
   .
   else if country > 0.23 && rock > 0.31) then
   if punk > fuzz then
   (‘The Knitters’, ‘Poor Little Critter..’)
   else
   (‘The Sadies’, ‘Darker Circles’)
   .
   .
   .
   “user_19234”: {
   “genre”: {
   “rock”: {
   “affinity”: 0.89,
   “subgenres”: {
   “classic”: …
   …
   “progressive”: …
   .
   .
   .
   

   View Slide

8. All set?
   

   View Slide

9. “We were too
   conservative.
   The failure rate is
   closer to 85%.
   And the problem
   isn’t technology.”
   Nick Heudecker
   @nheudecker
   

   View Slide

10. Outline
    What goes wrong?
    What actions can help
    mitigate common causes
    of failure?
    What can the
    reactive community
    can bring?
    1 2 3
    

    View Slide

11. Machine
    Learning 101
    

    View Slide

12. Predictions
    

    View Slide

13. Predictions
    Classical approach:
    Huff the “spirit of the gods”
    

    View Slide

14. 20th Century Predictions: Expert Systems
    .
    .
    .
    else if metal > 0.3 && classical > 0.2 then
    (‘Apocalyptica’, ‘Inquisition Symphony’)
    .
    .
    .
    else if country > 0.23 && rock > 0.31 then
    if punk > fuzz then
    (‘The Knitters’, ‘Poor Little Critter..’)
    else
    (‘The Sadies’, ‘Darker Circles’)
    .
    .
    .
    

    View Slide

15. State of the Art: Machine Learning
    Math algorithms
    

    View Slide

16. Machine Learning:
    Ingredients
    ● Data
    ● Learning algorithms
    ● Serving models
    

    View Slide

17. What Goes Wrong?
    

    View Slide

18. What goes wrong?
    ● Objectives
    ● Approach & Execution
    ● Technical
    

    View Slide

19. Objectives Organizational level understanding
    ● Wrong problem
    ● Wrong solution
    ● Wrong problem
    ● Wrong solution
    

    View Slide

20. Translate a
    business problem
    into mathematics...
    Organizational level understanding
    ● Wrong problem
    ● Wrong solution
    

    View Slide

21. … and back.
    Organizational level understanding
    ● Wrong problem
    ● Wrong solution
    

    View Slide

22. Framing the problem
    Loan approvals:
    Which applications
    should we approve?
    Which applications
    should we deny?
    

    View Slide

23. The Monkey’s Paw
    Be very careful what you wish for.
    

    View Slide

24. “Produce an approval process that we deny any applicant
    that has a high chance of defaulting on their loan.”
    The Monkey’s Paw
    

    View Slide

25. What goes wrong
    applications map { applicant =>
    deny(applicant)
    }
    

    View Slide

26. Setting up the problem
    If I could predict ...
    I would take the following action(s) …
    And would expect to observe a change in ...
    

    View Slide

27. Approach &
    Execution
    ❖ Lack of team communication
    and/or coordination
    ❖ Wrong mix of skill sets
    ❖ Misunderstanding or
    misapplying data
    ❖ Wrong model evaluation metric
    

    View Slide

28. Predicting
    Real-Estate Values
    Use various factors such as building
    qualities, infrastructure and interest
    rates to predict the value of
    real-estate.
    

    View Slide

29. What are
    we trying to
    build?
    

    View Slide

30. We need
    to sort out:
    What needs to be done?
    Who is doing it?
    What skills do they need?
    What artefacts are produced and
    handed off?
    

    View Slide

31. Data engineering
    

    View Slide

32. Feature engineering
    

    View Slide

33. ML Model Selection
    Type of problem: Regression, classification, etc.
    Predict a
    continuous ‘value’,
    e.g. property price.
    Predict a discrete
    category,
    e.g. approve / deny.
    

    View Slide

34. ML Model Selection
    Type of problem: Regression, classification, etc.
    Predict a
    continuous ‘value’,
    e.g. property value.
    

    View Slide

35. ML Model Selection
    Type of problem: Regression, classification, etc.
    What type of learning model (training algorithm)?
    (Linear, neural nets, trees/forests, etc.)
    This will determine the general ‘shape’ of a trained model.
    

    View Slide

36. ML Model Selection
    What type of training algorithm? Linear regression
    What general ‘shape’ does a trained model have?
    “Line of best fit
    through the data”
    

    View Slide

37. Training Data
    

    View Slide

38. Training models
    

    View Slide

39. Our machine learning algorithm will determine optimal
    values for m
    i
    and b from data (model training).
    

    View Slide

40. Evaluating
    trained
    models
    

    View Slide

41. Aside: Evaluating trained models
    Evaluation is the only practical way we have of knowing
    how well the model works (without going to production and
    waiting).
    

    View Slide

42. Aside: Evaluating trained models
    There are important, business relevant considerations here!
    E.g. cost of false positive (denied loan to good applicant)
    vs. false negative (approved loan for bad credit risk).
    

    View Slide

43. Serving models
    

    View Slide

44. Model serving is the process of using a trained model to
    serve predictions at speed and scale in production.
    From model training
    From request instance
    Nbhd
    Sq ft
    Yr Built
    

    View Slide

45. Training vs. Serving Models
    Linear regression
    

    View Slide

46. Training vs. Serving Models
    Linear regression
    y = 0.15*x + 5
    Given values for m, x and b
    determine y
    

    View Slide

47. Linear regression
    Loss function
    Regularization
    Standardization
    m
    1
    = 0.127
    m
    2
    = 0.341
    m
    3
    = 1.97
    b = 2.44
    y = 0.127 * x
    1
    +
    0.341 * x
    2
    +
    1.97 * x
    3
    +
    2.44
    Complexity:
    Training vs.
    Serving
    Variable substitution
    Multiplication
    Addition
    

    View Slide

48. Gradient descent
    Ensembles
    Boosting
    Bagging
    Loss function
    Residuals
    if rock > 0.4 then
    if lowTempo > 0.6
    …
    else
    ...
    Boolean expression
    Nested if-else
    Complexity:
    Training vs.
    Serving
    

    View Slide

49. Approach & Execution: Summary
    Lots of steps to the process.
    Lots of technical details and jargon.
    

    View Slide

50. Approach & Execution: Summary
    What is my piece of the process?
    What do I need to understand to accomplish it?
    

    View Slide

51. What Goes Right?
    

    View Slide

52. What do we need to get right?
    ● Embracing events
    ● Handoff and collaboration
    ● Testing
    ● DevOps, DataOps, and “closing the loop”
    

    View Slide

53. Ingredients for success
    ● Raw materials
    ○ Big data (data lakes, data warehouses),
    events, other data sources (databases, etc)
    ● Science
    ○ Exploration, hypothesis testing, statistical
    methods, machine learning
    ● Engineering
    ○ Execute on the science using raw materials to
    build a finished product
    

    View Slide

54. Embrace Events
    

    View Slide

55. Embrace Events
    In order to predict the future
    you must understand the past.
    

    View Slide

56. Embrace Events
    Events are interesting things that have already
    happened. Events are always in the past.
    

    View Slide

57. Embrace Events
    Events span all of history. An event can be 1ms
    ago or 10 years ago.
    

    View Slide

58. Embrace Events
    Events are the core of our data analytics system.
    

    View Slide

59. View Slide

60. Align on the
    objectives and
    domain
    

    View Slide

61. Event Storming
    Capturing the key business
    events.
    Can map analytics events
    back to the business context.
    

    View Slide

62. Feature Extraction
    

    View Slide

63. Handoffs and
    collaboration
    

    View Slide

64. Who does what?
    Data science handoff:
    ● a trained model (i.e. parameter values
    that have been learned)
    ● how to extract the features needed
    by the model from the raw data
    ● start by reviewing and handing off
    versioned “design docs”
    ● once maturity is reached, automate
    handoffs
    Engineering next steps:
    ● turn a trained model into efficient
    production-quality code
    ● ensure efficient access to the data
    required by the trained model to
    make predictions
    ● reactive machine learning is critical
    to ensuring SLAs are met (latency,
    availability, etc)
    

    View Slide

65. Training
    pipeline
    

    View Slide

66. Model serving pipeline
    

    View Slide

67. DevOps, DataOps,
    and closing the loop
    

    View Slide

68. Testing
    Trained ML model
    

    View Slide

69. Determine fit
    

    View Slide

70. Full
    Testing
    Coverage
    ● Trained models
    ● Data (drift)
    ● Data models
    ● Data pipeline
    

    View Slide

71. Version control
    ● Learning algorithm configuration
    ○ Hyperparameters, etc
    ● Pipeline processes and data transformations
    

    View Slide

72. DataOps Goals
    ● ML processes is fully version controlled and
    reproducible
    ● Commiting changes kicks off tests to validate those
    changes, and triggers downstream processes
    ● CI/CD for ML
    

    View Slide

73. Conclusion
    

    View Slide

74. Actions
    

    View Slide

75. Recap
    

    View Slide

76. Let’s chat!
    Kevin Webber, RedElastic
    Principal Consultant
    kevin.webber@redelastic.com
    Dana Harrington, RedElastic
    Chief Scientist
    dana.harrington@redelastic.com
    

    View Slide