Cognitive Biases in Engineering Organizations

72f8175ac13e367bb500dd4da1f1aa32?s=47 Jonathan Klein
November 17, 2014

Cognitive Biases in Engineering Organizations

Resources at jkle.in/biases. This talk was given at Velocity Barcelona in November 2014.

72f8175ac13e367bb500dd4da1f1aa32?s=128

Jonathan Klein

November 17, 2014
Tweet

Transcript

  1. Velocity Barcelona – November 17, 2014 @jonathanklein Jonathan Klein in

    Engineering Organizations Cognitive Biases
  2. Talk Resources (Slides/Links) http://jkle.in/biases

  3. jkle.in/biases @jonathanklein 3

  4. jkle.in/biases @jonathanklein Who Am I? 3

  5. jkle.in/biases @jonathanklein Who Am I? • Senior performance engineer at

    Etsy 3
  6. jkle.in/biases @jonathanklein Who Am I? • Senior performance engineer at

    Etsy • I write the Etsy Site Performance Reports 3
  7. jkle.in/biases @jonathanklein Who Am I? • Senior performance engineer at

    Etsy • I write the Etsy Site Performance Reports • I like figuring out why we believe what we believe 3
  8. 4

  9. @jonathanklein jkle.in/biases Cognitive Bias: A deviation in judgement where inferences

    may be illogical 5
  10. @jonathanklein jkle.in/biases System 1 Operates automatically and quickly, with little

    or no effort and no sense of voluntary control 6
  11. @jonathanklein jkle.in/biases System 2 Allocates attention to the effortful mental

    activities that demand it, including complex computations 7
  12. @jonathanklein jkle.in/biases System 1 Operates automatically and quickly, with little

    or no effort and no sense of voluntary control 8
  13. None
  14. None
  15. @jonathanklein jkle.in/biases 5-10% of videos were upside-down 11

  16. “It was designed for right-handed users, but phones are usually

    rotated 180 degrees when held in left hands.” 12
  17. jkle.in/biases @jonathanklein • Projection Bias • Planning Fallacy • Bandwagon

    Effect • Sunk Cost Fallacy • Hyperbolic Discounting • Fundamental Attribution Error 13
  18. Part 1: A Fictional Story

  19. None
  20. @jonathanklein jkle.in/biases Sales Guy 16

  21. @jonathanklein jkle.in/biases Engineering Manager 17

  22. @jonathanklein jkle.in/biases Software Engineer 18

  23. @jonathanklein jkle.in/biases CEO 19

  24. 20

  25. @jonathanklein jkle.in/biases Sales Guy Sells a feature that doesn’t exist

    yet 21
  26. @jonathanklein jkle.in/biases Engineering Manager ROI is positive at < 6

    weeks, thinks we can do it in 4 22
  27. @jonathanklein jkle.in/biases Software Engineer “Yeah, we can do that” 23

  28. @jonathanklein jkle.in/biases Two weeks of development… 24

  29. @jonathanklein jkle.in/biases Software Engineer “Oh sh*t, we need 8 more

    weeks” 25
  30. @jonathanklein jkle.in/biases Engineering Manager “Let’s push forward, and hope we

    beat expectations” 26
  31. @jonathanklein jkle.in/biases Software Engineer “Okay, we’ll cut some corners” 27

  32. @jonathanklein jkle.in/biases Six more weeks go by… 28

  33. 29

  34. @jonathanklein jkle.in/biases CEO “Who do we need to fire?” 30

  35. This could have been avoided 31

  36. Deconstructing the Story

  37. @jonathanklein jkle.in/biases Sales Guy Sells a feature that doesn’t exist

    yet 33
  38. @jonathanklein jkle.in/biases Projection Bias One thinks that others have the

    same priority, attitude or belief that one harbors oneself, even if this is unlikely to be the case. 34
  39. @jonathanklein jkle.in/biases Projection Bias One thinks that others have the

    same priority, attitude or belief that one harbors oneself, even if this is unlikely to be the case. Solution 34
  40. @jonathanklein jkle.in/biases Projection Bias One thinks that others have the

    same priority, attitude or belief that one harbors oneself, even if this is unlikely to be the case. Solution Empathy 34
  41. @jonathanklein jkle.in/biases Projection Bias One thinks that others have the

    same priority, attitude or belief that one harbors oneself, even if this is unlikely to be the case. Solution Empathy Have people switch roles 34
  42. @jonathanklein jkle.in/biases Engineering Manager ROI is positive at < 6

    weeks, thinks we can do it in 4 35
  43. @jonathanklein jkle.in/biases Planning Fallacy The tendency for people to underestimate

    how long they will need to complete a task, even when they have experience of similar tasks over-running 36
  44. @jonathanklein jkle.in/biases Planning Fallacy The tendency for people to underestimate

    how long they will need to complete a task, even when they have experience of similar tasks over-running Solution 36
  45. @jonathanklein jkle.in/biases Planning Fallacy The tendency for people to underestimate

    how long they will need to complete a task, even when they have experience of similar tasks over-running Solution Prototyping 36
  46. @jonathanklein jkle.in/biases Planning Fallacy The tendency for people to underestimate

    how long they will need to complete a task, even when they have experience of similar tasks over-running Solution Prototyping Use information from similar ventures 36
  47. @jonathanklein jkle.in/biases Planning Fallacy The tendency for people to underestimate

    how long they will need to complete a task, even when they have experience of similar tasks over-running Solution Prototyping Use information from similar ventures “Planning poker” exercise 36
  48. @jonathanklein jkle.in/biases Planning Poker Cards 37

  49. @jonathanklein jkle.in/biases 38

  50. @jonathanklein jkle.in/biases Software Engineer “Yeah, we can do that” 39

  51. @jonathanklein jkle.in/biases Bandwagon Effect The rate of uptake of beliefs,

    ideas, fads and trends increases the more that they have already been adopted by others 40
  52. @jonathanklein jkle.in/biases Bandwagon Effect The rate of uptake of beliefs,

    ideas, fads and trends increases the more that they have already been adopted by others Solution 40
  53. @jonathanklein jkle.in/biases Bandwagon Effect The rate of uptake of beliefs,

    ideas, fads and trends increases the more that they have already been adopted by others Solution Make decisions asynchronously 40
  54. @jonathanklein jkle.in/biases Bandwagon Effect The rate of uptake of beliefs,

    ideas, fads and trends increases the more that they have already been adopted by others Solution Make decisions asynchronously Ask for confidential feedback 40
  55. @jonathanklein jkle.in/biases Bandwagon Effect The rate of uptake of beliefs,

    ideas, fads and trends increases the more that they have already been adopted by others Solution Make decisions asynchronously Ask for confidential feedback Don’t prime the estimate 40
  56. @jonathanklein jkle.in/biases Bandwagon Effect The rate of uptake of beliefs,

    ideas, fads and trends increases the more that they have already been adopted by others Solution Make decisions asynchronously Ask for confidential feedback Don’t prime the estimate “Planning poker” exercise 40
  57. @jonathanklein jkle.in/biases Software Engineer “Oh sh*t, we need 8 more

    weeks” 41
  58. @jonathanklein jkle.in/biases Engineering Manager “Let’s push forward, and hope we

    beat expectations” 42
  59. @jonathanklein jkle.in/biases Sunk Cost Fallacy A sunk cost is a

    retrospective (past) cost that has already been incurred and cannot be recovered 43
  60. @jonathanklein jkle.in/biases Sunk Cost Fallacy A sunk cost is a

    retrospective (past) cost that has already been incurred and cannot be recovered Solution 43
  61. @jonathanklein jkle.in/biases Sunk Cost Fallacy A sunk cost is a

    retrospective (past) cost that has already been incurred and cannot be recovered Solution Do some math 43
  62. @jonathanklein jkle.in/biases Sunk Cost Fallacy A sunk cost is a

    retrospective (past) cost that has already been incurred and cannot be recovered Solution Do some math At every point in the project, the ROI should still be positive 43
  63. @jonathanklein jkle.in/biases 44 Return on Investment

  64. @jonathanklein jkle.in/biases 45 Return on Investment

  65. @jonathanklein jkle.in/biases 45 Return on Investment Project Completed

  66. @jonathanklein jkle.in/biases Our Example Estimated Development Time: 4 Weeks Estimated

    Development Cost: $8000 Client Is Paying: $12,000 46
  67. @jonathanklein jkle.in/biases Our Example Estimated Development Time: 8 Weeks Estimated

    Development Cost: $16,000 Client Is Paying: $12,000 47
  68. 48

  69. @jonathanklein jkle.in/biases Software Engineer “Okay, we’ll cut some corners” 49

  70. @jonathanklein jkle.in/biases Hyperbolic Discounting Given two similar rewards, humans show

    a preference for one that arrives sooner rather than later. Humans are said to discount the value of the later reward, by a factor that increases with the length of the delay. 50
  71. @jonathanklein jkle.in/biases Hyperbolic Discounting Given two similar rewards, humans show

    a preference for one that arrives sooner rather than later. Humans are said to discount the value of the later reward, by a factor that increases with the length of the delay. Solution 50
  72. @jonathanklein jkle.in/biases Hyperbolic Discounting Given two similar rewards, humans show

    a preference for one that arrives sooner rather than later. Humans are said to discount the value of the later reward, by a factor that increases with the length of the delay. Solution Deliberately stop and do an estimation exercise 50
  73. @jonathanklein jkle.in/biases Hyperbolic Discounting Given two similar rewards, humans show

    a preference for one that arrives sooner rather than later. Humans are said to discount the value of the later reward, by a factor that increases with the length of the delay. Solution Deliberately stop and do an estimation exercise Consider implications for future developers 50
  74. @jonathanklein jkle.in/biases The Stanford Marshmallow Experiment 51

  75. None
  76. None
  77. None
  78. jkle.in/biases @jonathanklein Experimental Outcomes Researchers found that children who were

    able to wait longer for the preferred rewards tended to have better life outcomes: • SAT Scores • Educational attainment • Body Mass Index (BMI) 55
  79. 56

  80. @jonathanklein jkle.in/biases CEO “Who do we need to fire?” 57

  81. @jonathanklein jkle.in/biases Fundamental Attribution Error People's tendency to place an

    undue emphasis on internal characteristics to explain someone else's behavior in a given situation, rather than considering external factors. 58
  82. @jonathanklein jkle.in/biases Fundamental Attribution Error People's tendency to place an

    undue emphasis on internal characteristics to explain someone else's behavior in a given situation, rather than considering external factors. Solution Blameless postmortems 59
  83. @jonathanklein jkle.in/biases Everyone had good intentions 60

  84. Our intuition is imperfect 61

  85. Our intuition is imperfect 61

  86. 62

  87. Part 2: Real Studies

  88. @jonathanklein jkle.in/biases 64

  89. @jonathanklein jkle.in/biases Which is More Dangerous? 64

  90. @jonathanklein jkle.in/biases Which is More Dangerous? A disease that kills

    1,286 people out of every 10,000 64
  91. @jonathanklein jkle.in/biases Which is More Dangerous? A disease that kills

    1,286 people out of every 10,000 A disease that kills 24.4 out of 100 64
  92. @jonathanklein jkle.in/biases Which is More Dangerous? ! A disease that

    kills 1,286 people out of every 10,000 A disease that kills 24.4 out of 100 65
  93. @jonathanklein jkle.in/biases Which is More Dangerous? ! A disease that

    kills 1,286 people out of every 10,000 A disease that kills 24.4 out of 100 65 Most people say this
  94. @jonathanklein jkle.in/biases Which is More Dangerous? ! A disease that

    kills 1,286 people out of every 10,000 A disease that kills 24.4 out of 100 65 Most people say this This is twice as bad
  95. @jonathanklein jkle.in/biases Denominator Neglect 66

  96. @jonathanklein jkle.in/biases We Can’t Always Trust Our Intuition 67

  97. @jonathanklein jkle.in/biases Dinner Plates Soup/Salad bowls Dessert plates Cups Saucers

    8, all good condition 8, all good condition 8, all good condition 8, 2 broken 8, 7 broken 8, all good condition 8, all good condition 8, all good condition None None 68 Set A: 40 Pieces Set B: 24 Pieces
  98. @jonathanklein jkle.in/biases Dinner Plates Soup/Salad bowls Dessert plates Cups Saucers

    8, all good condition 8, all good condition 8, all good condition 8, 2 broken 8, 7 broken 8, all good condition 8, all good condition 8, all good condition None None 69 Set A: 40 Pieces Set B: 24 Pieces
  99. jkle.in/biases @jonathanklein Three Groups in the Experiment ! 1. Shown

    both sets together 2. Shown only set A 3. Shown only set B 70
  100. @jonathanklein jkle.in/biases Group 1 - Joint Evaluation: Set A: $32

    Set B: $30 71
  101. @jonathanklein jkle.in/biases 72

  102. @jonathanklein jkle.in/biases Groups 2 and 3 - Single Evaluation: 72

  103. @jonathanklein jkle.in/biases Groups 2 and 3 - Single Evaluation: Set

    A: $23 Set B: $33 72
  104. @jonathanklein jkle.in/biases Dinner Plates Soup/Salad bowls Dessert plates Cups Saucers

    8, all good condition 8, all good condition 8, all good condition 8, 2 broken 8, 7 broken 8, all good condition 8, all good condition 8, all good condition None None 73 Set A: 40 Pieces Set B: 24 Pieces
  105. Manipulation

  106. jkle.in/biases @jonathanklein 75

  107. jkle.in/biases @jonathanklein Subscriptions to The Economist 75

  108. jkle.in/biases @jonathanklein Subscriptions to The Economist 1. A web-only subscription

    for $59 75
  109. jkle.in/biases @jonathanklein Subscriptions to The Economist 1. A web-only subscription

    for $59 2. A print-only subscription for $125 75
  110. jkle.in/biases @jonathanklein Subscriptions to The Economist 1. A web-only subscription

    for $59 2. A print-only subscription for $125 3. A web + print subscription for $125 75
  111. jkle.in/biases @jonathanklein Subscriptions to The Economist ! 1. A web-only

    subscription for $59 2. A print-only subscription for $125 3. A web + print subscription for $125 76 Useless
  112. jkle.in/biases @jonathanklein Subscriptions to The Economist ! 1. A web-only

    subscription for $59 2. A print-only subscription for $125 3. A web + print subscription for $125 77 16% 0% 84%
  113. jkle.in/biases @jonathanklein Subscriptions to The Economist ! 1. A web-only

    subscription for $59 2. A print-only subscription for $125 3. A web + print subscription for $125 78
  114. jkle.in/biases @jonathanklein Subscriptions to The Economist ! 1. A web-only

    subscription for $59 2. A print-only subscription for $125 3. A web + print subscription for $125 79 16% 84% 68% 32%
  115. @jonathanklein jkle.in/biases Removing the middle option costs $3400+ per 100

    subscribers 80
  116. @jonathanklein jkle.in/biases Based on 2013 circulation, this could cost The

    Economist $53M 81
  117. @jonathanklein jkle.in/biases 82

  118. @jonathanklein jkle.in/biases What’s the best way to sell a $3,000

    suit? 82
  119. @jonathanklein jkle.in/biases What’s the best way to sell a $3,000

    suit? Put it next to a $10,000 suit 82
  120. None
  121. None
  122. None
  123. None
  124. @jonathanklein jkle.in/biases Revisiting the Sunk Cost Fallacy 87

  125. @jonathanklein jkle.in/biases It Preys on Loss Aversion 88

  126. @jonathanklein jkle.in/biases Another Experiment ! 89 $0.01 $0.15

  127. @jonathanklein jkle.in/biases Another Experiment ! 90 Most people choose this

    $0.01 $0.15
  128. @jonathanklein jkle.in/biases Lower the Price ! 91 Free! $0.14

  129. @jonathanklein jkle.in/biases Lower the Price ! 92 Most people choose

    this Free! $0.14
  130. None
  131. @jonathanklein jkle.in/biases We need a framework for decision making 94

  132. Bayesian Reasoning

  133. @jonathanklein jkle.in/biases 96 Bayes' Theorem

  134. @jonathanklein jkle.in/biases 1. Anchor your judgement of the probability of

    an outcome on a plausible base rate 97
  135. @jonathanklein jkle.in/biases 2. Question the importance and relevancy of your

    evidence 98
  136. @jonathanklein jkle.in/biases Engineering Manager ROI is positive at < 6

    weeks, thinks we can do it in 4 99
  137. @jonathanklein jkle.in/biases “What is the base rate for estimation accuracy?”

    100
  138. @jonathanklein jkle.in/biases “Why is this situation different?” 101

  139. None
  140. None
  141. None
  142. None
  143. None
  144. @jonathanklein jkle.in/biases To make rational decisions, you must understand the

    base rate 107
  145. @jonathanklein jkle.in/biases You must also objectively evaluate the evidence for

    departing from the base rate 108
  146. Two Principles: Dramatically better decisions 109

  147. @jonathanklein jkle.in/biases 110 jkle.in/biases

  148. @jonathanklein jkle.in/biases Thanks! jkle.in/biases @jonathanklein jonathan@etsy.com 111