Cognitive Biases in Engineering Organizations - Craft Conf

Cognitive Biases in Engineering Organizations - Craft Conf

72f8175ac13e367bb500dd4da1f1aa32?s=128

Jonathan Klein

April 29, 2016
Tweet

Transcript

  1. Craft Conf – April 29th, 2016 @jonathanklein Jonathan Klein in

    Engineering Organizations Cognitive Biases
  2. Talk Resources (Slides/Links) jkle.in/craft

  3. jkle.in/craft @jonathanklein 3

  4. jkle.in/craft @jonathanklein Who Am I? 3

  5. jkle.in/craft @jonathanklein Who Am I? • Director of Engineering at

    Attend 3
  6. jkle.in/craft @jonathanklein Who Am I? • Director of Engineering at

    Attend • Co-Author of Hound (a code search tool) 3
  7. jkle.in/craft @jonathanklein Who Am I? • Director of Engineering at

    Attend • Co-Author of Hound (a code search tool) • I like figuring out why we believe what we believe 3
  8. 4

  9. @jonathanklein jkle.in/craft Cognitive Bias: A deviation in judgement where inferences

    may be illogical 5
  10. @jonathanklein jkle.in/craft System 1 Operates automatically and quickly, with little

    or no effort and no sense of voluntary control 6
  11. @jonathanklein jkle.in/craft System 2 Allocates attention to the effortful mental

    activities that demand it, including complex computations 7
  12. @jonathanklein jkle.in/craft System 1 Operates automatically and quickly, with little

    or no effort and no sense of voluntary control 8
  13. None
  14. None
  15. @jonathanklein jkle.in/craft Daniel Kahneman Focuses on Problematic Biases 11 Gary

    Klein Focuses on Helpful Biases (NDM)
  16. None
  17. “Although DK is thrilled by the remarkable intuitive skills of

    experts that GK and others have described, he also takes considerable pleasure in demonstrations of human folly and in the comeuppance of overconfident pseudo-experts. For his part, GK recognizes that formal procedures and algorithms sometimes outdo human judgment, but he enjoys hearing about cases in which the bureaucratization of decision making fails.” Conditions for Intuitive Expertise A Failure to Disagree
  18. @jonathanklein jkle.in/craft 5-10% of videos were upside-down 14

  19. “It was designed for right-handed users, but phones are usually

    rotated 180 degrees when held in left hands.” 15
  20. jkle.in/craft @jonathanklein • Projection Bias • Planning Fallacy • Bandwagon

    Effect • Sunk Cost Fallacy • Hyperbolic Discounting • Fundamental Attribution Error 16
  21. Part 1: A Fictional Story

  22. None
  23. @jonathanklein jkle.in/craft Sales Guy 19

  24. @jonathanklein jkle.in/craft Engineering Manager 20

  25. @jonathanklein jkle.in/craft Software Engineer 21

  26. @jonathanklein jkle.in/craft CEO 22

  27. 23

  28. @jonathanklein jkle.in/craft Sales Guy Sells a feature that doesn’t exist

    yet 24
  29. @jonathanklein jkle.in/craft Engineering Manager ROI is positive at < 6

    weeks, thinks we can do it in 4 25
  30. @jonathanklein jkle.in/craft Software Engineer “Yeah, we can do that” 26

  31. @jonathanklein jkle.in/craft Two weeks of development… 27

  32. @jonathanklein jkle.in/craft Software Engineer “Oh sh*t, we need 8 more

    weeks” 28
  33. @jonathanklein jkle.in/craft Engineering Manager “Let’s push forward, and hope we

    beat expectations” 29
  34. @jonathanklein jkle.in/craft Software Engineer “Okay, we’ll cut some corners” 30

  35. @jonathanklein jkle.in/craft Six more weeks go by… 31

  36. 32

  37. @jonathanklein jkle.in/craft CEO “Who do we need to fire?” 33

  38. This could have been avoided 34

  39. Deconstructing the Story

  40. @jonathanklein jkle.in/craft Sales Guy Sells a feature that doesn’t exist

    yet 36
  41. @jonathanklein jkle.in/craft Projection Bias One thinks that others have the

    same priority, attitude or belief that one harbors oneself, even if this is unlikely to be the case. 37
  42. @jonathanklein jkle.in/craft Projection Bias One thinks that others have the

    same priority, attitude or belief that one harbors oneself, even if this is unlikely to be the case. Solution 37
  43. @jonathanklein jkle.in/craft Projection Bias One thinks that others have the

    same priority, attitude or belief that one harbors oneself, even if this is unlikely to be the case. Solution Empathy 37
  44. @jonathanklein jkle.in/craft Projection Bias One thinks that others have the

    same priority, attitude or belief that one harbors oneself, even if this is unlikely to be the case. Solution Empathy Have people switch roles 37
  45. @jonathanklein jkle.in/craft Engineering Manager ROI is positive at < 6

    weeks, thinks we can do it in 4 38
  46. @jonathanklein jkle.in/craft Planning Fallacy The tendency for people to underestimate

    how long they will need to complete a task, even when they have experience of similar tasks over-running 39
  47. @jonathanklein jkle.in/craft Planning Fallacy The tendency for people to underestimate

    how long they will need to complete a task, even when they have experience of similar tasks over-running Solution 39
  48. @jonathanklein jkle.in/craft Planning Fallacy The tendency for people to underestimate

    how long they will need to complete a task, even when they have experience of similar tasks over-running Solution Prototyping 39
  49. @jonathanklein jkle.in/craft Planning Fallacy The tendency for people to underestimate

    how long they will need to complete a task, even when they have experience of similar tasks over-running Solution Prototyping Use information from similar ventures 39
  50. @jonathanklein jkle.in/craft Planning Fallacy The tendency for people to underestimate

    how long they will need to complete a task, even when they have experience of similar tasks over-running Solution Prototyping Use information from similar ventures “Planning poker” exercise 39
  51. @jonathanklein jkle.in/craft Planning Poker Cards 40

  52. @jonathanklein jkle.in/craft 41

  53. @jonathanklein jkle.in/craft This works because it changes the reference point

    42
  54. @jonathanklein jkle.in/craft “Reference Class Forecasting” 43

  55. @jonathanklein jkle.in/craft 44

  56. @jonathanklein jkle.in/craft Story Point Best Practices 44

  57. @jonathanklein jkle.in/craft Story Point Best Practices Points != Time 44

  58. @jonathanklein jkle.in/craft Story Point Best Practices Points != Time Points

    Don’t Translate Across Teams 44
  59. @jonathanklein jkle.in/craft Story Point Best Practices Points != Time Points

    Don’t Translate Across Teams Points Don’t Work For Huge Projects 44
  60. @jonathanklein jkle.in/craft Planning Poker Cards 45

  61. @jonathanklein jkle.in/craft Planning Poker Cards 46

  62. @jonathanklein jkle.in/craft Software Engineer “Yeah, we can do that” 47

  63. @jonathanklein jkle.in/craft Bandwagon Effect The rate of uptake of beliefs,

    ideas, fads and trends increases the more that they have already been adopted by others 48
  64. @jonathanklein jkle.in/craft Bandwagon Effect The rate of uptake of beliefs,

    ideas, fads and trends increases the more that they have already been adopted by others Solution 48
  65. @jonathanklein jkle.in/craft Bandwagon Effect The rate of uptake of beliefs,

    ideas, fads and trends increases the more that they have already been adopted by others Solution Make decisions asynchronously 48
  66. @jonathanklein jkle.in/craft Bandwagon Effect The rate of uptake of beliefs,

    ideas, fads and trends increases the more that they have already been adopted by others Solution Make decisions asynchronously Ask for confidential feedback 48
  67. @jonathanklein jkle.in/craft Bandwagon Effect The rate of uptake of beliefs,

    ideas, fads and trends increases the more that they have already been adopted by others Solution Make decisions asynchronously Ask for confidential feedback Don’t prime the estimate 48
  68. @jonathanklein jkle.in/craft Bandwagon Effect The rate of uptake of beliefs,

    ideas, fads and trends increases the more that they have already been adopted by others Solution Make decisions asynchronously Ask for confidential feedback Don’t prime the estimate “Planning poker” exercise 48
  69. @jonathanklein jkle.in/craft Software Engineer “Oh sh*t, we need 8 more

    weeks” 49
  70. @jonathanklein jkle.in/craft Engineering Manager “Let’s push forward, and hope we

    beat expectations” 50
  71. @jonathanklein jkle.in/craft Sunk Cost Fallacy A sunk cost is a

    retrospective (past) cost that has already been incurred and cannot be recovered 51
  72. @jonathanklein jkle.in/craft Sunk Cost Fallacy A sunk cost is a

    retrospective (past) cost that has already been incurred and cannot be recovered Solution 51
  73. @jonathanklein jkle.in/craft Sunk Cost Fallacy A sunk cost is a

    retrospective (past) cost that has already been incurred and cannot be recovered Solution Do some math 51
  74. @jonathanklein jkle.in/craft Sunk Cost Fallacy A sunk cost is a

    retrospective (past) cost that has already been incurred and cannot be recovered Solution Do some math At every point in the project, the ROI should still be positive 51
  75. @jonathanklein jkle.in/craft 52 Return on Investment

  76. @jonathanklein jkle.in/craft 53 Return on Investment

  77. @jonathanklein jkle.in/craft 53 Return on Investment Project Completed

  78. @jonathanklein jkle.in/craft Our Example Estimated Development Time: 4 Weeks Estimated

    Development Cost: $8000 Client Is Paying: $12,000 54
  79. @jonathanklein jkle.in/craft Our Example Estimated Development Time: 8 Weeks Estimated

    Development Cost: $16,000 Client Is Paying: $12,000 55
  80. 56

  81. @jonathanklein jkle.in/craft Software Engineer “Okay, we’ll cut some corners” 57

  82. @jonathanklein jkle.in/craft Hyperbolic Discounting Given two similar rewards, humans show

    a preference for one that arrives sooner rather than later. Humans are said to discount the value of the later reward, by a factor that increases with the length of the delay. 58
  83. @jonathanklein jkle.in/craft Hyperbolic Discounting Given two similar rewards, humans show

    a preference for one that arrives sooner rather than later. Humans are said to discount the value of the later reward, by a factor that increases with the length of the delay. Solution 58
  84. @jonathanklein jkle.in/craft Hyperbolic Discounting Given two similar rewards, humans show

    a preference for one that arrives sooner rather than later. Humans are said to discount the value of the later reward, by a factor that increases with the length of the delay. Solution Deliberately stop and do an estimation exercise 58
  85. @jonathanklein jkle.in/craft Hyperbolic Discounting Given two similar rewards, humans show

    a preference for one that arrives sooner rather than later. Humans are said to discount the value of the later reward, by a factor that increases with the length of the delay. Solution Deliberately stop and do an estimation exercise Consider implications for future developers 58
  86. @jonathanklein jkle.in/craft The Stanford Marshmallow Experiment 59

  87. None
  88. None
  89. None
  90. jkle.in/craft @jonathanklein Experimental Outcomes Researchers found that children who were

    able to wait longer for the preferred rewards tended to have better life outcomes: • SAT Scores • Educational attainment • Body Mass Index (BMI) 63
  91. 64

  92. @jonathanklein jkle.in/craft CEO “Who do we need to fire?” 65

  93. @jonathanklein jkle.in/craft Fundamental Attribution Error People's tendency to place an

    undue emphasis on internal characteristics to explain someone else's behavior in a given situation, rather than considering external factors. 66
  94. @jonathanklein jkle.in/craft Fundamental Attribution Error People's tendency to place an

    undue emphasis on internal characteristics to explain someone else's behavior in a given situation, rather than considering external factors. Solution Blameless postmortems 67
  95. @jonathanklein jkle.in/craft Everyone had good intentions 68

  96. Our intuition is imperfect 69

  97. Our intuition is imperfect 69

  98. 70

  99. Part 2: Real Studies

  100. @jonathanklein jkle.in/craft 72

  101. @jonathanklein jkle.in/craft A Quiz 72

  102. @jonathanklein jkle.in/craft A Quiz A ball and a bat together

    cost $1.10. The bat costs a dollar more than the ball. 72
  103. @jonathanklein jkle.in/craft A Quiz A ball and a bat together

    cost $1.10. The bat costs a dollar more than the ball. How much does the ball cost? 72
  104. @jonathanklein jkle.in/craft 73

  105. @jonathanklein jkle.in/craft A Quiz 73

  106. @jonathanklein jkle.in/craft A Quiz Most people say $0.10 73

  107. @jonathanklein jkle.in/craft A Quiz Most people say $0.10 The answer

    is actually $0.05 73
  108. @jonathanklein jkle.in/craft We Can’t Always Trust Our Intuition 74

  109. @jonathanklein jkle.in/craft Dinner Plates Soup/Salad bowls Dessert plates Cups Saucers

    8, all good condition 8, all good condition 8, all good condition 8, 2 broken 8, 7 broken 8, all good condition 8, all good condition 8, all good condition None None 75 Set A: 40 Pieces Set B: 24 Pieces
  110. @jonathanklein jkle.in/craft Dinner Plates Soup/Salad bowls Dessert plates Cups Saucers

    8, all good condition 8, all good condition 8, all good condition 8, 2 broken 8, 7 broken 8, all good condition 8, all good condition 8, all good condition None None 76 Set A: 40 Pieces Set B: 24 Pieces
  111. jkle.in/craft @jonathanklein Three Groups in the Experiment 1. Shown both

    sets together 2. Shown only set A 3. Shown only set B 77
  112. @jonathanklein jkle.in/craft Group 1 - Joint Evaluation: Set A: $32

    Set B: $30 78
  113. @jonathanklein jkle.in/craft 79

  114. @jonathanklein jkle.in/craft Groups 2 and 3 - Single Evaluation: 79

  115. @jonathanklein jkle.in/craft Groups 2 and 3 - Single Evaluation: Set

    A: $23 Set B: $33 79
  116. @jonathanklein jkle.in/craft Dinner Plates Soup/Salad bowls Dessert plates Cups Saucers

    8, all good condition 8, all good condition 8, all good condition 8, 2 broken 8, 7 broken 8, all good condition 8, all good condition 8, all good condition None None 80 Set A: 40 Pieces Set B: 24 Pieces
  117. Manipulation

  118. jkle.in/craft @jonathanklein 82

  119. jkle.in/craft @jonathanklein Subscriptions to The Economist 82

  120. jkle.in/craft @jonathanklein Subscriptions to The Economist 1. A web-only subscription

    for $59 82
  121. jkle.in/craft @jonathanklein Subscriptions to The Economist 1. A web-only subscription

    for $59 2. A print-only subscription for $125 82
  122. jkle.in/craft @jonathanklein Subscriptions to The Economist 1. A web-only subscription

    for $59 2. A print-only subscription for $125 3. A web + print subscription for $125 82
  123. jkle.in/craft @jonathanklein Subscriptions to The Economist 1. A web-only subscription

    for $59 2. A print-only subscription for $125 3. A web + print subscription for $125 83 Useless
  124. jkle.in/craft @jonathanklein Subscriptions to The Economist 1. A web-only subscription

    for $59 2. A print-only subscription for $125 3. A web + print subscription for $125 84 16% 0% 84%
  125. jkle.in/craft @jonathanklein Subscriptions to The Economist 1. A web-only subscription

    for $59 2. A print-only subscription for $125 3. A web + print subscription for $125 85
  126. jkle.in/craft @jonathanklein Subscriptions to The Economist 1. A web-only subscription

    for $59 2. A print-only subscription for $125 3. A web + print subscription for $125 86 16% 84% 68% 32%
  127. @jonathanklein jkle.in/craft Removing the middle option costs $3400+ per 100

    subscribers 87
  128. @jonathanklein jkle.in/craft Based on 2013 circulation, this could cost The

    Economist $53M 88
  129. @jonathanklein jkle.in/craft 89

  130. @jonathanklein jkle.in/craft What’s the best way to sell a $3,000

    suit? 89
  131. @jonathanklein jkle.in/craft What’s the best way to sell a $3,000

    suit? Put it next to a $10,000 suit 89
  132. None
  133. None
  134. None
  135. None
  136. None
  137. @jonathanklein jkle.in/craft Revisiting the Sunk Cost Fallacy 95

  138. @jonathanklein jkle.in/craft It Preys on Loss Aversion 96

  139. @jonathanklein jkle.in/craft Another Experiment 97 $0.01 $0.15

  140. @jonathanklein jkle.in/craft Another Experiment 98 Most people choose this $0.01

    $0.15
  141. @jonathanklein jkle.in/craft Lower the Price 99 Free! $0.14

  142. @jonathanklein jkle.in/craft Lower the Price 100 Most people choose this

    Free! $0.14
  143. None
  144. @jonathanklein jkle.in/craft We need a framework for decision making 102

  145. Bayesian Reasoning

  146. @jonathanklein jkle.in/craft 104 Bayes' Theorem

  147. @jonathanklein jkle.in/craft 1. Anchor your judgement of the probability of

    an outcome on a plausible base rate 105
  148. @jonathanklein jkle.in/craft 2. Question the importance and relevancy of your

    evidence 106
  149. @jonathanklein jkle.in/craft Engineering Manager ROI is positive at < 6

    weeks, thinks we can do it in 4 107
  150. @jonathanklein jkle.in/craft “What is the base rate for estimation accuracy?”

    108
  151. @jonathanklein jkle.in/craft “Why is this situation different?” 109

  152. None
  153. None
  154. None
  155. None
  156. None
  157. @jonathanklein jkle.in/craft To make rational decisions, you must understand the

    base rate 115
  158. @jonathanklein jkle.in/craft You must also objectively evaluate the evidence for

    departing from the base rate 116
  159. Two Principles: Dramatically better decisions 117

  160. @jonathanklein jkle.in/craft 118 jkle.in/craft

  161. @jonathanklein jkle.in/craft Thanks! jkle.in/craft @jonathanklein jonathan@attend.com 119