Beware of Survivorship Bias! (JCON)

Most talks on a typical conference schedule contain success stories of technology. This could lead to survivorship bias. Survivorship bias causes you to draw false conclusions because you mostly heard about successes, but hardly ever about failures. Yet failures provide us with lots of valuable knowledge: when not to apply a certain technique, pattern or process.

So no success stories in this talk! Just some ‘silver bullets’ we tried to use to solve our problems, but turned out to be Very Bad Ideas™. We’ll share how survivorship bias can easily influence your ideas and cloud your judgement.

After attending this talk you’ll be more aware of survivorship bias and what you can do to keep your head cool, no matter how many ‘silver bullets’ are fired at you.

Transcript

1. Beware of
   Survivorship Bias!
   Hanno Embregts
   Maarten Mulders
   @hannotify
   @mthmulders
   #SurvivorshipBias
   

   View Slide

2. #SurvivorshipBias
   

   View Slide

3. #SurvivorshipBias
   

   View Slide

4. https://nos.nl/data/image/2017/01/29/349992/xxl.jpg
   #SurvivorshipBias
   

   View Slide

5. https://nl.wikipedia.org/wiki/Bestand:Verkiezingen2.jpg
   #SurvivorshipBias
   

   View Slide

6. ¯\_(
   ツ
   )_/¯
   #SurvivorshipBias
   

   View Slide

7. #SurvivorshipBias
   

   View Slide

8. ¯\_(
   ツ
   )_/¯
   #SurvivorshipBias
   

   View Slide

9. https://www.pexels.com/photo/grayscale-photography-of-man-and-woman-sitting-on-chair-1181735
   #SurvivorshipBias
   

   View Slide

10. Do I need a blockchain?
    No.
    https://medium.com/@sbmeunier/when-do-you-need-blockchain-decision-models-a5c40e7c9ba1
    #SurvivorshipBias
    

    View Slide

11. Do you need a database?
    Don't use
    a blockchain
    Does it require shared
    write access?
    Are writers known
    and trusted?
    Do you want/need
    to use a trusted 3rd
    party?
    Yes
    Yes
    No
    No
    No
    Are writers'
    interests
    unified?
    Yes
    Yes
    No
    Do you need to
    control functionality?
    No
    Yes
    Do you want transactions
    to be public or private?
    No
    Use a public
    blockchain
    Public
    Where is consensus
    determined?
    Use a hybrid
    blockchain
    Private
    Inter-firm
    Use a private
    blockchain
    Intra-firm
    Yes
    https://medium.com/@sbmeunier/when-do-you-need-blockchain-decision-models-a5c40e7c9ba1
    #SurvivorshipBias
    

    View Slide

12. Can you articulate a
    real business problem?
    Try a normal
    database.
    Could this have
    been fixed before
    blockchains?
    Is this a 'digital
    identities' and 'blockchains
    are free' play?
    Should or could an entity
    have overall control?
    Yes
    No
    No
    Yes
    Yes
    Will all
    participants need
    to upgrade/replace
    systems?
    No
    Yes
    Will participants mind their
    data being visible?
    No
    You may have
    a use case
    No
    Can they adopt?
    Do the benefits justify the
    costs of adoption?
    Yes
    Yes
    Yes
    https://medium.com/@sbmeunier/when-do-you-need-blockchain-decision-models-a5c40e7c9ba1
    #SurvivorshipBias
    

    View Slide

13. election blockchain
    writers known?
    yes no
    writers trusted?
    yes no
    writers' interests uni ed?
    yes no
    trusted third party?
    yes no
    private transactions?
    yes no
    #SurvivorshipBias
    

    View Slide

14. Survivorship
    Survivorship
    Survivorship
    Survivorship
    Survivorship
    Bias
    Bias
    Bias
    Bias
    Bias
    https://pxhere.com/nl/photo/973302
    #SurvivorshipBias
    

    View Slide

15. https://commons.wikimedia.org/wiki/File:Survivorship-bias.png
    #SurvivorshipBias
    

    View Slide

16. It won't solve
    It won't solve
    It won't solve
    It won't solve
    It won't solve
    everything!
    everything!
    everything!
    everything!
    everything!
    Just because a conference speaker solved
    his/her problem with some new, cool
    technique doesn't mean it will also solve
    yours.
    https://www.pexels.com/photo/2-hands-holding-1-jigsaw-puzzle-piece-each-164531/
    #SurvivorshipBias
    

    View Slide

17. Conference-driven
    Conference-driven
    Conference-driven
    Conference-driven
    Conference-driven
    development
    development
    development
    development
    development
    Learn new skills at a conference, and then
    use it to solve pretty much everything.
    https://pxhere.com/en/photo/1130503
    #SurvivorshipBias
    

    View Slide

18. Overengineering
    Overengineering
    Overengineering
    Overengineering
    Overengineering
    If you apply CDD to everything you do,
    your solutions will quickly become
    overengineered.
    https://www. ickr.com/photos/footfun/2454000816
    #SurvivorshipBias
    

    View Slide

19. http://gph.is/18wDuA0
    #SurvivorshipBias
    

    View Slide

20. https://en.wikipedia.org/wiki/
    List_of_failed_and_overbudget_custom_software_projects
    #SurvivorshipBias
    

    View Slide

21. #SurvivorshipBias
    

    View Slide

22. https://github.com/MinBZK/OperatieBRP
    https://github.com/MinBZK/OperatieBRP
    https://github.com/MinBZK/OperatieBRP
    https://github.com/MinBZK/OperatieBRP
    https://github.com/MinBZK/OperatieBRP
    #SurvivorshipBias
    

    View Slide

23. #SurvivorshipBias
    

    View Slide

24. #SurvivorshipBias
    

    View Slide

25. https://pxhere.com/en/photo/655299
    #SurvivorshipBias
    

    View Slide

26. Alternatives
    Alternatives
    Alternatives
    Alternatives
    Alternatives
    https://pxhere.com/en/photo/1267965
    #SurvivorshipBias
    

    View Slide

27. Beating
    Beating
    Beating
    Beating
    Beating
    Survivorship
    Survivorship
    Survivorship
    Survivorship
    Survivorship
    Bias
    Bias
    Bias
    Bias
    Bias
    https://pxhere.com/en/photo/958105
    #SurvivorshipBias
    

    View Slide

28. Bullet image by , from
    Clker-Free-Vector-Images Pixabay
    #SurvivorshipBias
    

    View Slide

29. Understand the
    Understand the
    Understand the
    Understand the
    Understand the
    problem first
    problem first
    problem first
    problem first
    problem first
    https://pxhere.com/en/photo/843810
    #SurvivorshipBias
    

    View Slide

30. Track A Track B
    09:00-
    09:50
    QWERTY or DVORAK?
    Debunking the Keyboard
    Layout Myths
    Beware of
    Survivorship Bias!
    10:00-
    10:50
    How I Created a MMORPG
    Game Using Haskell Only
    An Introduction to
    Kubernetes
    11:00-
    11:50
    Monitoring Kubernetes with
    Prometheus
    Return of the
    Blockchain
    #SurvivorshipBias
    

    View Slide

31. Gain some experience
    Gain some experience
    Gain some experience
    Gain some experience
    Gain some experience
    first
    first
    first
    first
    first
    This could mean you have to wait until
    next year's conference in order to
    appreciate fully the contents of an
    advanced talk.
    https://pxhere.com/en/photo/918274
    #SurvivorshipBias
    

    View Slide

32. Build your personal
    Build your personal
    Build your personal
    Build your personal
    Build your personal
    conference schedule
    conference schedule
    conference schedule
    conference schedule
    conference schedule
    accordingly
    accordingly
    accordingly
    accordingly
    accordingly
    Value topics relevant to your experience
    over 'the next silver bullet'.
    https://pxhere.com/en/photo/1370392
    #SurvivorshipBias
    

    View Slide

33. Make sure you meet
    Make sure you meet
    Make sure you meet
    Make sure you meet
    Make sure you meet
    the prerequisites
    the prerequisites
    the prerequisites
    the prerequisites
    the prerequisites
    Don't attend a talk that proposes a
    solution to a problem, if you don't
    understand the problem yet.
    https://pxhere.com/en/photo/843810
    #SurvivorshipBias
    

    View Slide

34. Remember the
    Remember the
    Remember the
    Remember the
    Remember the
    Hype Cycle
    Hype Cycle
    Hype Cycle
    Hype Cycle
    Hype Cycle
    https://pxhere.com/en/photo/1262347
    #SurvivorshipBias
    

    View Slide

35. https://upload.wikimedia.org/wikipedia/commons/b/bf/Hype-Cycle-General.png
    #SurvivorshipBias
    

    View Slide

36. Possible Talk Title
    On the Rise NoSQL: The New Reality
    At the Peak NoSQL Will Change Your Life
    and Fix Your Projects
    Sliding Into the
    Trough
    How We Do NoSQL at Cool, Inc.
    Climbing the
    Slope
    Modern NoSQL Patterns and
    Practices
    Entering the
    Plateau
    NoSQL 2.0 with SilverBulletDB
    #SurvivorshipBias
    

    View Slide

37. Does the speaker
    Does the speaker
    Does the speaker
    Does the speaker
    Does the speaker
    mention any
    mention any
    mention any
    mention any
    mention any
    drawbacks?
    drawbacks?
    drawbacks?
    drawbacks?
    drawbacks?
    The best talks contain detailed
    comparisons of pros
    and cons
    https://upload.wikimedia.org/wikipedia/commons/b/b2/Balance_scale_IMGP9728.jpg
    #SurvivorshipBias
    

    View Slide

38. Use the Q&A at the
    Use the Q&A at the
    Use the Q&A at the
    Use the Q&A at the
    Use the Q&A at the
    end of the talk
    end of the talk
    end of the talk
    end of the talk
    end of the talk
    Scary at rst perhaps, but take comfort in
    the fact that the rest of the audience is
    rooting for you.
    https://www.pexels.com/photo/monopoly-car-piece-1634213/
    #SurvivorshipBias
    

    View Slide

39. Try to arrange a
    Try to arrange a
    Try to arrange a
    Try to arrange a
    Try to arrange a
    'hallway meeting'
    'hallway meeting'
    'hallway meeting'
    'hallway meeting'
    'hallway meeting'
    Which basically means just bumping into
    them in the hallways!
    https://pxhere.com/en/photo/1454179
    #SurvivorshipBias
    

    View Slide

40. They seem to really like that, by the way!
    #SurvivorshipBias
    

    View Slide

41. How would you
    you
    you
    you
    you solve
    the puzzle?
    https://pxhere.com/en/photo/1207943
    #SurvivorshipBias
    

    View Slide

42. Start tinkering
    Start tinkering
    Start tinkering
    Start tinkering
    Start tinkering
    https://pxhere.com/en/photo/901709
    Learning Through Tinkering (Tom Cools)
    #SurvivorshipBias
    

    View Slide

43. Experiment
    Experiment
    Experiment
    Experiment
    Experiment
    If you cannot reproduce the problem
    and
    the solution, there must be another factor
    at play.
    https://pxhere.com/en/photo/1004202
    #SurvivorshipBias
    

    View Slide

44. Look at the numbers
    Look at the numbers
    Look at the numbers
    Look at the numbers
    Look at the numbers
    Succeeded Failed
    Same approach
    Di erent approach
    →
    What do they tell you?
    https://pxhere.com/en/photo/1437973
    #SurvivorshipBias
    

    View Slide

45. The community needs
    The community needs
    The community needs
    The community needs
    The community needs
    you!
    you!
    you!
    you!
    you!
    Blog, tweet or speak about your failures
    and what you've learned from them!
    #SurvivorshipBias
    

    View Slide

46. Search StackOverflow
    Search StackOverflow
    Search StackOverflow
    Search StackOverflow
    Search StackOverflow
    Blogs and videos will tell you the happy
    path. StackOver ow will tell you the non-
    happy path.
    https://pxhere.com/en/photo/625549
    #SurvivorshipBias
    

    View Slide

47. Master Your Tools
    Master Your Tools
    Master Your Tools
    Master Your Tools
    Master Your Tools
    Use your tools long enough to
    really get to
    know them.
    https://pxhere.com/en/photo/704019
    #SurvivorshipBias
    

    View Slide

48. Maven's Dependency Tree
    mvn dependency:tree | grep spring-batch
    mvn dependency:tree | grep spring-batch -C 3
    mvn dependency:tree -Dincludes=*:spring-batch*:*
    #SurvivorshipBias
    

    View Slide

49. Tempting
    Tempting
    Tempting
    Tempting
    Tempting
    Abraham Maslow
    Toward a Psychology of Being
    "I suppose it is tempting, if the only tool
    you have is a hammer, to treat everything
    as if it were a nail."
    https://www.pexels.com/photo/black-claw-hammer-on-brown-wooden-plank-209235/
    #SurvivorshipBias
    

    View Slide

50. Take Aways
    Take Aways
    Take Aways
    Take Aways
    Take Aways
    1. Share your failures and lessons learned!
    2. Start tinkering!
    3. See you in the Hallway track!
    #SurvivorshipBias
    

    View Slide

51. Thank you!
    bit.ly/survivorship-bias-jcon
    hanno.codes maarten.mulders.it
    @hannotify @mthmulders
    #SurvivorshipBias
    

    View Slide