Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Fallible Humans: Dealing With Failure In The Absence Of Scapegoats

Fallible Humans: Dealing With Failure In The Absence Of Scapegoats

Functional siloing - "dev", "ops", etc. - make it easier to pass the buck and blame others when things go wrong. In an environment where dev, ops, and other business functions co-operate and work together, it's not so easy to fall back on traditional scapegoats. DevOps approaches encourage us to move away from the harmful (and counterproductive) apportioning of blame for failure, but it takes effort and commitment from all involved.

I'll discuss the philosophy and motivations of "blamelessness", along with ways to understand the vital role humans play in making complex socio-technical systems safe, even when things go awry, along with advice on how to promote and foster a blameless culture in your workplace, and techniques for analyzing and learning from failures without resorting to finger-pointing and scapegoating.

This talk was originally given at DevOps Days Minneapolis on 18 July, 2014: http://devopsdays.org/events/2014-minneapolis/

A transcript is available at http://ind.ec/fallible

Ian Malpass

July 18, 2014
Tweet

More Decks by Ian Malpass

Other Decks in Technology

Transcript

  1. Fallible Humans
    Dealing With Failure In The Absence Of Scapegoats
    Ian Malpass, Etsy
    @indec

    View full-size slide

  2. "Jerusalem Ugglan 1". Licensed under Public domain via Wikimedia Commons - http://commons.wikimedia.org/wiki/File:Jerusalem_Ugglan_1.jpg

    View full-size slide

  3. By Rodhullandemu (Own work) [CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0) or GFDL (http://www.gnu.org/copyleft/fdl.html)], via Wikimedia Commons
    GET IN THE CAR
    GET IN THE CAR
    NO TIME TO EXPLAIN
    NO TIME TO EXPLAIN

    View full-size slide

  4. “Fire of Troy" by Kerstiaen de Keuninck (Coninck) - http://fotki.yandex.ru/users/fchstudents/view/133880?page=69. Licensed under Public domain via Wikimedia Commons

    View full-size slide

  5. It’s not me, it’s you

    View full-size slide

  6. Dev
    Ops
    Legal
    Marketing
    Support
    Finance
    Security

    View full-size slide

  7. Dev Ops
    Legal
    Marketing
    Support
    Finance
    Security

    View full-size slide

  8. With the unknown, one is confronted with
    danger, discomfort, and care; the first instinct
    is to abolish these painful states. First
    principle: any explanation is better than none.
    !
    Friedrich Nietzsche, The Twilight Of The Idols

    View full-size slide

  9. A person who has been punished is not thereby
    simply less inclined to behave in a given way;
    at best, he learns how to avoid punishment.
    !
    B. F. Skinner, Beyond Freedom and Dignity

    View full-size slide

  10. Failure is an emergent
    property of a complex system

    View full-size slide

  11. Chalk Farm Banksy by grahamc99 https://flic.kr/p/39M4pM

    View full-size slide

  12. We should learn from
    our mistakes

    View full-size slide

  13. You don’t come to work
    to do a bad job

    View full-size slide

  14. Get the
    feature out on
    time
    Don’t have
    any bugs
    Don’t use too
    many
    resources

    View full-size slide

  15. Monitor
    everything
    Alert
    appropriately
    Avoid alert
    fatigue

    View full-size slide

  16. Efficiency
    !
    !
    !
    Thoroughness

    View full-size slide

  17. If I'd observed all the rules,
    I'd never have got anywhere.
    !
    Marilyn Monroe

    View full-size slide

  18. Every day I’m ETTOing

    View full-size slide

  19. Trust and understanding

    View full-size slide

  20. Don’t blame yourself

    View full-size slide

  21. Blamelessness is more
    fun with friends

    View full-size slide

  22. Practice makes perfect
    as close to perfect
    as we can hope
    for in a complex
    system based on
    reasonable
    expectations

    View full-size slide

  23. Looking back, we should
    have eliminated hindsight

    View full-size slide

  24. Could have,
    would have,
    should have

    View full-size slide

  25. There is no root cause

    View full-size slide

  26. Seek good remediation items

    View full-size slide

  27. Process is an embedded reaction to prior stupidity
    !
    Clay Shirky, Wikis, Grafitti, and Process

    View full-size slide

  28. Communicate and advocate

    View full-size slide

  29. etsy.me/justculture

    View full-size slide