Upgrade to Pro — share decks privately, control downloads, hide ads and more …

nrrd 911 ic me: The Incident Commander Role

nrrd 911 ic me: The Incident Commander Role

Shit hit the fan—now what?

You know to build resilient systems and make small, planned changes, but computers (and humans) still fail. How do you deal with such failures? How do you recover?

Enter the Incident Commander. Adapted from the government and military’s incident response process, the Incident Commander handles the technical triage and orchestration necessary to get a swift resolution during crisis. The IC process focuses on clear communication, delegation, and trust between teams working in harmony.

New Relic has used the IC process for over two years, iterating and refining the process as we go. We train all our engineers to be ICs and have used this process to handle small deployment hiccups to network outages. We’ve built tools to support and archive our incident responses and have seen significant improvement in our understanding and response to such situations.

This talk will discuss the IC role, why you want it, how we iterated over it, lessons learned in the field, and the tools we built to support it.

Alice Goldfuss

April 07, 2016
Tweet

More Decks by Alice Goldfuss

Other Decks in Technology

Transcript

  1. Confidential ©2008-15 New Relic, Inc. All rights reserved.
    nrrd 911 ic me:
    The Incident Commander role
    1
    Alice Goldfuss
    @alicegoldfuss

    View Slide

  2. Confidential ©2008-15 New Relic, Inc. All rights reserved.
    I’m Alice
    2
    SRE @

    View Slide

  3. Confidential ©2008-15 New Relic, Inc. All rights reserved. 3

    View Slide

  4. Confidential ©2008-15 New Relic, Inc. All rights reserved. 4
    Things break

    View Slide

  5. Confidential ©2008-15 New Relic, Inc. All rights reserved.
    Who?
    What?
    Where?
    When?
    Why?
    How?
    5

    View Slide

  6. Confidential ©2008-15 New Relic, Inc. All rights reserved.
    Who?
    6
    What?
    Where?
    When?
    Why?
    How?

    View Slide

  7. Confidential ©2008-15 New Relic, Inc. All rights reserved. 7
    The Incident Command System

    View Slide

  8. Confidential ©2008-15 New Relic, Inc. All rights reserved.
    In 2004
    8

    View Slide

  9. Confidential ©2008-15 New Relic, Inc. All rights reserved. 9

    View Slide

  10. Confidential ©2008-15 New Relic, Inc. All rights reserved. 10
    TL CL
    IC
    EC

    View Slide

  11. Confidential ©2008-15 New Relic, Inc. All rights reserved. 11
    Incident Commander

    View Slide

  12. Confidential ©2008-15 New Relic, Inc. All rights reserved.
    The Incident Commander
    ▪ Does NOT fix the problem

    ▪ but knows the systems involved

    ▪ Keeps pulse on entire effort

    ▪ A trained volunteer

    ▪ Handles internal communication
    12

    View Slide

  13. Confidential ©2008-15 New Relic, Inc. All rights reserved. 13
    Technical Lead(s)

    View Slide

  14. Confidential ©2008-15 New Relic, Inc. All rights reserved.
    The Technical Lead(s)
    ▪ Fix the problem

    ▪ Update the IC on progress

    ▪ Run impactful changes by IC
    14

    View Slide

  15. Confidential ©2008-15 New Relic, Inc. All rights reserved.
    Communications Lead
    15

    View Slide

  16. Confidential ©2008-15 New Relic, Inc. All rights reserved.
    The Communications Lead
    ▪ Acts as link to public/customers

    ▪ Translates technical details to
    consumable statuses

    ▪ Updates IC on customer
    communication

    ▪ Handles external communications
    16

    View Slide

  17. Confidential ©2008-15 New Relic, Inc. All rights reserved. 17
    Severity Levels

    View Slide

  18. Confidential ©2008-15 New Relic, Inc. All rights reserved. 18
    Severity Levels
    5 Everything is ok…for now

    View Slide

  19. Confidential ©2008-15 New Relic, Inc. All rights reserved. 19
    Severity Levels
    4 A thing is smoldering
    5 Everything is ok…for now

    View Slide

  20. Confidential ©2008-15 New Relic, Inc. All rights reserved. 20
    Severity Levels
    3 A part of a thing exploded
    4 A thing is smoldering
    5 Everything is ok…for now

    View Slide

  21. Confidential ©2008-15 New Relic, Inc. All rights reserved. 21
    Severity Levels
    2 One thing exploded
    3 A part of a thing exploded
    4 A thing is smoldering
    5 Everything is ok…for now

    View Slide

  22. Confidential ©2008-15 New Relic, Inc. All rights reserved. 22
    Severity Levels
    1 Everything exploded
    2 One thing exploded
    3 A part of a thing exploded
    4 A thing is smoldering
    5 Everything is ok…for now

    View Slide

  23. Confidential ©2008-15 New Relic, Inc. All rights reserved. 23
    TL LL CL
    IC
    EC

    View Slide

  24. Confidential ©2008-15 New Relic, Inc. All rights reserved. 24
    Why?

    View Slide

  25. Confidential ©2008-15 New Relic, Inc. All rights reserved. 25
    I got this

    View Slide

  26. Confidential ©2008-15 New Relic, Inc. All rights reserved. 26
    Squad Goals

    View Slide

  27. Confidential ©2008-15 New Relic, Inc. All rights reserved. 27
    Distributed Systems

    View Slide

  28. Confidential ©2008-15 New Relic, Inc. All rights reserved. 28
    ???

    View Slide

  29. Confidential ©2008-15 New Relic, Inc. All rights reserved.
    Misallocated Resources
    29

    View Slide

  30. Confidential ©2008-15 New Relic, Inc. All rights reserved. 30
    Organized Effort

    View Slide

  31. Confidential ©2008-15 New Relic, Inc. All rights reserved.
    Why the ICS?
    ▪ Prevents panic

    ▪ Coordinates efforts

    ▪ Maintains reliable line of
    communication

    ▪ Allows for best possible incident
    resolution
    31

    View Slide

  32. Confidential ©2008-15 New Relic, Inc. All rights reserved. 32
    How?

    View Slide

  33. Confidential ©2008-15 New Relic, Inc. All rights reserved. 33
    Training

    View Slide

  34. Confidential ©2008-15 New Relic, Inc. All rights reserved. 34
    Train everyone

    View Slide

  35. Confidential ©2008-15 New Relic, Inc. All rights reserved.
    Training Plan
    ▪ Coordinate IC/CL sessions

    ▪ Roleplay/hands-on activities

    ▪ Offer refreshers
    35

    View Slide

  36. Confidential ©2008-15 New Relic, Inc. All rights reserved. 36
    Tools

    View Slide

  37. Confidential ©2008-15 New Relic, Inc. All rights reserved. 37
    hubot.github.com

    View Slide

  38. Confidential ©2008-15 New Relic, Inc. All rights reserved. 38

    View Slide

  39. Confidential ©2008-15 New Relic, Inc. All rights reserved. 39

    View Slide

  40. Confidential ©2008-15 New Relic, Inc. All rights reserved. 40

    View Slide

  41. Confidential ©2008-15 New Relic, Inc. All rights reserved. 41

    View Slide

  42. Confidential ©2008-15 New Relic, Inc. All rights reserved. 42

    View Slide

  43. Confidential ©2008-15 New Relic, Inc. All rights reserved.
    Other Tools
    ▪ Upboard

    ▪ Google docs / Quip

    ▪ New Relic products

    ▪ Blameless retros
    43

    View Slide

  44. Confidential ©2008-15 New Relic, Inc. All rights reserved. 44
    Lessons learned

    View Slide

  45. Confidential ©2008-15 New Relic, Inc. All rights reserved. 45

    View Slide

  46. Confidential ©2008-15 New Relic, Inc. All rights reserved. 46
    Tools break

    View Slide

  47. Confidential ©2008-15 New Relic, Inc. All rights reserved. 47

    View Slide

  48. Confidential ©2008-15 New Relic, Inc. All rights reserved. 48
    Worth it?

    View Slide

  49. Confidential ©2008-15 New Relic, Inc. All rights reserved.
    Thanks!
    49
    @alicegoldfuss

    View Slide

  50. Confidential ©2008-15 New Relic, Inc. All rights reserved. 50
    This document and the information herein (including any information that may be incorporated by reference) is
    provided for informational purposes only and should not be construed as an offer, commitment, promise or
    obligation on behalf of New Relic, Inc. (“New Relic”) to sell securities or deliver any product, material, code,
    functionality, or other feature. Any information provided hereby is proprietary to New Relic and may not be
    replicated or disclosed without New Relic’s express written permission.

    Such information may contain forward-looking statements within the meaning of federal securities laws. Any
    statement that is not a historical fact or refers to expectations, projections, future plans, objectives, estimates,
    goals, or other characterizations of future events is a forward-looking statement. These forward-looking
    statements can often be identified as such because the context of the statement will include words such as
    “believes,” “anticipates,” “expects” or words of similar import.

    Actual results may differ materially from those expressed in these forward-looking statements, which speak
    only as of the date hereof, and are subject to change at any time without notice. Existing and prospective
    investors, customers and other third parties transacting business with New Relic are cautioned not to place
    undue reliance on this forward-looking information. The achievement or success of the matters covered by
    such forward-looking statements are based on New Relic’s current assumptions, expectations, and beliefs and
    are subject to substantial risks, uncertainties, assumptions, and changes in circumstances that may cause the
    actual results, performance, or achievements to differ materially from those expressed or implied in any
    forward-looking statement. Further information on factors that could affect such forward-looking statements is
    included in the filings we make with the SEC from time to time. Copies of these documents may be obtained
    by visiting New Relic’s Investor Relations website at ir.newrelic.com or the SEC’s website at www.sec.gov.

    New Relic assumes no obligation and does not intend to update these forward-looking statements, except as
    required by law. New Relic makes no warranties, expressed or implied, in this document or otherwise, with
    respect to the information provided.

    View Slide

  51. Confidential ©2008-15 New Relic, Inc. All rights reserved. 51
    1 https://www.flickr.com/photos/voxaeterno/14237475601/
    2 https://www.flickr.com/photos/nicoguaro/15277730776/
    4 https://upload.wikimedia.org/wikipedia/commons/9/96/ShadowRidgeRoadFire.JPG
    7 https://www.flickr.com/photos/rusty_clark/8300584752/
    8 https://www.flickr.com/photos/usfwssoutheast/4971832860/
    11 https://www.flickr.com/photos/dfmagazine/13597941983/
    13 https://www.flickr.com/photos/cfccreates/10578747285/
    15 https://www.flickr.com/photos/[email protected]/20828632455
    24 https://www.flickr.com/photos/[email protected]/15785915797
    25 https://www.flickr.com/photos/freakingnoob/3438012333
    26 https://www.flickr.com/photos/montanapets/7298181070/
    27 https://www.flickr.com/photos/peerlawther/6806367080/
    28 https://www.flickr.com/photos/montanapets/7298363036/
    29 https://www.flickr.com/photos/[email protected]/21584903408
    30 https://www.flickr.com/photos/arbutusridge/8672496907/ (Arbutus Photography)
    32 https://www.flickr.com/photos/wscullin/3770016707
    34 https://www.flickr.com/photos/seeminglee/9542930433/
    36 https://en.wikipedia.org/wiki/Hydraulic_rescue_tools#/media/File:Spreizer_schlossoeffnung.jpg
    47 computer https://www.flickr.com/photos/theyoungthousands/2482389516/
    48 https://www.flickr.com/photos/spam/3355834452
    All other images in the public domain

    View Slide