Big Red Button (Chatbots SG Nov 2019)

780e86312035da00762813aa2e443ae8?s=47 Amy Nguyen
November 07, 2019

Big Red Button (Chatbots SG Nov 2019)

When an incident starts, ten different things need to happen at once. You need to get an incident commander, you need to get all the right people in the room, you need to mitigate the incident, and you need to stay organized. At Stripe, we've built a tool for automating as much of the routine tasks as possible so responders can focus on what humans do best. In this talk, I'll show you the Big Red Button, a web form that sends emails, creates JIRA tickets, opens Slack channels, sends pages, and more. We'll talk about the unique constraints of this tool (such as, how much incident metadata do you ask for up-front?) and how our incident management philosophy influenced our design.

780e86312035da00762813aa2e443ae8?s=128

Amy Nguyen

November 07, 2019
Tweet

Transcript

  1. 2.

    @amyngyn What's Stripe? 2 • Stripe builds economic infrastructure for

    the Internet • Security and reliability are the most important values we can provide to our users • We started an engineering hub in Singapore last year! Who are you? • I'm a software engineer living here in Singapore developing payment APIs • I was on the Reliability Tooling team in San Francisco • Find me online at amy.dev and on twitter @amyngyn
  2. 4.

    @amyngyn 4 What do you do when something's broken? •

    Page someone • Find the largest conference room and take it over • Scream • Fix the problem
  3. 5.

    @amyngyn 5 What about... • Announce where incident firefighting is

    happening • Update your company's status page • Update stakeholders and leadership • Create a ticket to record the incident • Document the incident timeline • Write a public-facing retrospective • Track remediation items • Sending messages to users • Lock deploys
  4. 42.

    @amyngyn 42 slack service email service abstract service […] service

    incident close event handler recovery steps incident [...] event handler