SRE NEXT 2022: Sensible Incident Management for Software Startups

SRE NEXT 2022
Sensible Incident Management for So4ware Startups
Takayuki Watanabe
@Launchable, Inc.

View Slide

Who?
Name: Takayuki Watanabe
Affiliation: Launchable, Inc.
Role: Software Engineer
Sns:
Blog: blog.takanabe.tokyo
GitHub: takanabe
Twitter: @takanabe_w
Interests:
- Developer Productivity
- Site Reliability Engineering
- Sustainability Engineering
SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 2

View Slide

Your takeaways
You can understand:
• Incident management has a life cycle.
• Incident response roles and structures exist to embody 3T mental models.
• Choosing strategies and tools makes incident managements at startups sensible.

View Slide

Out of scope
• Fundamental SRE terminology (e.g. SLO, SLI, Error budget, Postmortem)

View Slide

Disclaimer
• This session refers a lot of exis0ng incident management and SRE prac0ces.
• But contains a lot of opinionated ideas and philosophy as well.
• So, the ideas might contradict to some people's.
• Let's discuss on TwiAer using #srenext with @takanabe_w

View Slide

Today's agenda
• About Launchable
• Does a startup need incident management?
• Dissect incident management prac8ces.
• 3T mental models and life cycles
• How can we improve incident management?
• Choosing right strategies and tools

View Slide

Chapter 1:
About Launchable

View Slide


View Slide

What is Launchable?
A SaaS accelera)ng so.ware development cycles.

View Slide

What is Launchable?
Current focus is machine learning based test selec0ons by:
• Predic(ng a meaningful subset of tests.
• Iden(fying ﬂaky tests.
• Visualizing test trends with metrics.

View Slide

What is Launchable?
e.g. Reordering tests based on likelihood of failures.

View Slide

Our team size
• Launchable is a startup
• 2 CEOs + 15 employees
• So3ware engineer (7 people)
• Product manager
• Marke>ng
• Sales
• etc...

View Slide

Phases and the number of so0ware engineers
Note: the numbers are es/mated by the presenter based on previous experiences.
• Phase 0: Founding ~ 4 so3ware engineers
• Phase 1: 5 ~ 10 so3ware engineers
• Phase 2: 11 ~ so3ware engineers

View Slide

My SRE NEXT 2022 is about ...

View Slide

Incident management at so#ware startups

View Slide

Does a startup need incident management?

View Slide

Yes, it's obvious if products have customers.

View Slide

Do you have enough engineering members?

View Slide

No! but...

View Slide

Learning from previous careers1
• I've worked at various sizes and stages.
• Company A: +300,000 people
• Company B: +400 people (Joined when they had +300 people)
• Company C: +150 people (Joined when they only had less than 10 people)
• Product developments are always the highest priority concerns.
• OperaHon improvement != Product development velocity degradaHon.
• We will never have enough engineering members to improve opera;ons.
Never.
1 SRE NEXT 2020: Designing fault-tolerant microservices with SRE and circuit breaker centric architecture

View Slide

Are speed and quality trade-oﬀ?
• I personally don't think so 2 3.
• I believe sensible incident management accelerates our development velocity.
3 A Philosophy of So.ware Deisgn, Chapter 3: Working Code Isnt' Enough, pp. 13 - 18.
2 mar&nFowler.com: Is High Quality So;ware Worth the Cost?

View Slide

Can we reframe the original ques3on?
• We want to reframe "Does a startup need incident management?" to:
• Which incident management processes won't change even for rapid
developments?
• Which processes should we improve?
• Let's dissect incident management prac=ces in the industry.

View Slide

Chapter 2:
Dissect incident management prac/ces

View Slide

What is incident management?
Incident management
• High level and overall process for handling incidents in an organiza5on.
Incident response
• Part of incident management for actual technical steps including detec5on,
repor5ng, mi5ga5on, and recovery during incidents.

View Slide

Exis%ng prac%ces

View Slide

e.g. Terminology

View Slide

Examples of terminology 4
• CAN Reports
• Deputy
• Execu3ve Swoop
• Grenade Thrower
• Incident Commander (IC)
• Resolver
• Severity
• Scribe
• Subject Ma4er Expert (SME)
4 h$ps:/
/response.pagerduty.com/training/glossary/

View Slide

e.g. Roles

View Slide

Examples of roles at Google 5 6
6 Anatomy of an Incident Google’s Approach to Incident Management for Produc;on Services, Chapter 4: Mi;ga;on and
Recovery, pp. 31-32.
5 Google SRE Workbook, Chapter 9: Incident Response

View Slide

Examples of roles at PagerDuty 7 8
8 Google SRE Workbook, Chapter 9: Incident Response
7 PagerDuty Incident Response Documenta6on, Diﬀerent Roles -

View Slide

Too much!

View Slide

Can we translate these prac.ces
into more higher level concepts?

View Slide

Chapter 3:
3T mental models and life cycles

View Slide

Examples of roles at PagerDuty

View Slide

Command role
• Responsibility is managing incident responses to align in organiza5ons.
• Understand ongoing opera5ons
• Understand who is doing what
• Delegate sub-commander responsibility to others if necessary.
• Make incident response tangible.

View Slide

Liason role
• Responsibility is smooth repor1ng and communica1ons.
• For both internally and externally.
• Make incident response transparent.

View Slide

Opera&on role
• Responsibility is actual technical ac2vi2es to solve issues.
• Focus on triage, analysis, mi2ga2on and recovery.
• Communica2on with rest of organiza2ons is not a primary concern.
• In many cases, operators produce root causes of incidents but don't blame them.
• Nobody wants to cause incidents.
• All par2cipants focus on assigned roles based on chain of trust.

View Slide

3T mental models for incident response
The incident response roles embody 3T mental models.
• Transparency
• Keep informa-on of incident responses reachable for everybody.
• Tangibility
• Manage status of incidents.
• Manage who handles what.
• Trust
• Believe everybody makes best eﬀorts during incidents.
• Don't blame anybody because nobody wants to cause incidents.

View Slide

High level view of incident management cycles

View Slide


View Slide

Examples:
• Incident management policy
• Documenta3on
• Repor3ng mechanism
• Observability
• Aler.ng policy
• Incident response training

View Slide

Examples:
• Aler&ng
• Triage
• Root-cause analysis
• Escala'ons
• Opening war rooms

View Slide

Examples:
• Rollback deployment (mi3ga3on)
• Kill slow queries (mi3ga3on)
• Fix bug (recovery)
• Add index to tables (recovery)

View Slide

Examples:
• Addi%onal triage
• Prepara%on for postmortems
• Postmortems
• Handle ac*on items raised at
postmortems

View Slide

Postmortem vs FtS
h"ps:/
/twi"er.com/takanabe_w/status/1510943694467186699

View Slide

Chapter 4:
How can we improve incident management?

View Slide

Where should we invest our 0me?

View Slide

Key %mes of incident response

View Slide

Key %mes of incident response
• Time to detect (TTD)
• Time to engagement (TTE)
• Time to ﬁx (TTF)
• Time to repair/recovery (TTR)
• Time between failures (TBF)

View Slide

Time to detect (TTD)

View Slide

Time to engagement (TTE)

View Slide

Time to ﬁx (TTF)

View Slide

Time to recovery (TTR)

View Slide

Time between failures (TBF)

View Slide

Which &me do we want to improve?

View Slide

TTD at Launchable
Current status
• We've already had several detec0on mechanisms using Datadog and Sentry.
Solu%on
• Introduc*on of SLO and Error Budget makes our aler*ng criteria more clear.
• But don't forget "Law of diminishing returns" to make decisions.

View Slide

TTE at Launchable
Current status
• Easy enough to no.ce during oﬃce hours at Slack channels.
• We don't have on-call rota.ons ATM, which makes TTE uncontrollable.
Solu%on
• Apply follow-the-sun strategy to cover wide-range hours.
• Introducing on-call rota:ons and pager.
• But we don't feel it's necessary now.

View Slide

TTF at Launchable
Current status
• We don't have enough observability mechanisms
• Depending on each developer's debug skill
• During this window, developers cannot spend .me on product developments.
Solu%on
• Introducing more team-shared observability dashboards.
• Introducing more observability mechanism to drill down root causes.

View Slide

Which &me do we want to improve?
TTF improvement brings us high returns with small eﬀorts.

View Slide

Which do we op+mize? MTTR vs MTBF
• Short MTTR and long MTBF are the best
• Short MTTR but short MTBF
= Incidents frequently occur but are recovered quickly.
• Long MTTR but long MTBF
= Incidents don't occur frequently but once occur, they aren't recovered soon.

View Slide

Startups should focus on MTTR improvement
• There is no evolu.on without high cadence itera.ons at startups.
• TTD and TTE are diﬃcult to improve for us.
• Reducing TTF results in reducing MTTR.

View Slide

Do we have other +mes
we haven't ar+culated?

View Slide

Hidden key )mes of incident life cycles

View Slide


View Slide

Don't underes,mate the ,mes we spend as post-incident ac,vi,es.
• Time to (addi,onal )triage (TTT)
• Time to learn (TTL)
• Time to improvement (TTI)
• Time to prepara,on (TTP)

View Slide

Power ques*on:
Which process do you hate?

View Slide

Which process do you hate?
I personally don't want to spend 0me on the following processes.
• Addi$onal triages to dig root causes.
• Prepara$on for learning ( != I don't like joining postmortem sessions).
• Maintainance of incident management processes.

View Slide

Why addi(onal triages?
• Startups don't have enough observability mechanisms.
• We some:mes cannot ﬁnd root causes (this is acceptable).
• We tend to spend a lot of :me here in that situa:on.

View Slide

Why prepara)on for learning?
• Prepara'on for team-wise learning sessions take 'me.
• Documen'ng for Postmortems.
• Copy & paste dances to create 'meline sca=ered various places.
• Timeline needs to consider 'me-zones.
• There is a gravity which prevent people from announcing incident casually.
• For starups, the most important ac'vi'es are learning as a team.
• If TTL is long, people cannot announce incidents casually.
• As a result, postmortems ruin short MTTR with high cadence learning
itera'ons.

View Slide

Why maintenance of incident management processes?
• Maintainance of incident management processes contains:
• Upda.ng incident management policy.
• Improving incident management structures.
• Upda.ng documents.
• Training people to align with the updates.
• Characteris.cally, incidents don't occur frequently,
• Too tough to memorize incident response processes for everybody.
• In urgent situa.on, people don't read documents.

View Slide

Can we reduce TTI?
• It's depending on ac0on items coming from postmortems.
• No teams can handle all ac0on items we discussed during postmortems.
• Common an0-papriori0es.

View Slide

Approach for unbalanced ac0on items
• Think of engineering members' capacity
• Priori7ze and classify the work9 10
10 Anatomy of an incident management, Chapter 5
9 Postmortem Ac,on Items: Plan the Work and Work the Plan, USENIX SRECon 2017

View Slide

Importance / Size / Urgency (ISU) Matrix
• Assignee's conﬁdence is also valuable to declare.

View Slide

ISU Matrix on GitHub Projects

View Slide

My focus is reduc-on of TTT, TTL, and TTP

View Slide

Chapter 5:
Choosing right strategies and tools

View Slide

Phases and the number of so0ware engineers
Note: the numbers are es/mated by the presenter based on previous experiences.
• Phase 0: Founding ~ 4 so3ware engineers
• Phase 1: 5 ~ 10 so3ware engineers
• Phase 2: 11 ~ so3ware engineers

View Slide

Let's reframe the original ques3on again
• Reframe "Does a startup need incident management?"
• At startups, how can we:
• Build an incident management structure enforcing the 3T mental models?
• Improve the ":mes" of the incident management life cycle?

View Slide

Evolu&on of incident management at Launchable
Improvement target Ac/ons from phase 0 to 1 Ac/ons from phase 1 to 2
Transparency - Encourage push communica3on - Encourage pull communica3on
- Create war rooms
- Share status pages
Tangibility - Automate parts of incident response flow - Automate en3re incident response flow
- Introduce incident lead role
Trust - Introduce blameless culture - Split lead and opera3on roles for complex incidents
Time to Engagement (TTE) - Automate incident announcements - Automate en3re incident response flow
- Introduce on-call rota3ons
- Expand follow-the-sun coverages
Time to Fix (TTF) - Introduce observability - Improve observability
Time to Triage (TTT) - Introduce observability - Improve observability
Time to Learn (TTL) - Introduce postmortem template - Generate postmortem
Time to Prepara3on (TTP) - Create incident management policies - Enforce incident management policies
- Self-service incident response trainings

View Slide

Phase 0: Founding ~ 4 so2ware engineers

View Slide

No strategy
• Product does not have customers.
• We don't need incident responses.
• Build incident management structure based on product growth.
• All members do everything if necessary.

View Slide

Incident management system

View Slide

Phase 1: 5 ~ 10 so-ware engineers

View Slide

Environmental changes from phase 0 to 1
• When products have customers, we need an incident management.
• The more so8ware engineers join, the more incidents happen.

View Slide

Strategy
Make everything simple and easy to follow

View Slide

Incident management changes from phase 0 to 1
Transparency - Encourage push communica/on - Encourage pull communica3on
- Create war rooms
Tangibility - Automate parts of incident response flow - Automate en3re incident response flow
Trust - Introduce blameless culture - Split lead and opera3on roles for complex incidents
Time to Engagement (TTE) - Automate incident announcements - Automate en3re incident response flow

View Slide

Incident management system (phase 0 to 1)

View Slide


View Slide

Founda'on of incident management policies
• We maintain policies on Conﬂuence.

View Slide

Automa'on of incident escala'ons
• We escalate incidents using Slack Workﬂow.
• We handle incidents in Slack channel and Google Meet.

View Slide


View Slide

Introduc)on of postmortem
• We keep all postmortems on Conﬂuence.
• We create a new postmortem page using a Conﬂuence template feature.

View Slide

Very simple and easy to follow

View Slide

Postmortems as strong fact-based data
• Even we cannot solve root causes, you can use the postmortems as data.

View Slide

Can we improve the incident management?

View Slide

e.g. Does this what human should take care?

View Slide

e.g. Does this what human should take care?
• We don't have solid policy but policy does not scale.
• Employees are living in Japan and US.
• Sharing all informa>on on Slack is easy to miss.

View Slide

e.g. Do we need roles?

View Slide

Phase 2: 11 ~ ?? so-ware engineers

View Slide

Environmental changes from Phase 1 to 2
• Our products have more customers.
• The more so3ware engineers join, the more incidents happen.
• Increase of employees and >me-zone gaps make sync and push-style
communica>ons tough.
• In the ﬁrst place, Launchable encourages async and wriEen communica>ons.

View Slide

Strategy
• Enforce incident management policies by so4ware not by documents.
• Involve appropriate people based on pull-style communica8ons.
• Use the current tool chains in the company.
• Too many new tools degrade teams' performance.
• Use Slack as interac?ve communica?on places to keep ﬂow info.
• Use Conﬂuence to keep stock info (non-urgent communica?ons).

View Slide


View Slide

SaaS: incident.io
• h#ps:/
/incident.io/

View Slide

SaaS: Blameless
• h#ps:/
/www.blameless.com/product/incident-resolu8on

View Slide

SaaS: Datadog Incident
• h#ps:/
/www.datadoghq.com/blog/incident-response-with-datadog/

View Slide

SaaS: Grafana Incident
• h#ps:/
/go2.grafana.com/incident-beta-interest.html

View Slide

OSS: monzo/response
OSS version of incident.io
• h#ps:/
/github.com/monzo/response
• h#ps:/
/monzo.com/blog/2019/07/08/how-we-respond-to-incidents/

View Slide

in-house tool: Slack App + Web App
• It's not so diﬃcult to implement Slack App and Web App for this purpose.
• But... I want to use my ?me for other stuﬀ.

View Slide

SaaS vs OSS vs in-house tool
• We want to maximize developers' disposal 5me for product developments.
• We don't want to increase cogni5ve loads.
• OSS and in-house tool needs code and document maintenance.
• OSS and in-house tool needs evangelical ac5vi5es for this type of tools.
• Use SaaS if money allows (Buy, Not Build)
• Salaries for so@ware engineers are way more expensive than SaaS cost.
• SaaS improves their features as their business.
• SaaS maintains documents as product features.

View Slide

incident.io covers wide improvement targets
Transparency - Encourage push communica3on - Encourage pull communica/on
- Create war rooms
Tangibility - Automate parts of incident response flow - Automate en/re incident response flow
Trust - Introduce blameless culture - Split lead and opera/on roles for complex incidents
Time to Engagement (TTE) - Automate incident announcements - Automate en/re incident response flow

View Slide

Central channel for all incidents
incident.io can share all incidents in the speciﬁed Slack channel.

View Slide

Dedicated war rooms (Slack channel)
incident.io handles all tasks we want to complete for incident response
ini4aliza4ons.

View Slide

Dedicated war rooms (Slack channel)
incident.io can assist incident responses.

View Slide

Dedicated Slack channel (closing incident)
At the end of incident responses, incident.io tells us what we need to be done next.

View Slide

Status updates at central channel
incident.io automa-cally syncs the latest status of incidents at the central channel.

View Slide

Postmortem genera,on
incident.io can collect ,melines from war rooms and generates postmortems.

View Slide

We can generate a postmortem documents using incident.io.

View Slide

We can collect *melines from dedicated Slack channels.

View Slide


View Slide

Self-training mode
incident.io has a mode to walk though dummy incident responses on Slack.

View Slide

Introduc)on of lead role
• We need communica-on leads when incidents are complex
• However, for most of incident, a single person can be responsible for opera-ons
and communica-ons.
• So, adding a lead role only is prudent so we don't make incident managements
overly complex.

View Slide

We have more rooms to improve!

View Slide

Recap
• Incident management has a life cycle.
• Prepara6on -> Detec6on -> Recovery -> Post-incident ac6ons -> Prepara6on
• Incident response roles and structures exist to embody 3T.
• Transparency
• Tangibility
• Trust
• Choosing strategy and tools makes incident managements at startups sensible.

View Slide

Thanks

View Slide

References

View Slide

Incident management
1. Atlassian: Understanding incident response roles and responsibili5es, h8ps:/
/www.atlassian.com/incident-management/incident-
response/roles-responsibili5es
2. PagerDuty Incident Response Training, h8ps:/
/response.pagerduty.com/training/overview/.
3. Anatomy of an Incident, Ayelet Sachto, Adrienne Walcer, and Jessie Yang, 2022.
4. US Federal Emergency Management Agency, Emergency Management Ins5tute ICS Resource Center, h8ps:/
/training.fema.gov/
emiweb/is/icsresource/.
5. The Na5onal Ins5tute of Standards and Technology SP 800-61, Computer Security Incident Handling Guide, h8p:/
/dx.doi.org/
10.6028/NIST.SP.800-61r2.
6. Introduc5on: Incident Response overview, Gov UK Na5onal Cyber Security Centre, h8ps:/
/www.ncsc.gov.uk/collec5on/incident-
management/incident-response
7. Incident Review and Postmortem Best Prac5ces, h8ps:/
/newsle8er.pragma5cengineer.com/p/incident-review-best-prac5ces
8. Incident Review Prac5ces [The Pragma5c Engineer Newsle8er], h8ps:/
/docs.google.com/spreadsheets/d/1GPINipdf-
l2H05iKOUbpkrqwlZ61ZCJDnwY5iE8LtRM/edit#gid=0
SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022)
135

View Slide

SRE
1. Google SRE book Chapter 14 - Managing Incidents, h>ps:/
/sre.google/sre-book/managing-incidents/
2. Postmortem AcEon Items: Plan the Work and Work the Plan, Sue Lueder and Betsy Beyer (Google), USENIX SRECon 2017,
h>ps:/
/www.usenix.org/conference/srecon17americas/program/presentaEon/lueder.
3. Google SRE book Chapter 15 - Postmortem Culture: Learning from Failure, h>ps:/
/sre.google/sre-book/postmortem-culture/.
4. Postmortem Metadata Index, h>ps:/
/postmortems.app/.
5. The Art of SLOs, Google Site Reliability Engineering, h>ps:/
/sre.google/resources/pracEces-and-processes/art-of-slos/
6. danluu/post-mortems: A collecEon of postmortems, h>ps:/
/github.com/danluu/post-mortems.
7. Great Incident Review Examples, The PragmaEc Engineer, h>ps:/
/blog.pragmaEcengineer.com/postmortem-best-pracEces/#great-
incident-review-examples
136

View Slide

DevOps performance metrics
1. Accelerate: The Science of Lean So4ware and DevOps: Building and Scaling High Performing Technology OrganizaDons, 2018.
2. GoogleCloudPlaKorm/fourkeys, hNps:/
/github.com/GoogleCloudPlaKorm/fourkeys
3. Are you an Elite DevOps performer? Find out with the Four Keys Project, Google Cloud, hNps:/
/cloud.google.com/blog/products/
devops-sre/using-the-four-keys-to-measure-your-devops-performance
4. DORA DevOps Quick Check., hNps:/
/www.devops-research.com/quickcheck.html
137

View Slide

SaaS and OSS
1. Datadog, h,ps:/
/www.datadoghq.com/blog/incident-response-with-datadog/
2. incident.io, h,ps:/
/incident.io/
3. jeli, h,ps:/
/www.jeli.io/
4. monzo/response, h,ps:/
/monzo.com/blog/2019/07/08/how-we-respond-to-incidents
5. Etsy/morgue, h,ps:/
/github.com/etsy/morgue
138

View Slide

SRE NEXT 2022: Sensible Incident Management for Software Startups

SRE NEXT 2022: Sensible Incident Management for Software Startups

Takayuki WATANABE (渡辺喬之)

More Decks by Takayuki WATANABE (渡辺喬之)

Other Decks in Programming

Featured

Transcript

SRE NEXT 2022: Sensible Incident Management for Software Startups

SRE NEXT 2022: Sensible Incident Management for Software Startups

Takayuki WATANABE (渡辺 喬之)

More Decks by Takayuki WATANABE (渡辺 喬之)

Other Decks in Programming

Featured

Transcript

Takayuki WATANABE (渡辺喬之)

More Decks by Takayuki WATANABE (渡辺喬之)