Neptune_Facebook_Autoremediation_talk

Talks by Neptune and Facebook (FBAR) on automated diagnosis and remediation at scale.

https://www.meetup.com/Auto-Remediation-and-Event-Driven-Automation/

Transcript

1. Reducing incident MTTR via automated diagnosis and remediation Auto-remediation & Event Driven Automation Meetup Kiran Gollu, Founder Neptune.io © 2017
2. Neptune Neptune.io © 2017 2 •  One-stop incident automation-as-a-service for DevOps •  Less Alerts, More Uptime & More sleep •  Highly scalable (millions of alerts), secure, and reliable (tolerate data center region wide outages) Trusted by Backed by
3. Me Neptune.io © 2017 •  Present: •  Founder, Neptune •  YCombinator and Data Collective VC backed startup •  Past: •  Founding engineer/architect at AWS S3, and DynamoDB •  Published research papers on distributed systems collaborating with Microsoft Research
4. What I will to talk about •  State of incident response today •  Alert enrichment and auto-remediation techniques / tradeoffs •  How do you solve this problem for your company? •  Demo Neptune •  Our learning's & recommendations Neptune.io © 2017
5. What is Incident Response? How to handle alerts/incidents/ outages? Many more.. Alerts Neptune.io © 2017
6. Maturity level of incident response teams @jpaulreed @kfinnbraun DevOps Enterprise Summit Neptune.io © 2017
7. Problem#1: Lots of alerts 7 Neptune.io © 2017
8. Problem#2: Failures are complicated Neptune.io © 2017
9. Problem#3: Debugging is hard! 9
10. Source : DevOps survey; Victor Ops incident response #4: 95% of Time To Recovery(TTR) is still manual today Alert Troubleshooting Triage | Investigate | Identify Resolution Documentation 73% 10% 5% 12% Snapshots •  Graphs & metrics •  Logs •  Webpages Service health checks •  Internal •  External Host/App diagnostics •  “Top”, “df –H” etc. •  Heap dumps/Stack traces Runbooks •  On single/cluster of hosts •  Any script, any language Cloud API/CLI actions •  Start/Stop/ Reboot •  Scale up/down Root-cause analysis & Audit •  Heap dumps •  Logs •  Graphs Post-mortem •  History •  Diagnostics Neptune.io © 2017
11. FBAR : Facebook Auto Remediation Platform “…Its doing the work of approximately 200 sys admins…” “We built an internal tool for AWS” Nurse: Auto-remediation platform “60% of problems are fixed automatically…” Winston: Event driven automation tool 11 Neptune.io © 2017
12. Your Options Neptune.io © 2017 SaaS product, On-Premise offering on AWS, deep integrations with monitoring tools Open source event driven automation Build it in house
13. Common Issues •  Noise •  non-actionable alerts •  false positives •  self-recovering alerts •  Engineer burnout: Too much manual work •  Not measuring cost of dealing with incidents •  Incorrect monitoring thresholds •  Bandaids instead of root causing problems •  Alert correlation is hard – downstream/upstream dependencies •  Not having clear incident response and escalation processeses Neptune.io © 2017
14. Architecting automated diagnosis and remediation Neptune.io © 2017
15. 1. One-stop incident tracking •  Helps identify those top-20% alerts causing 80% of pain •  Sorted by frequency and MTTR •  Capture: •  MTTA (mean time to acknowledgement) •  MTTR (mean time to resolution) •  Frequency of occurrence (#times a particular alert has occurred) •  Reporting + Auditing •  Audit all activity (both manual + automated) •  Leads to data-driven post mortems Neptune.io © 2017
16. 2. Enrich Alerts •  When an alert occurs: •  Gather debugging context automatically from different tools Use cases: 1.  High memory alert •  capture top-10 memory hogs, thread dump, memory usage graphs 2.  High app error rate •  capture error rate & latency trends •  app logs for 5xx errors from Splunk/Sumologic •  app health checks Neptune.io © 2017
17. 3. Auto-Remediate repetitive alerts •  When an alert occurs: •  If it’s a known alert ! Run a remediation runbook •  Use cases: •  Process crashed ! restart process •  Host is unpingable ! restart 3 times and escalate if still fails •  Service is down ! capture graphs, run a remediation workflow Neptune.io © 2017
18. Cultural aspects •  Have a clear incident resolution and escalation process •  Break silos : Dev / Ops can resolve and share how incidents are resolved •  Document and version your runbooks •  Single consolidated report per each incident to make post-mortems easy •  Audit all manual and automated actions for an incident •  Use your own communication tools (Slack, HipChat) but record incident logs •  Use tools to log team collaboration activity Neptune.io © 2017
19. Neptune Demo Neptune.io © 2017
20. How do we ensure military grade security? Neptune.io © 2017
21. Our learning’s from 200+ SRE/Ops teams •  Automate simple things first •  Have checks in place to avoid cascading failures •  Rate limiting, handling correlated failures •  Capture state and snapshots for self-recovering alerts •  Don’t automatically fix when you don’t know root cause •  Enriching incidents is as important as automating repetitive incidents •  Availability of automation tool should be >>> your apps Neptune.io © 2017
22. What I talked about •  Reducing MTTR gives you: •  More agility ! frees up engineers to work on business critical problems •  More uptime, thus better customer experience •  More sleep, Happier engineers •  To reduce MTTR for production alerts incidents: •  create actionable alerts and fix monitoring thresholds •  Embrace automation – enrich and automate alerts •  Instill good incident response processes Neptune.io © 2017
23. Signup for 14 day free trail! kiran@neptune.io Neptune.io © 2017