Save 37% off PRO during our Black Friday Sale! »

Who Watches The Watchmen (Updated)

Who Watches The Watchmen (Updated)

This was a revised version of my talk on how we monitor PagerDuty. This was presented at DevOpsDays Chicago 2014. Video link:
https://www.youtube.com/watch?v=jUfd3XmJtfo

Ebe1d126c7c859171156efb4c08db14f?s=128

Arup Chakrabarti

October 08, 2014
Tweet

Transcript

  1. Who Watches the Watchmen? DevOps Days Chicago 2014 Arup Chakrabarti

    @arupchak or arup@pagerduty.com
  2. @arupchak What is PagerDuty? 10/23/14 Ops Guys know all too

    well... •  Alert and Incident Tracking •  On-Call Management •  Integrates with monitoring tools •  Alert the right person, every time Who Watches the Watchmen? DevOps Days Chicago 2014
  3. @arupchak What is PagerDuty? 10/23/14 Who Watches the Watchmen? DevOps

    Days Chicago 2014
  4. @arupchak Why do we care about monitoring? 10/23/14 Oct 2014

    US East Outage – Outgoing Traffic Who Watches the Watchmen? DevOps Days Chicago 2014
  5. @arupchak Today’s talk is about: 10/23/14 •  What is PagerDuty?

    •  Philosophies •  Tools •  Security •  Distributed Systems •  Dependency •  How we cheat by using Chef •  Validation •  Q and A Who Watches the Watchmen? DevOps Days Chicago 2014
  6. @arupchak Quick Disclaimer 10/23/14 I did not come up with

    everything Who Watches the Watchmen? DevOps Days Chicago 2014 •  I work with smart people •  Slides will be posted
  7. @arupchak Philosophies 10/23/14 Thou Shall: •  Use the right tool

    •  Avoid single host monitoring Who Watches the Watchmen? DevOps Days Chicago 2014
  8. @arupchak Philosophies 10/23/14 Thou Shall: •  Alert on what customers

    care about Who Watches the Watchmen? DevOps Days Chicago 2014
  9. @arupchak Philosophies 10/23/14 Thou Shall: •  Alert on expected values

    •  High and Low Who Watches the Watchmen? DevOps Days Chicago 2014
  10. @arupchak Philosophies 10/23/14 Thou Shall: •  Make it Self-Service Who

    Watches the Watchmen? DevOps Days Chicago 2014
  11. @arupchak Philosophies 10/23/14 Thou Shall: •  Validate Alerts Work Who

    Watches the Watchmen? DevOps Days Chicago 2014
  12. @arupchak Monitoring Tools New Relic 10/23/14 Who Watches the Watchmen?

    DevOps Days Chicago 2014
  13. @arupchak New Relic Oooooo Graphs 10/23/14 Who Watches the Watchmen?

    DevOps Days Chicago 2014
  14. @arupchak New Relic Stacked lines! 10/23/14 Who Watches the Watchmen?

    DevOps Days Chicago 2014
  15. @arupchak New Relic Lines AND Bars! 10/23/14 Who Watches the

    Watchmen? DevOps Days Chicago 2014
  16. @arupchak New Relic Reports! 10/23/14 Who Watches the Watchmen? DevOps

    Days Chicago 2014
  17. @arupchak New Relic Moar Reports! 10/23/14 Who Watches the Watchmen?

    DevOps Days Chicago 2014
  18. @arupchak New Relic / APMs 10/23/14 Great for small env’s

    •  Pros •  Great for new stacks •  Helpful for tracing transactions •  Gives a lot of data Who Watches the Watchmen? DevOps Days Chicago 2014
  19. @arupchak Problem with APMs 10/23/14 Not a silver bullet • 

    Cons •  They can be overly prescriptive •  They can be hard to tune/ customize •  Gives a lot of data Who Watches the Watchmen? DevOps Days Chicago 2014
  20. @arupchak StatsD / DataDog 10/23/14 All hail self service metrics

    •  StatsD is the client •  DataDog is the backend •  Super easy to use •  statsd.gauge(metric_name, val) •  statsd.counter(metric_name) •  statsd.histogram(metric_name,val) Who Watches the Watchmen? DevOps Days Chicago 2014
  21. @arupchak StatsD / DataDog Custom Alerts 10/23/14 Who Watches the

    Watchmen? DevOps Days Chicago 2014
  22. @arupchak StatsD / DataDog 10/23/14 Custom Notifications •  PagerDuty Integration

    •  Email •  HipChat Who Watches the Watchmen? DevOps Days Chicago 2014
  23. @arupchak StatsD / DataDog 10/23/14 Customize all the things • 

    Pros •  Very customizable •  Can change as you grow •  Self Service Who Watches the Watchmen? DevOps Days Chicago 2014
  24. @arupchak StatsD / DataDog 10/23/14 Needs some hand holding • 

    Cons •  Need to have Configuration Management •  Hard to ramp teams up Who Watches the Watchmen? DevOps Days Chicago 2014
  25. @arupchak SumoLogic 10/23/14 Logging as Monitoring •  Ship Critical App

    Logs •  Engineers setup alerts on patterns •  “Too many 500’s in the last 10m” •  Somewhat self-service •  Initial setup is in Chef •  Hard to use for realtime debugging Who Watches the Watchmen? DevOps Days Chicago 2014
  26. @arupchak PagerDuty at PagerDuty? 10/23/14 Who Watches the Watchmen? DevOps

    Days Chicago 2014
  27. @arupchak PagerDuty at PagerDuty? 10/23/14 Who Watches the Watchmen? DevOps

    Days Chicago 2014 YES
  28. @arupchak Simple External Monitoring 10/23/14 Dumb health checks •  Wormly

    and Monitis •  Simple tools •  Backup alerting •  Very naive in the health checks •  Had to build out smarter health check page Who Watches the Watchmen? DevOps Days Chicago 2014
  29. @arupchak Simple External Monitoring 10/23/14 Dumb health checks made smarter

    •  Health Check Page •  Lightly touches internal services •  Gives back an expected value for each service •  Alert on non-expected value Who Watches the Watchmen? DevOps Days Chicago 2014
  30. @arupchak Security Monitoring 10/23/14 Why do security monitoring? •  Audits

    are tedious •  Continuous Audits •  Earlier alerting •  Easier fixing Who Watches the Watchmen? DevOps Days Chicago 2014
  31. @arupchak Security Monitoring 10/23/14 Audits are tedious •  IDS via

    OSSEC •  Monitor Logs / Checksum Dir’s •  Port scanning •  nmap •  Scrape IPSec data Who Watches the Watchmen? DevOps Days Chicago 2014
  32. @arupchak Distributed Systems 10/23/14 The single host does not matter

    anymore •  Alert on cluster level metrics •  Overall number of 500’s •  % of nodes down •  Overall latency Who Watches the Watchmen? DevOps Days Chicago 2014
  33. @arupchak Avoid Single Host Alerts 10/23/14 Crons should not be

    used for creating alerts Who Watches the Watchmen? DevOps Days Chicago 2014 PAGERDUTY US West 1 US West 2 Linode Monitoring System
  34. @arupchak Same model for service alerting 10/23/14 Who Watches the

    Watchmen? DevOps Days Chicago 2014 PAGERDUTY Service A Service B Service C Monitoring System
  35. @arupchak Dependency Monitoring 10/23/14 Stuff that you do not control

    •  Dependencies Everywhere •  Operations •  DNS •  Monitoring Tools •  Logging Who Watches the Watchmen? DevOps Days Chicago 2014
  36. @arupchak Dependency Monitoring 10/23/14 Stuff that you do not control

    •  How to monitor? •  Operations •  DNS -> Create/Delete records •  Monitoring Tools -> Basic ping •  Logging -> Validate that logs are being pushed •  Status Pages Who Watches the Watchmen? DevOps Days Chicago 2014
  37. @arupchak Dependency Monitoring 10/23/14 What keeps us up at night

    •  Dependencies Everywhere •  Software •  Email •  SMS •  Phone •  Push Notifications Who Watches the Watchmen? DevOps Days Chicago 2014
  38. @arupchak Quick Story 10/23/14 When SMS providers screw us over

    •  Primary SMS provider was “Up” •  Customer was not getting their SMS •  Found out in the worst way possible •  Customer called us •  Provider was working but T-Mobile prepaid was not passing our short code through Who Watches the Watchmen? DevOps Days Chicago 2014
  39. @arupchak End to End testing 10/23/14 aka how to abuse

    unlimited messaging plans •  Every minute we send a SMS alert •  Every SMS provider we use •  Main Carriers •  Verizon •  AT&T •  T-Mobile •  Sprint •  Measure Response times Who Watches the Watchmen? DevOps Days Chicago 2014
  40. @arupchak “Device Lab” It looks more official now 10/23/14 Who

    Watches the Watchmen? DevOps Days Chicago 2014
  41. @arupchak Some stats (Averages) 10/23/14 Sorry, cannot tell you which

    carrier is which •  Carrier A •  15 Seconds •  Carrier B •  5 Seconds •  Carrier C •  15 Seconds •  Carrier D •  50 Seconds Who Watches the Watchmen? DevOps Days Chicago 2014
  42. @arupchak Very Spikey 10/23/14 Who Watches the Watchmen? DevOps Days

    Chicago 2014
  43. @arupchak How to cheat with Chef 10/23/14 Automate all the

    things •  Install all the agents •  New Relic •  DataDog – Easy alerts as well •  SumoLogic •  OSSEC •  Backup alerts are not automated •  Cluster alert setup is not automated Who Watches the Watchmen? DevOps Days Chicago 2014
  44. @arupchak How to Validate 10/23/14 Failure Friday •  We attack

    our own services •  Process failure •  Datacenter failure •  Network failure •  https://blog.pagerduty.com/failure-friday- at-pagerduty Who Watches the Watchmen? DevOps Days Chicago 2014
  45. @arupchak What we have learned 10/23/14 Failure Friday •  Process

    monitoring is co-mingled with the process running •  Only localhost checks on service •  Alerts require outbound network conn Who Watches the Watchmen? DevOps Days Chicago 2014
  46. Thank you. We are Hiring! pagerduty.com/jobs @arupchak or arup@pagerduty.com