Save 37% off PRO during our Black Friday Sale! »

PDX 2017 - Roy Rapoport

PDX 2017 - Roy Rapoport

0580d500edfdb2e5e80e4732ac8df1ea?s=128

Monitorama

May 22, 2017
Tweet

Transcript

  1. Monitoring Monitoring Systems at Roy Rapoport - @royrapoport May 22,

    2017
  2. The Hero’s Journey

  3. An ordinary person A software engineer

  4. Receives a call Lets build a monitoring system!

  5. is dubious These specs are silly and you’re silly

  6. is persuaded nonetheless Sure, I’d love to learn Scala!

  7. vanquishes enemies Product Managers Engineering Managers Opinionated Coworkers

  8. undergoes the ordeal Damn, this won’t scale. Also, we probably

    shouldn’t have coded a JS interpreter in Erlang.
  9. returns, successful and transformed This will scale for at least

    a month. Ship it!
  10. The Beginning

  11. Real-Time Operational Insight

  12. None
  13. Monitoring ain’t Alerting

  14. Alerting

  15. Alert Action Server Stream Processing Atlas SQS Hot Checker Cold

    Checker AWS Email Page …
  16. Monitoring

  17. Preamble

  18. 6H 3D 18D Publish MR Query S3 Anatomy of an

    Atlas Deployment Group (ADG) m etrics
  19. Server Proxy us-west-2 Server Proxy us-east-1 eu-west-1 Server Proxy

  20. Server Proxy us-west-2 Server Proxy us-east-1 eu-west-1 Server Proxy ADG

    ADG
  21. Server Proxy us-west-2 Server Proxy us-east-1 eu-west-1 Server Proxy ADG

    ADG iep ADG
 iep ADG
 iep ADG

  22. Server Proxy us-west-2 Server Proxy us-east-1 eu-west-1 Server Proxy ADG

    ADG iep ADG
 iep ADG
 iep ADG
 Critical Critical Critical
  23. Tradeoffs: Fail loud Lower TTD

  24. “This morning around 0504 US/Pacific, the active primary region hosting

    Atlas metrics data began experiencing issues and a number of alerts were triggered across the board.“ February 9, 2017
  25. None
  26. Managing Cost

  27. Cost Doesn’t Matter

  28. None
  29. None
  30. None
  31. “Everything Counts in Large Amounts”

  32. “Everything Counts in Large Amounts” - Depeche Mode

  33. Cardinal Sins

  34. What’s a Metric? • name = cpu • nf.region =

    us-east-1 • nf.zone = us-east-1a • nf.app = api • nf.cluster = api-main • nf.asg = api-main-v103 • nf.node = i-103efbdd • device.operationalName = roku • isSupplementalVideo = false • OnComplete = operation • …
  35. OOPS

  36. Fun Metric Dimensions
 A True Story • IP address (~

    232) • Account ID (~109) • Customer ID (~109) • Execution time, ms (~30K)
  37. None
  38. None
  39. None
  40. None
  41. “The graph on the wall tells the story of it

    all” - Depeche Mode
  42. None
  43. What are you (not) talking about?

  44. The latest thing

  45. None
  46. None
  47. Meet Swaroop

  48. The End

  49. The Beginning

  50. ?

  51. None
  52. Attributions • https://www.flickr.com/photos/kalleboo/2240042641/ • http://imgur.com/gallery/QCrR73q • https://www.flickr.com/photos/35028096@N04/6119259650 • https://www.flickr.com/photos/86639298@N02/9420223184 •

    http://paristext2015.com/2015/06/1-5-degrees-celsius-or-2-degrees-maybe-its-turtles-all- the-way-down/ • https://www.flickr.com/photos/60969081@N00/8292545493 • By Photo courtesy of National Nuclear Security Administration / Nevada Site Office - This image is available from the National Nuclear Security Administration Nevada Site Office Photo Library under number XX-34.This tag does not indicate the copyright status of the attached work. A normal copyright tag is still required., Public Domain, https:// commons.wikimedia.org/w/index.php?curid=190949 • http://www.animal-space.net/2010/12/16-funny-pictures-of-shocked-cats.html • https://www.flickr.com/photos/10459174@N03/34563622796/