PDX 2017 - Roy Rapoport

Transcript

1. Monitoring Monitoring
   Systems at
   Roy Rapoport - @royrapoport
   May 22, 2017
   

   View Slide

2. The Hero’s Journey
   

   View Slide

3. An ordinary
   person
   A software engineer
   

   View Slide

4. Receives a call
   Lets build a
   monitoring system!
   

   View Slide

5. is dubious
   These specs are
   silly and you’re silly
   

   View Slide

6. is persuaded
   nonetheless
   Sure, I’d love to
   learn Scala!
   

   View Slide

7. vanquishes enemies
   Product Managers Engineering Managers Opinionated Coworkers
   

   View Slide

8. undergoes the
   ordeal
   Damn, this won’t scale.
   Also, we probably
   shouldn’t have coded
   a JS interpreter in
   Erlang.
   

   View Slide

9. returns,
   successful and
   transformed
   This will scale for at
   least a month.
   Ship it!
   

   View Slide

10. The Beginning
    

    View Slide

11. Real-Time Operational Insight
    

    View Slide

12. View Slide

13. Monitoring
    ain’t
    Alerting
    

    View Slide

14. Alerting
    

    View Slide

15. Alert Action
    Server
    Stream
    Processing
    Atlas SQS
    Hot
    Checker
    Cold
    Checker
    AWS Email Page …
    

    View Slide

16. Monitoring
    

    View Slide

17. Preamble
    

    View Slide

18. 6H 3D 18D
    Publish
    MR
    Query
    S3
    Anatomy of an Atlas Deployment Group (ADG)
    m
    etrics
    

    View Slide

19. Server Proxy
    us-west-2
    Server Proxy
    us-east-1
    eu-west-1
    Server Proxy
    

    View Slide

20. Server Proxy
    us-west-2
    Server Proxy
    us-east-1
    eu-west-1
    Server Proxy
    ADG
    ADG
    

    View Slide

21. Server Proxy
    us-west-2
    Server Proxy
    us-east-1
    eu-west-1
    Server Proxy
    ADG
    ADG
    iep
    ADG

    iep
    ADG

    iep
    ADG

    

    View Slide

22. Server Proxy
    us-west-2
    Server Proxy
    us-east-1
    eu-west-1
    Server Proxy
    ADG
    ADG
    iep
    ADG

    iep
    ADG

    iep
    ADG

    Critical
    Critical
    Critical
    

    View Slide

23. Tradeoffs:
    Fail loud
    Lower TTD
    

    View Slide

24. “This morning around 0504 US/Pacific, the
    active primary region hosting Atlas metrics
    data began experiencing issues and a
    number of alerts were triggered across the
    board.“
    February 9, 2017
    

    View Slide

25. View Slide

26. Managing
    Cost
    

    View Slide

27. Cost
    Doesn’t
    Matter
    

    View Slide

28. View Slide

29. View Slide

30. View Slide

31. “Everything Counts in Large Amounts”
    

    View Slide

32. “Everything Counts in Large Amounts”
    - Depeche Mode
    

    View Slide

33. Cardinal Sins
    

    View Slide

34. What’s a Metric?
    • name = cpu
    • nf.region = us-east-1
    • nf.zone = us-east-1a
    • nf.app = api
    • nf.cluster = api-main
    • nf.asg = api-main-v103
    • nf.node = i-103efbdd
    • device.operationalName = roku
    • isSupplementalVideo = false
    • OnComplete = operation
    • …
    

    View Slide

35. OOPS
    

    View Slide

36. Fun Metric Dimensions

    A True Story
    • IP address (~ 232)
    • Account ID (~109)
    • Customer ID (~109)
    • Execution time, ms (~30K)
    

    View Slide

37. View Slide

38. View Slide

39. View Slide

40. View Slide

41. “The graph on the wall tells the story of it all”
    - Depeche Mode
    

    View Slide

42. View Slide

43. What are you (not)
    talking about?
    

    View Slide

44. The latest thing
    

    View Slide

45. View Slide

46. View Slide

47. Meet Swaroop
    

    View Slide

48. The End
    

    View Slide

49. The Beginning
    

    View Slide

50. ?
    

    View Slide

51. View Slide

52. Attributions
    • https://www.flickr.com/photos/kalleboo/2240042641/
    • http://imgur.com/gallery/QCrR73q
    • https://www.flickr.com/photos/35028096@N04/6119259650
    • https://www.flickr.com/photos/86639298@N02/9420223184
    • http://paristext2015.com/2015/06/1-5-degrees-celsius-or-2-degrees-maybe-its-turtles-all-
    the-way-down/
    • https://www.flickr.com/photos/60969081@N00/8292545493
    • By Photo courtesy of National Nuclear Security Administration / Nevada Site Office - This
    image is available from the National Nuclear Security Administration Nevada Site Office
    Photo Library under number XX-34.This tag does not indicate the copyright status of the
    attached work. A normal copyright tag is still required., Public Domain, https://
    commons.wikimedia.org/w/index.php?curid=190949
    • http://www.animal-space.net/2010/12/16-funny-pictures-of-shocked-cats.html
    • https://www.flickr.com/photos/10459174@N03/34563622796/
    

    View Slide