$30 off During Our Annual Pro Sale. View Details »

PDX 2017 - Roy Rapoport

PDX 2017 - Roy Rapoport

Monitorama

May 22, 2017
Tweet

More Decks by Monitorama

Other Decks in Technology

Transcript

  1. Monitoring Monitoring
    Systems at
    Roy Rapoport - @royrapoport
    May 22, 2017

    View Slide

  2. The Hero’s Journey

    View Slide

  3. An ordinary
    person
    A software engineer

    View Slide

  4. Receives a call
    Lets build a
    monitoring system!

    View Slide

  5. is dubious
    These specs are
    silly and you’re silly

    View Slide

  6. is persuaded
    nonetheless
    Sure, I’d love to
    learn Scala!

    View Slide

  7. vanquishes enemies
    Product Managers Engineering Managers Opinionated Coworkers

    View Slide

  8. undergoes the
    ordeal
    Damn, this won’t scale.
    Also, we probably
    shouldn’t have coded
    a JS interpreter in
    Erlang.

    View Slide

  9. returns,
    successful and
    transformed
    This will scale for at
    least a month.
    Ship it!

    View Slide

  10. The Beginning

    View Slide

  11. Real-Time Operational Insight

    View Slide

  12. View Slide

  13. Monitoring
    ain’t
    Alerting

    View Slide

  14. Alerting

    View Slide

  15. Alert Action
    Server
    Stream
    Processing
    Atlas SQS
    Hot
    Checker
    Cold
    Checker
    AWS Email Page …

    View Slide

  16. Monitoring

    View Slide

  17. Preamble

    View Slide

  18. 6H 3D 18D
    Publish
    MR
    Query
    S3
    Anatomy of an Atlas Deployment Group (ADG)
    m
    etrics

    View Slide

  19. Server Proxy
    us-west-2
    Server Proxy
    us-east-1
    eu-west-1
    Server Proxy

    View Slide

  20. Server Proxy
    us-west-2
    Server Proxy
    us-east-1
    eu-west-1
    Server Proxy
    ADG
    ADG

    View Slide

  21. Server Proxy
    us-west-2
    Server Proxy
    us-east-1
    eu-west-1
    Server Proxy
    ADG
    ADG
    iep
    ADG

    iep
    ADG

    iep
    ADG


    View Slide

  22. Server Proxy
    us-west-2
    Server Proxy
    us-east-1
    eu-west-1
    Server Proxy
    ADG
    ADG
    iep
    ADG

    iep
    ADG

    iep
    ADG

    Critical
    Critical
    Critical

    View Slide

  23. Tradeoffs:
    Fail loud
    Lower TTD

    View Slide

  24. “This morning around 0504 US/Pacific, the
    active primary region hosting Atlas metrics
    data began experiencing issues and a
    number of alerts were triggered across the
    board.“
    February 9, 2017

    View Slide

  25. View Slide

  26. Managing
    Cost

    View Slide

  27. Cost
    Doesn’t
    Matter

    View Slide

  28. View Slide

  29. View Slide

  30. View Slide

  31. “Everything Counts in Large Amounts”

    View Slide

  32. “Everything Counts in Large Amounts”
    - Depeche Mode

    View Slide

  33. Cardinal Sins

    View Slide

  34. What’s a Metric?
    • name = cpu
    • nf.region = us-east-1
    • nf.zone = us-east-1a
    • nf.app = api
    • nf.cluster = api-main
    • nf.asg = api-main-v103
    • nf.node = i-103efbdd
    • device.operationalName = roku
    • isSupplementalVideo = false
    • OnComplete = operation
    • …

    View Slide

  35. OOPS

    View Slide

  36. Fun Metric Dimensions

    A True Story
    • IP address (~ 232)
    • Account ID (~109)
    • Customer ID (~109)
    • Execution time, ms (~30K)

    View Slide

  37. View Slide

  38. View Slide

  39. View Slide

  40. View Slide

  41. “The graph on the wall tells the story of it all”
    - Depeche Mode

    View Slide

  42. View Slide

  43. What are you (not)
    talking about?

    View Slide

  44. The latest thing

    View Slide

  45. View Slide

  46. View Slide

  47. Meet Swaroop

    View Slide

  48. The End

    View Slide

  49. The Beginning

    View Slide

  50. ?

    View Slide

  51. View Slide

  52. Attributions
    • https://www.flickr.com/photos/kalleboo/2240042641/
    • http://imgur.com/gallery/QCrR73q
    • https://www.flickr.com/photos/35028096@N04/6119259650
    • https://www.flickr.com/photos/86639298@N02/9420223184
    • http://paristext2015.com/2015/06/1-5-degrees-celsius-or-2-degrees-maybe-its-turtles-all-
    the-way-down/
    • https://www.flickr.com/photos/60969081@N00/8292545493
    • By Photo courtesy of National Nuclear Security Administration / Nevada Site Office - This
    image is available from the National Nuclear Security Administration Nevada Site Office
    Photo Library under number XX-34.This tag does not indicate the copyright status of the
    attached work. A normal copyright tag is still required., Public Domain, https://
    commons.wikimedia.org/w/index.php?curid=190949
    • http://www.animal-space.net/2010/12/16-funny-pictures-of-shocked-cats.html
    • https://www.flickr.com/photos/10459174@N03/34563622796/

    View Slide