Monitoring Casual in Kyoto / Sensu + Graphite

Monitoring Casual in Kyoto 11/30

Transcript

1. Sensu & Graphite
   Moden Monitoring Tool
   .POJUPSJOH$BTVBM5BML*O,ZPUP
   JEZ@VVLJ
   

   View Slide

2. id:y_uuki
   

   View Slide

3. 2011 / 08
   ͸ͯͳΠϯλʔϯ
   

   View Slide

4. 2012 / 09
   ΞϧόΠτࢯ
   

   View Slide

5. 2013 / 07
   

   View Slide

6. 2013 / 09
   

   View Slide

7. 2013 / 11
   

   View Slide

8. 2013 / 12
   

   View Slide

9. View Slide

10. In Kyoto
    

    View Slide

11. Monitoring
    

    View Slide

12. NFUSJDTDPMMFDUJPO
    Today
    NFUSJDTWJTVBMJTBUJPO
    health check
    alert notification
    

    View Slide

13. Monitoring Tool
    

    View Slide

14. Monitoring Tool Requirement
    ✓ Scalable & Robust
    ✓ Pluggable
    ✓ Programmable
    ✓ Friendly to immutable infrastructure
    ✓ Simple
    
    

    View Slide

15. Case 1
    

    View Slide

16. View Slide

17. ? Scalable & Robust
    - Munin 2.0 ?
    ✓ Pluggable
    - munin-node plugin
    ✗ Programmable
    - API ͳ͠
    ✗ Friendly to immutable infrastructure
    - ઃఆϑΝΠϧʹ؂ࢹϊʔυͷϦετ͕ඞཁ (serf ͰͳΜͱ͔ͳΔ)
    ✗ Simple
    - because of RRDtool
    

    View Slide

18. Case 2
    

    View Slide

19. Collectd + Cacti + RRDtool
    

    View Slide

20. ✗ Scalable & Robust
    - because of RRDtool (cannot replicate)
    ✓ Pluggable
    - collectd input / output plugin (like fluentd)
    ✗ Programmable
    - Cacti ͸ API ͳ͠
    ✗ Friendly to immutable infrastructure
    - Must configure node list
    ✗ Simple
    - because of RRDtool
    

    View Slide

21. Case 3
    

    View Slide

22. Graduate from
    

    View Slide

23. (SBQIJUF
    Scalable Realtime Graphing
    

    View Slide

24. Graphite Data Flow
    Graphite
    Webapp
    Carbon
    Whisper
    Collector
    TCP
    HTTP
    metrics
    Graph
    JSON
    Store
    Get
    Graphite
    

    View Slide

25. 3&45"1*
    - Graph rendering
    - JSON
    !
    )JHI*OEFQFOEFODZ"SDIJUFDUVSF
    - Webapp Frontend
    - Storage
    

    View Slide

26. Replication / Sharding
    Carbon Relay
    Whisper
    metrics
    Carbon
    Whisper
    Carbon
    Whisper
    Carbon
    

    View Slide

27. Many Better Dashboards
    Graphite
    Webapp
    Carbon
    Whisper
    Collector
    TCP
    HTTP
    metrics
    JSON
    Store
    Get
    Graphite
    Dashboard
    Graphiti, Gdash, Pencil, Graphene…
    

    View Slide

28. 4FOTV
    A monitoring framework that aims!
    to be simple, malleable, and scalable
    

    View Slide

29. .POJUPSJOH3PVUFS
    - Checks plugin: ex. Apache is up or down
    - Handlers plugin: ex. sending alert e-mails
    !
    )JHI*OEFQFOEFODZ"SDIJUFDUVSF
    - RabbitMQ
    - Redis
    - JSON
    

    View Slide

30. Sensu Data Flow
    sensu-
    client
    sensu-
    client
    RabbitMQ
    sensu
    server
    Redis
    sensu-
    client
    run checks
    checks
    run handlers
    store
    notification
    checks
    check request
    

    View Slide

31. View Slide

32. View Slide

33. More Detail
    • https://speakerdeck.com/joemiller/introduction-
    to-sensu
    • http://www.slideshare.net/jeremy_carroll/
    sensu-14485155
    • https://speakerdeck.com/joemiller/practical-
    examples-with-sensu-monitoring-framework
    

    View Slide

34. (SBQIJUF 4FOTV
    

    View Slide

35. Sensu & Graphite
    Data Flow
    sensu-
    client
    sensu-
    client
    sensu-
    client
    RabbitMQ
    sensu
    server
    Redis
    Graphite
    Webapp
    Carbon
    Whisper
    

    View Slide

36. ✓ Scalable & Robust
    - Maybe
    ✓ Pluggable
    - sensu plugin & sensu handler
    ✓ Programmable
    - Graphite: REST API & Carbon TCP API!
    - Sensu: Events API & Checks API … etc
    ✓ Friendly to Immutable Infrastructure
    - ؂ࢹϊʔυͷઃఆ͍Βͣ
    ✓ Simple
    - Sensu: Ruby, Redis, RabbitMQ, JSON!
    - Graphite: REST API
    

    View Slide

37. Graphite & RabbitMQ
    Integration
    

    View Slide

38. Sensu & Graphite
    Data Flow
    sensu-
    client
    sensu-
    client
    sensu-
    client
    RabbitMQ
    sensu
    server
    Redis
    Graphite
    Webapp
    Carbon
    Whisper
    

    View Slide

39. Stack
    Visualisation
    Storage
    Collector
    Collector Agent
    .VOJO
    $BDUJ
    $PMMFDUE
    (SBQIJUF
    4FOTV
    33%UPPM
    

    View Slide

40. Other Solution
    Storage
    

    View Slide

41. OpenTSDB
    Kairos
    InfluxDB
    

    View Slide

42. Other Solution
    Collector
    

    View Slide

43. fluentd?
    

    View Slide

44. Integration with other tools
    

    View Slide

45. log health
    Infrastructure Information
    metrics
    Dynamic
    Static Host, Service, Role, Network, Location …
    

    View Slide

46. log health
    Integration Viewer
    metrics
    Dynamic
    Static
    Integration
    View
    Host, Service, Role, Network, Location …
    

    View Slide

47. log health
    Host, Service, Role, Network, Location …
    Integration with API
    metrics
    Dynamic
    Static
    Integration
    View
    JSON
    

    View Slide

48. Monitoring Tool Requirement
    ✓ Scalable & Robust
    ✓ Pluggable
    ✓ Programmable
    ✓ Friendly to immutable infrastructure
    ✓ Simple
    
    

    View Slide

49. ҟৗͳ౒ྗ͕ඞཁͳπʔϧʹݻࣥͤͣʹ
    ےͷྑ͍πʔϧΛબΜͰ͍͖͍ͨ
    

    View Slide

50. Graduate from
    

    View Slide

51. Sensu & Graphite
    Moden Monitoring Tool
    Monitoring Casual Talk In Kyoto
    id:y_uuki
    

    View Slide

52. Thanks
    

    View Slide