Monitoring Casual in Kyoto / Sensu + Graphite

Transcript

1. 1.

   Sensu & Graphite Moden Monitoring Tool .POJUPSJOH
   $BTVBM
   5BML
   *O
   ,ZPUP
   JEZ@VVLJ

2. 2.

   id:y_uuki

3. 3.

   2011 / 08 ͸ͯͳΠϯλʔϯ

4. 4.

   2012 / 09 ΞϧόΠτࢯ

5. 5.

   2013 / 07

6. 6.

   2013 / 09

7. 7.

   2013 / 11

8. 8.

   2013 / 12

9. 9.

   None
10. 10.

    In Kyoto

11. 11.

    Monitoring

12. 12.

    
    NFUSJDT
    DPMMFDUJPO Today
    NFUSJDT
    WJTVBMJTBUJPO health check alert notification

13. 13.

    Monitoring Tool

14. 14.

    Monitoring Tool Requirement ✓ Scalable & Robust ✓ Pluggable ✓

    Programmable ✓ Friendly to immutable infrastructure ✓ Simple 
15. 15.

    Case 1

16. 16.

    None
17. 17.

    ? Scalable & Robust - Munin 2.0 ? ✓ Pluggable

    - munin-node plugin ✗ Programmable - API ͳ͠ ✗ Friendly to immutable infrastructure - ઃఆϑΝΠϧʹ؂ࢹϊʔυͷϦετ͕ඞཁ (serf ͰͳΜͱ͔ͳΔ) ✗ Simple - because of RRDtool
18. 18.

    Case 2

19. 19.

    Collectd + Cacti + RRDtool

20. 20.

    ✗ Scalable & Robust - because of RRDtool (cannot replicate)

    ✓ Pluggable - collectd input / output plugin (like fluentd) ✗ Programmable - Cacti ͸ API ͳ͠ ✗ Friendly to immutable infrastructure - Must configure node list ✗ Simple - because of RRDtool
21. 21.

    Case 3

22. 22.

    Graduate from

23. 23.

    (SBQIJUF Scalable Realtime Graphing

24. 24.

    Graphite Data Flow Graphite Webapp Carbon Whisper Collector TCP HTTP

    metrics Graph JSON Store Get Graphite
25. 25.

    3&45
    "1*
    - Graph rendering - JSON ! )JHI
    *OEFQFOEFODZ
    "SDIJUFDUVSF
    - Webapp

    Frontend - Storage
26. 26.

    Replication / Sharding Carbon Relay Whisper metrics Carbon Whisper Carbon

    Whisper Carbon
27. 27.

    Many Better Dashboards Graphite Webapp Carbon Whisper Collector TCP HTTP

    metrics JSON Store Get Graphite Dashboard Graphiti, Gdash, Pencil, Graphene…
28. 28.

    4FOTV A monitoring framework that aims! to be simple, malleable,

    and scalable
29. 29.

    .POJUPSJOH
    3PVUFS
    - Checks plugin: ex. Apache is up or down

    - Handlers plugin: ex. sending alert e-mails ! )JHI
    *OEFQFOEFODZ
    "SDIJUFDUVSF
    - RabbitMQ - Redis - JSON
30. 30.

    Sensu Data Flow sensu- client sensu- client RabbitMQ sensu server

    Redis sensu- client run checks checks run handlers store notification checks check request
31. 31.

    None
32. 32.

    None
33. 33.

    More Detail • https://speakerdeck.com/joemiller/introduction- to-sensu • http://www.slideshare.net/jeremy_carroll/ sensu-14485155 • https://speakerdeck.com/joemiller/practical-

    examples-with-sensu-monitoring-framework
34. 34.

    (SBQIJUF 4FOTV

35. 35.

    Sensu & Graphite Data Flow sensu- client sensu- client sensu-

    client RabbitMQ sensu server Redis Graphite Webapp Carbon Whisper
36. 36.

    ✓ Scalable & Robust - Maybe ✓ Pluggable - sensu

    plugin & sensu handler ✓ Programmable - Graphite: REST API & Carbon TCP API! - Sensu: Events API & Checks API … etc ✓ Friendly to Immutable Infrastructure - ؂ࢹϊʔυͷઃఆ͍Βͣ ✓ Simple - Sensu: Ruby, Redis, RabbitMQ, JSON! - Graphite: REST API
37. 37.

    Graphite & RabbitMQ Integration

38. 38.

    Sensu & Graphite Data Flow sensu- client sensu- client sensu-

    client RabbitMQ sensu server Redis Graphite Webapp Carbon Whisper
39. 39.

    Stack Visualisation Storage Collector Collector Agent .VOJO $BDUJ $PMMFDUE (SBQIJUF

    4FOTV 33%UPPM
40. 40.

    Other Solution Storage

41. 41.

    OpenTSDB Kairos InfluxDB

42. 42.

    Other Solution Collector

43. 43.

    fluentd?

44. 44.

    Integration with other tools

45. 45.

    log health Infrastructure Information metrics Dynamic Static Host, Service, Role,

    Network, Location …
46. 46.

    log health Integration Viewer metrics Dynamic Static Integration View Host,

    Service, Role, Network, Location …
47. 47.

    log health Host, Service, Role, Network, Location … Integration with

    API metrics Dynamic Static Integration View JSON
48. 48.

    Monitoring Tool Requirement ✓ Scalable & Robust ✓ Pluggable ✓

    Programmable ✓ Friendly to immutable infrastructure ✓ Simple 
49. 49.

    ҟৗͳ౒ྗ͕ඞཁͳπʔϧʹݻࣥͤͣʹ ےͷྑ͍πʔϧΛબΜͰ͍͖͍ͨ

50. 50.

    Graduate from

51. 51.

    Sensu & Graphite Moden Monitoring Tool Monitoring Casual Talk In

    Kyoto id:y_uuki
52. 52.

    Thanks