Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Monitoring Casual in Kyoto / Sensu + Graphite

Monitoring Casual in Kyoto / Sensu + Graphite

Monitoring Casual in Kyoto 11/30

Yuuki Tsubouchi (yuuk1)

November 30, 2013
Tweet

More Decks by Yuuki Tsubouchi (yuuk1)

Other Decks in Technology

Transcript

  1. Sensu & Graphite
    Moden Monitoring Tool
    .POJUPSJOH$BTVBM5BML*O,ZPUP
    JEZ@VVLJ

    View full-size slide

  2. 2011 / 08
    ͸ͯͳΠϯλʔϯ

    View full-size slide

  3. 2012 / 09
    ΞϧόΠτࢯ

    View full-size slide

  4. NFUSJDTDPMMFDUJPO
    Today
    NFUSJDTWJTVBMJTBUJPO
    health check
    alert notification

    View full-size slide

  5. Monitoring Tool

    View full-size slide

  6. Monitoring Tool Requirement
    ✓ Scalable & Robust
    ✓ Pluggable
    ✓ Programmable
    ✓ Friendly to immutable infrastructure
    ✓ Simple

    View full-size slide

  7. ? Scalable & Robust
    - Munin 2.0 ?
    ✓ Pluggable
    - munin-node plugin
    ✗ Programmable
    - API ͳ͠
    ✗ Friendly to immutable infrastructure
    - ઃఆϑΝΠϧʹ؂ࢹϊʔυͷϦετ͕ඞཁ (serf ͰͳΜͱ͔ͳΔ)
    ✗ Simple
    - because of RRDtool

    View full-size slide

  8. Collectd + Cacti + RRDtool

    View full-size slide

  9. ✗ Scalable & Robust
    - because of RRDtool (cannot replicate)
    ✓ Pluggable
    - collectd input / output plugin (like fluentd)
    ✗ Programmable
    - Cacti ͸ API ͳ͠
    ✗ Friendly to immutable infrastructure
    - Must configure node list
    ✗ Simple
    - because of RRDtool

    View full-size slide

  10. Graduate from

    View full-size slide

  11. (SBQIJUF
    Scalable Realtime Graphing

    View full-size slide

  12. Graphite Data Flow
    Graphite
    Webapp
    Carbon
    Whisper
    Collector
    TCP
    HTTP
    metrics
    Graph
    JSON
    Store
    Get
    Graphite

    View full-size slide

  13. 3&45"1*
    - Graph rendering
    - JSON
    !
    )JHI*OEFQFOEFODZ"SDIJUFDUVSF
    - Webapp Frontend
    - Storage

    View full-size slide

  14. Replication / Sharding
    Carbon Relay
    Whisper
    metrics
    Carbon
    Whisper
    Carbon
    Whisper
    Carbon

    View full-size slide

  15. Many Better Dashboards
    Graphite
    Webapp
    Carbon
    Whisper
    Collector
    TCP
    HTTP
    metrics
    JSON
    Store
    Get
    Graphite
    Dashboard
    Graphiti, Gdash, Pencil, Graphene…

    View full-size slide

  16. 4FOTV
    A monitoring framework that aims!
    to be simple, malleable, and scalable

    View full-size slide

  17. .POJUPSJOH3PVUFS
    - Checks plugin: ex. Apache is up or down
    - Handlers plugin: ex. sending alert e-mails
    !
    )JHI*OEFQFOEFODZ"SDIJUFDUVSF
    - RabbitMQ
    - Redis
    - JSON

    View full-size slide

  18. Sensu Data Flow
    sensu-
    client
    sensu-
    client
    RabbitMQ
    sensu
    server
    Redis
    sensu-
    client
    run checks
    checks
    run handlers
    store
    notification
    checks
    check request

    View full-size slide

  19. More Detail
    • https://speakerdeck.com/joemiller/introduction-
    to-sensu
    • http://www.slideshare.net/jeremy_carroll/
    sensu-14485155
    • https://speakerdeck.com/joemiller/practical-
    examples-with-sensu-monitoring-framework

    View full-size slide

  20. (SBQIJUF 4FOTV

    View full-size slide

  21. Sensu & Graphite
    Data Flow
    sensu-
    client
    sensu-
    client
    sensu-
    client
    RabbitMQ
    sensu
    server
    Redis
    Graphite
    Webapp
    Carbon
    Whisper

    View full-size slide

  22. ✓ Scalable & Robust
    - Maybe
    ✓ Pluggable
    - sensu plugin & sensu handler
    ✓ Programmable
    - Graphite: REST API & Carbon TCP API!
    - Sensu: Events API & Checks API … etc
    ✓ Friendly to Immutable Infrastructure
    - ؂ࢹϊʔυͷઃఆ͍Βͣ
    ✓ Simple
    - Sensu: Ruby, Redis, RabbitMQ, JSON!
    - Graphite: REST API

    View full-size slide

  23. Graphite & RabbitMQ
    Integration

    View full-size slide

  24. Sensu & Graphite
    Data Flow
    sensu-
    client
    sensu-
    client
    sensu-
    client
    RabbitMQ
    sensu
    server
    Redis
    Graphite
    Webapp
    Carbon
    Whisper

    View full-size slide

  25. Stack
    Visualisation
    Storage
    Collector
    Collector Agent
    .VOJO
    $BDUJ
    $PMMFDUE
    (SBQIJUF
    4FOTV
    33%UPPM

    View full-size slide

  26. Other Solution
    Storage

    View full-size slide

  27. OpenTSDB
    Kairos
    InfluxDB

    View full-size slide

  28. Other Solution
    Collector

    View full-size slide

  29. Integration with other tools

    View full-size slide

  30. log health
    Infrastructure Information
    metrics
    Dynamic
    Static Host, Service, Role, Network, Location …

    View full-size slide

  31. log health
    Integration Viewer
    metrics
    Dynamic
    Static
    Integration
    View
    Host, Service, Role, Network, Location …

    View full-size slide

  32. log health
    Host, Service, Role, Network, Location …
    Integration with API
    metrics
    Dynamic
    Static
    Integration
    View
    JSON

    View full-size slide

  33. Monitoring Tool Requirement
    ✓ Scalable & Robust
    ✓ Pluggable
    ✓ Programmable
    ✓ Friendly to immutable infrastructure
    ✓ Simple

    View full-size slide

  34. ҟৗͳ౒ྗ͕ඞཁͳπʔϧʹݻࣥͤͣʹ
    ےͷྑ͍πʔϧΛબΜͰ͍͖͍ͨ

    View full-size slide

  35. Graduate from

    View full-size slide

  36. Sensu & Graphite
    Moden Monitoring Tool
    Monitoring Casual Talk In Kyoto
    id:y_uuki

    View full-size slide