Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Flapjack workshop - Monitorama PDX 2014

Flapjack workshop - Monitorama PDX 2014

Lindsay Holmwood

May 07, 2014
Tweet

More Decks by Lindsay Holmwood

Other Decks in Technology

Transcript

  1. Developed + used in production at: Developers: Ali Graham Jesse

    Reynolds Project manager: Lindsay Holmwood
  2. +

  3. • vagrant box add precise64 precise64.box • git clone https://github.com/flpjck/vagrant-flapjack.git

    • vagrant up • # Flapjack web UI • open http://localhost:3080
  4. • vagrant box add precise64 precise64.box • git clone https://github.com/flpjck/vagrant-flapjack.git

    • vagrant up • # Flapjack web UI • open http://localhost:3080 • # Check execution engines • open http://localhost:3083/nagios3 • open http://localhost:3083/incinga
  5. • simulate-failed-check fail-and-recover \ • --entity foo-app-01.example.com \ • --check

    Sausage \ • --time 3 • # View result • open http://localhost:3080/check? entity=foo-app-01.example.com&check=Sausage
  6. processor: enabled: yes queue: events notifier_queue: notifications archive_events: true events_archive_maxage:

    10800 new_check_scheduled_maintenance_duration: 100 years logger: level: INFO syslog_errors: yes
  7. Contact Checks Checks Media Checks Checks Notification Rules History (maintenance,

    acks, state changes) Checks Checks Checks Checks Checks Entities
  8. Contact Checks Checks Media Checks Checks Notification Rules History (maintenance,

    acks, state changes) Checks Checks Checks Checks Checks Entities
  9. Contact Checks Checks Media Checks Checks Notification Rules History (maintenance,

    acks, state changes) Checks Checks Checks Checks Checks Entities
  10. { "entity": ENTITY, "check": CHECK, "tags": TAGS, "state": STATE, "time":

    TIMESTAMP, "summary": SUMMARY } simulate-failed-check fail-and-recover \ --entity foo-app-01 \ --check Sausage \ --time 3
  11. { "entity": ENTITY, "check": CHECK, "tags": TAGS, "state": STATE, "time":

    TIMESTAMP, "summary": SUMMARY } { "entity": "foo-app-01", "check": "Sausage", "tags": null, "state": "critical", "time": 1399241087, "summary": "Simulated check" } simulate-failed-check fail-and-recover \ --entity foo-app-01 \ --check Sausage \ --time 3
  12. processor gateways notifier Sensu Sensu::Extension::Flapjack Nagios nagios-receiver Icinga flapjackfeeder •

    Processor: • Read events • Update state (counters, history) processor
  13. processor gateways notifier Sensu Sensu::Extension::Flapjack Nagios nagios-receiver Icinga flapjackfeeder •

    Processor: • Read events • Update state (counters, history) • Determine if notification should be sent processor
  14. processor gateways notifier Nagios flapjackfeeder Sensu Sensu::Extension::Flapjack Icinga nagios-receiver •

    Notifier: • Reads notifications • Routes notifications notifier
  15. processor gateways notifier Nagios flapjackfeeder Sensu Sensu::Extension::Flapjack Icinga nagios-receiver •

    Notifier: • Reads notifications • Routes notifications • Dispatches alerts to gateways notifier
  16. Find people interested in entity map [ alice bob, carol

    ] notification event filters Find failing events Processor Notifier
  17. Find people interested in entity map map Find media owned

    by people [ [ alice, email ], [ alice, sms ], [ bob, email ], [ bob, sms ], [ carol, sms ], ] notification event filters Find failing events Processor Notifier
  18. Find people interested in entity map map reduce Find media

    owned by people Delete media based on tags, severity, time of day [ [ alice, email ], [ alice, sms ], [ bob, sms ], ] notification event filters Find failing events Processor Notifier
  19. Find people interested in entity map map reduce reduce Find

    media owned by people Delete media based on tags, severity, time of day Delete media based on blackholes [ [ alice, sms ], [ bob, sms ], ] notification event filters Find failing events Processor Notifier
  20. Find people interested in entity map map reduce reduce reduce

    Find media owned by people Delete media based on tags, severity, time of day Delete media based on blackholes Delete media based on notification intervals notification event filters Find failing events [ [ alice, sms ], [ bob, sms ], ] Processor Notifier
  21. Find people interested in entity map map reduce reduce reduce

    Find media owned by people Delete media based on tags, severity, time of day Delete media based on blackholes Delete media based on notification intervals notification event filters Find failing events alert alert [ [ alice, sms ], [ bob, sms ], ] Processor Notifier
  22. curl -w 'response: %{http_code} \n' -X POST \ -H "Content-Type:

    application/vnd.api+json" -d \ '{ "notification_rules": [ { "entities": [ "foo-app-01.example.com" ], "regex_entities" : [ "^foo-\S{3}-\d{2}.example.com$" ], "tags": [ "database", "physical" ], "regex_tags" : [], "time_restrictions": [], "unknown_media": [], "warning_media": [ "email" ], "critical_media": [ "sms", "email" ], "unknown_blackhole": false, "warning_blackhole": false, "critical_blackhole": false } ] }' \ http://localhost:3081/contacts/5/notification_rules
  23. processor notifier Email SMS Jabber PagerDuty Web API Nagios flapjackfeeder

    Sensu Sensu::Extension::Flapjack Icinga nagios-receiver
  24. processor notifier Email SMS Jabber PagerDuty Web API Nagios flapjackfeeder

    Sensu Sensu::Extension::Flapjack Icinga nagios-receiver
  25. processor notifier Email SMS Jabber PagerDuty Web API Nagios flapjackfeeder

    Sensu Sensu::Extension::Flapjack Icinga nagios-receiver Jabber PagerDuty
  26. Contact Checks Checks Media Checks Checks Notification Rules History (maintenance,

    acks, state changes) Checks Checks Checks Checks Checks Entities
  27. Contact Checks Checks Media Checks Checks Notification Rules History (maintenance,

    acks, state changes) Checks Checks Checks Checks Checks Entities Checks Checks Summary Threshold
  28. Thank you! Liked the workshop? Let @auxesis + @jessereynolds know!

    flapjack.io • github.com/flapjack/flapjack
  29. Credits: http://www.flickr.com/photos/lizadaly/4373330774 http://www.flickr.com/photos/meltwater/420749031 http://www.flickr.com/photos/jonmould/5393395335 http://vmfarms.com/static/img/logos/ruby-logo.png http://www.flickr.com/photos/l1v32r1d3bmx/3985457584 http://www.flickr.com/photos/thomasforsyth/4313764488 http://www.flickr.com/photos/rubodewig/5161937181 http://www.flickr.com/photos/ronwls/7001551988 http://tosbourn.com/wp-content/uploads/2013/12/redis-logo.png?e0df77

    http://www.flickr.com/photos/sparktography/83217827 http://www.flickr.com/photos/sdphotography/1570906849 http://www.flickr.com/photos/derekskey/9530097369 http://giphy.com/gifs/yeUxljCJjH1rW http://en.wikipedia.org/wiki/Broadcast_delay http://www.flickr.com/photos/whatknot/8642836187 http://www.flickr.com/photos/karen_d/8448507872 http://www.flickr.com/photos/buzzhoffman/4127280540 http://i.imgur.com/2UduUZ5.gif