Flapjack workshop - Monitorama PDX 2014

Transcript

1. Monitorama PDX 2014 Workshop
3. What is flapjack?
4. Monitoring alert routing system
5. Composable
6. Rollup
7. Alert routing
9. • event
10. • event • ↪ notify?
11. • event • ↪ notify? • ↪ who?
12. • event • ↪ notify? • ↪ who? • ↪ how?
13. API driven
14. No restarts required
15. Developed + used in production at:
16. Developed + used in production at: Developers: Ali Graham Jesse Reynolds Project manager: Lindsay Holmwood
17. +
18. Designed for humans
21. Why flapjack ?
22. Specific use cases
23. Large numbers of potential contacts
24. Multi-tenant
25. Segregated responsibility
26. Works well with others
27. Check engine independence
28. Single pane of glass
30. Getting started with flapjack
32. • vagrant box add precise64 precise64.box
33. • vagrant box add precise64 precise64.box • git clone https://github.com/flpjck/vagrant-flapjack.git
34. • vagrant box add precise64 precise64.box • git clone https://github.com/flpjck/vagrant-flapjack.git • vagrant up
35. • vagrant box add precise64 precise64.box • git clone https://github.com/flpjck/vagrant-flapjack.git • vagrant up • # Flapjack web UI • open http://localhost:3080
36. • vagrant box add precise64 precise64.box • git clone https://github.com/flpjck/vagrant-flapjack.git • vagrant up • # Flapjack web UI • open http://localhost:3080 • # Check execution engines • open http://localhost:3083/nagios3 • open http://localhost:3083/incinga
37. Simulate a failing check
39. • simulate-failed-check fail-and-recover \
40. • simulate-failed-check fail-and-recover \ • --entity foo-app-01.example.com \
41. • simulate-failed-check fail-and-recover \ • --entity foo-app-01.example.com \ • --check Sausage \
42. • simulate-failed-check fail-and-recover \ • --entity foo-app-01.example.com \ • --check Sausage \ • --time 3
43. • simulate-failed-check fail-and-recover \ • --entity foo-app-01.example.com \ • --check Sausage \ • --time 3 • # View result • open http://localhost:3080/check? entity=foo-app-01.example.com&check=Sausage
45. Check provisioning quality control
46. Grace period on notifications
47. processor: enabled: yes queue: events notifier_queue: notifications archive_events: true events_archive_maxage: 10800 new_check_scheduled_maintenance_duration: 100 years logger: level: INFO syslog_errors: yes
50. How does it work?
51. Data model
52. Contact
53. Contact Checks Checks Media
54. Contact Checks Checks Media Checks Checks Notification Rules
55. Contact Checks Checks Media Checks Checks Notification Rules Checks Checks Entities
56. Contact Checks Checks Media Checks Checks Notification Rules Checks Checks Checks Checks Checks Entities
57. Contact Checks Checks Media Checks Checks Notification Rules History (maintenance, acks, state changes) Checks Checks Checks Checks Checks Entities
58. Contacts
63. Contact Checks Checks Media Checks Checks Notification Rules History (maintenance, acks, state changes) Checks Checks Checks Checks Checks Entities
65. Contact Checks Checks Media Checks Checks Notification Rules History (maintenance, acks, state changes) Checks Checks Checks Checks Checks Entities
67. Architecture
69. event producers
70. event producers processors
71. event producers processors gateways
72. Event producers
73. processors gateways event producers
74. processors gateways Sensu Sensu::Extension::Flapjack Nagios nagios-receiver Icinga flapjackfeeder
75. # /etc/icinga/icinga.cfg broker_module=/usr/local/lib/flapjackfeeder.o ↪ redis_host=localhost,redis_port=6380
76. What is an event?
77. JSON
78. { "entity": ENTITY, "check": CHECK, "tags": TAGS, "state": STATE, "time": TIMESTAMP, "summary": SUMMARY }
79. { "entity": ENTITY, } ENTITY (string) Name of the relevant entity (e.g. FQDN)
80. { "check": CHECK, } CHECK (string) The check name
81. { "tags": TAGS, } TAGS (string[]) Array of tags pertaining to the event
82. { "state": STATE, } STATE (string) One of 'ok', 'warning', 'critical', 'unknown'
83. { "time": TIMESTAMP, } TIMESTAMP (integer) UNIX timestamp of the event's creation
84. { "summary": SUMMARY } SUMMARY (string) The check output
85. { "entity": ENTITY, "check": CHECK, "tags": TAGS, "state": STATE, "time": TIMESTAMP, "summary": SUMMARY }
86. { "entity": ENTITY, "check": CHECK, "tags": TAGS, "state": STATE, "time": TIMESTAMP, "summary": SUMMARY } simulate-failed-check fail-and-recover \ --entity foo-app-01 \ --check Sausage \ --time 3
87. { "entity": ENTITY, "check": CHECK, "tags": TAGS, "state": STATE, "time": TIMESTAMP, "summary": SUMMARY } { "entity": "foo-app-01", "check": "Sausage", "tags": null, "state": "critical", "time": 1399241087, "summary": "Simulated check" } simulate-failed-check fail-and-recover \ --entity foo-app-01 \ --check Sausage \ --time 3
88. Processors
89. processors gateways Sensu Sensu::Extension::Flapjack Nagios nagios-receiver Icinga flapjackfeeder
90. processor gateways notifier Sensu Sensu::Extension::Flapjack Nagios nagios-receiver Icinga flapjackfeeder
91. processor gateways notifier Sensu Sensu::Extension::Flapjack Nagios nagios-receiver Icinga flapjackfeeder
92. processor gateways notifier Sensu Sensu::Extension::Flapjack Nagios nagios-receiver Icinga flapjackfeeder • Processor: processor
93. processor gateways notifier Sensu Sensu::Extension::Flapjack Nagios nagios-receiver Icinga flapjackfeeder • Processor: • Read events processor
94. processor gateways notifier Sensu Sensu::Extension::Flapjack Nagios nagios-receiver Icinga flapjackfeeder • Processor: • Read events • Update state (counters, history) processor
95. processor gateways notifier Sensu Sensu::Extension::Flapjack Nagios nagios-receiver Icinga flapjackfeeder • Processor: • Read events • Update state (counters, history) • Determine if notification should be sent processor
96. Notifier
97. processor gateways notifier Nagios flapjackfeeder Sensu Sensu::Extension::Flapjack Icinga nagios-receiver
98. processor gateways notifier Nagios flapjackfeeder Sensu Sensu::Extension::Flapjack Icinga nagios-receiver • Notifier: notifier
99. processor gateways notifier Nagios flapjackfeeder Sensu Sensu::Extension::Flapjack Icinga nagios-receiver • Notifier: • Reads notifications notifier
100. processor gateways notifier Nagios flapjackfeeder Sensu Sensu::Extension::Flapjack Icinga nagios-receiver • Notifier: • Reads notifications • Routes notifications notifier
101. processor gateways notifier Nagios flapjackfeeder Sensu Sensu::Extension::Flapjack Icinga nagios-receiver • Notifier: • Reads notifications • Routes notifications • Dispatches alerts to gateways notifier
102. How are notifications routed?
103. event
104. event filters Find failing events
105. event filters Find failing events Processor Notifier
106. notification event filters Find failing events Processor Notifier
107. Find people interested in entity map [ alice bob, carol ] notification event filters Find failing events Processor Notifier
108. Find people interested in entity map map Find media owned by people [ [ alice, email ], [ alice, sms ], [ bob, email ], [ bob, sms ], [ carol, sms ], ] notification event filters Find failing events Processor Notifier
109. Find people interested in entity map map reduce Find media owned by people Delete media based on tags, severity, time of day [ [ alice, email ], [ alice, sms ], [ bob, sms ], ] notification event filters Find failing events Processor Notifier
110. Find people interested in entity map map reduce reduce Find media owned by people Delete media based on tags, severity, time of day Delete media based on blackholes [ [ alice, sms ], [ bob, sms ], ] notification event filters Find failing events Processor Notifier
111. Find people interested in entity map map reduce reduce reduce Find media owned by people Delete media based on tags, severity, time of day Delete media based on blackholes Delete media based on notification intervals notification event filters Find failing events [ [ alice, sms ], [ bob, sms ], ] Processor Notifier
112. Find people interested in entity map map reduce reduce reduce Find media owned by people Delete media based on tags, severity, time of day Delete media based on blackholes Delete media based on notification intervals notification event filters Find failing events alert alert [ [ alice, sms ], [ bob, sms ], ] Processor Notifier
113. Working with notification rules
114. curl -w 'response: %{http_code} \n' -X POST \ -H "Content-Type: application/vnd.api+json" -d \ '{ "notification_rules": [ { "entities": [ "foo-app-01.example.com" ], "regex_entities" : [ "^foo-\S{3}-\d{2}.example.com$" ], "tags": [ "database", "physical" ], "regex_tags" : [], "time_restrictions": [], "unknown_media": [], "warning_media": [ "email" ], "critical_media": [ "sms", "email" ], "unknown_blackhole": false, "warning_blackhole": false, "critical_blackhole": false } ] }' \ http://localhost:3081/contacts/5/notification_rules
115. flapjack.io/docs/jsonapi
117. Gateways
118. processor notifier Nagios flapjackfeeder Sensu Sensu::Extension::Flapjack Icinga nagios-receiver gateways
119. processor notifier Email SMS Jabber PagerDuty Web API Nagios flapjackfeeder Sensu Sensu::Extension::Flapjack Icinga nagios-receiver
120. Unidirectional vs Bidirectional
121. processor notifier Email SMS Jabber PagerDuty Web API Nagios flapjackfeeder Sensu Sensu::Extension::Flapjack Icinga nagios-receiver
122. processor notifier Email SMS Jabber PagerDuty Web API Nagios flapjackfeeder Sensu Sensu::Extension::Flapjack Icinga nagios-receiver Jabber PagerDuty
123. PagerDuty + Nagios Double ACK
124. Flapjack syncs PagerDuty state
126. Killer features
127. Self-checking (oobetet)
129. event producers
130. event producers flapjack
131. event producers flapjack oobetet
133. flapper
134. event producers flapper
135. event producers flapjack flapper
136. event producers flapjack jabber room flapper
137. event producers flapjack jabber room flapper oobetet
138. event producers flapjack jabber room flapper oobetet PagerDuty
140. Alert summarisation (rollup)
142. • per media,
143. • per media, • per contact
144. • per media, • per contact • thresholds
145. Contact Checks Checks Media Checks Checks Notification Rules History (maintenance, acks, state changes) Checks Checks Checks Checks Checks Entities
146. Contact Checks Checks Media Checks Checks Notification Rules History (maintenance, acks, state changes) Checks Checks Checks Checks Checks Entities Checks Checks Summary Threshold
148. Tagging [watch this space]
150. Things that may surprise you
151. Constant heartbeat
152. No one-off events
153. How long has a check been failing?
154. NOT "How many times has the check failed?"
155. Why?
156. No HARD/SOFT states
157. Broadcast delay
158. Alert summarisation (Rollup)
160. Open Source
161. • github.com/flapjack/flapjack
162. Release planning in open github.com/flapjack/flapjack/wiki/Releasing
163. Policy for triaging bugs + features github.com/flapjack/flapjack/wiki/Releasing
164. Semantic versioning (2.0.0)
165. We write tests! Unit + Integration
166. We write tests! ~80% coverage
167. We run tests!
169. •github.com /flapjack
170. •github.com /flapjack •github.com /flapjack/flapjack
171. •github.com /flapjack •github.com /flapjack/flapjack •github.com /flapjack/flapjack-diner
172. •github.com /flapjack •github.com /flapjack/flapjack •github.com /flapjack/flapjack-diner •github.com /flapjack/vagrant-flapjack
173. •github.com /flapjack •github.com /flapjack/flapjack •github.com /flapjack/flapjack-diner •github.com /flapjack/vagrant-flapjack •github.com /flapjack/omnibus-flapjack
174. •github.com /flapjack •github.com /flapjack/flapjack •github.com /flapjack/flapjack-diner •github.com /flapjack/vagrant-flapjack •github.com /flapjack/omnibus-flapjack •github.com /flapjack/packages.flapjack.io
175. •github.com /flapjack •github.com /flapjack/flapjack •github.com /flapjack/flapjack-diner •github.com /flapjack/vagrant-flapjack •github.com /flapjack/omnibus-flapjack •github.com /flapjack/packages.flapjack.io •
176. packages.flapjack.io
177. fat packages (omnibus)
178. .deb provided .rpm in the works (by you)
179. Quality documentation github.com/flapjack/flapjack/wiki
180. Bad documentation? BUG
181. Bad first experience? BUG
183. Large numbers of potential contacts
184. Segregated responsibility
185. Single pane of glass
187. Thank you! flapjack.io • github.com/flapjack/flapjack
188. Thank you! Liked the workshop? Let @auxesis + @jessereynolds know! flapjack.io • github.com/flapjack/flapjack
189. Post-Monitorama Monitoring Meetup @Jive - 18.30
190. Credits: http://www.flickr.com/photos/lizadaly/4373330774 http://www.flickr.com/photos/meltwater/420749031 http://www.flickr.com/photos/jonmould/5393395335 http://vmfarms.com/static/img/logos/ruby-logo.png http://www.flickr.com/photos/l1v32r1d3bmx/3985457584 http://www.flickr.com/photos/thomasforsyth/4313764488 http://www.flickr.com/photos/rubodewig/5161937181 http://www.flickr.com/photos/ronwls/7001551988 http://tosbourn.com/wp-content/uploads/2013/12/redis-logo.png?e0df77 http://www.flickr.com/photos/sparktography/83217827 http://www.flickr.com/photos/sdphotography/1570906849 http://www.flickr.com/photos/derekskey/9530097369 http://giphy.com/gifs/yeUxljCJjH1rW http://en.wikipedia.org/wiki/Broadcast_delay http://www.flickr.com/photos/whatknot/8642836187 http://www.flickr.com/photos/karen_d/8448507872 http://www.flickr.com/photos/buzzhoffman/4127280540 http://i.imgur.com/2UduUZ5.gif