Production Monitoring. Zef side.

Transcript

1. production monitoring. zef side. Saturday, October 27, 12
2. Saturday, October 27, 12
3. Saturday, October 27, 12
4. here not to teach Saturday, October 27, 12
5. but to bring your attention Saturday, October 27, 12
6. how many servers do you have? Saturday, October 27, 12
7. how many page views do you have? Saturday, October 27, 12
8. do you monitor your system extensively? Saturday, October 27, 12
9. what do you use? Saturday, October 27, 12
10. So, production?.. Saturday, October 27, 12
11. looks like that: Saturday, October 27, 12
12. Saturday, October 27, 12
13. yeah, that makes total sense!.. Saturday, October 27, 12
14. (c) noone. never. Saturday, October 27, 12
15. what’s there?... Saturday, October 27, 12
16. looking for patterns Saturday, October 27, 12
17. identify constructs Saturday, October 27, 12
18. repetitions Saturday, October 27, 12
19. trends Saturday, October 27, 12
20. outliers Saturday, October 27, 12
21. dispersion Saturday, October 27, 12
22. deviation Saturday, October 27, 12
23. fragmentation Saturday, October 27, 12
24. true story: Saturday, October 27, 12
25. sittin’ there, Saturday, October 27, 12
26. gathering exceptions from production Saturday, October 27, 12
27. suddenly Saturday, October 27, 12
28. Y SO MUCH YELLOW? Saturday, October 27, 12
29. check into exception details... Saturday, October 27, 12
30. 3 words: Saturday, October 27, 12
31. parse-query-string Saturday, October 27, 12
32. Ah shoo! Saturday, October 27, 12
33. That’s not really an exception! Saturday, October 27, 12
34. That’s just a 404! Saturday, October 27, 12
35. Eliminate the cause Avoid hitting complete stack when there’s no way to do it Saturday, October 27, 12
36. PROFIT! Saturday, October 27, 12
37. Also, we know that we now know how to identify exceptions that are 404s in reality Saturday, October 27, 12
38. exceptions => 404s that look like 404s are actually Saturday, October 27, 12
39. Still some repetition, but much better Saturday, October 27, 12
40. Repetition is not necessarily bad Saturday, October 27, 12
41. But you must understand why it occurs Saturday, October 27, 12
42. Good repetition. Saturday, October 27, 12
43. Garbage collection Saturday, October 27, 12
44. Bad Repetition Saturday, October 27, 12
45. Performance decrease Saturday, October 27, 12
46. any other hints? Saturday, October 27, 12
47. hmm... could not obtain database connection within N sec. Saturday, October 27, 12
48. ~N Saturday, October 27, 12
49. tweak parallelism / connection pool size. Saturday, October 27, 12
50. PROFIT! Saturday, October 27, 12
51. or... Saturday, October 27, 12
52. suddenly, application report rate decreases Saturday, October 27, 12
53. server responds only to every third/fourth request Saturday, October 27, 12
54. works absolutely fine after restart Saturday, October 27, 12
55. but behavior recurs after 5-7 minutes... Saturday, October 27, 12
56. somewhere in log files: Too many open files .... and java.net.UnknownHostException ... Saturday, October 27, 12
57. lsof Saturday, October 27, 12
58. Saturday, October 27, 12
59. Occupy UDP or what? Saturday, October 27, 12
60. Find connection leak Saturday, October 27, 12
61. Fix connection leak Saturday, October 27, 12
62. PROFIT! Saturday, October 27, 12
63. I would like to Saturday, October 27, 12
64. have a production monitoring system Saturday, October 27, 12
65. that will allow me to see these things early Saturday, October 27, 12
66. • very fast and easy access to recent events (recency is arbitrary) • fast arbitrary queries on recent events • real-time analytics, report generation • easy to go back in time • scalable queries across an entire dataset Expectations Saturday, October 27, 12
67. • pipelining, ability to inject additional processor • arbitrary data format • dumb client • extensible server (plug-n-play) • direct access to data, right in my repl Expectations Saturday, October 27, 12
68. • easy, extensible HTML / CSS / JS interface • graph generation (real-time, pre-calculated) • websockets for pushing stuff • re-broadcast of incoming events for custom analytics • support for multiple incoming channels Expectations Saturday, October 27, 12
69. • extensible alerting system (channels) • extensible alerting system (rules) • and (probably) many many more, which may be subsets of the mentioned ones in some way Expectations Saturday, October 27, 12
70. let’s draw a simple schema that will represent such a system Saturday, October 27, 12
71. and try our best to make it not opinionated Saturday, October 27, 12
72. opinionated stuff is about tight coupling Saturday, October 27, 12
73. let’s think of it in terms of a toolchain, not a framework (tm) Saturday, October 27, 12
74. reporter Processor + internal broadcast Persistent Store State Machine Alerts Reduce Engine (ad-hoc queries and analytics) Grouped Graphs Histograms Table Data Real-time graphs Real-time table data Latest Events (capped in-memory store) reporter reporter reporter reporter UDP, TCP, AMQP Health check polling Console (nrepl, custom consumers, real-time analysis) Saturday, October 27, 12
75. parts Saturday, October 27, 12
76. reporter Processor + internal broadcast Persistent Store State Machine Alerts Reduce Engine (ad-hoc queries and analytics) Grouped Graphs Histograms Table Data Real-time graphs Real-time table data Latest Events (capped in-memory store) reporter reporter reporter reporter UDP, TCP, AMQP Health check polling Console (nrepl, custom consumers, real-time analysis) Saturday, October 27, 12
77. simple, < 20 LOC software that sends events to the server reporter reporter reporter reporter reporter Saturday, October 27, 12
78. message format reporter reporter reporter reporter reporter { ;; Md5 hash of an event, that uniquely identifies it :md5 "cd0e351d2eefdf0f79e0b55a0efe543b", ;; Arbitrary additional info :additional_info {:execution_time 0.5469999, :url “http://mysite.com/page0” }, ;; Event Type identifier (404, exception, page load time) :type "page_load", ;; Dispatcher host name :hostname "dc0-web01", ;; Time when event was received :received_at ..., ;; Tags assigned to the event :tags ["metrics" "performance"] } Saturday, October 27, 12
79. reporter Processor + internal broadcast Persistent Store State Machine Alerts Reduce Engine (ad-hoc queries and analytics) Grouped Graphs Histograms Table Data Real-time graphs Real-time table data Latest Events (capped in-memory store) reporter reporter reporter reporter UDP, TCP, AMQP Health check polling Console (nrepl, custom consumers, real-time analysis) Saturday, October 27, 12
80. In nutshell, it’s a pipeline. Basic properties: • validate the event • classify / re-classify event • run processor based on current event type • add calculated params • distribute events internally Processor + internal broadcast Saturday, October 27, 12
81. Classification •remember the example with 404 that looked like an exception? •makes events harvesting and further processing easier •client will remain dumb, no need to redeploy on type changes •useful for sub-typing, e.q. page_load could become slow_page_load etc. •depends on your use case Processor + internal broadcast Saturday, October 27, 12
82. message format reporter reporter reporter reporter reporter { :additional_info {:execution_time 0.5469999, :url “http://mysite.com/page0” }, :type "page_load", :hostname "dc0-web01", :received_at ..., :tags ["metrics" "performance"] } Saturday, October 27, 12
83. message format reporter reporter reporter reporter reporter { :additional_info {:execution_time 0.5469999, :url “http://mysite.com/page0” }, :type "page_load", :hostname "dc0-web01", :received_at ..., :tags ["metrics" "performance"] } Saturday, October 27, 12
84. message format reporter reporter reporter reporter reporter { :additional_info {:execution_time 0.5469999, :url “http://mysite.com/” }, :type "page_load", :hostname "dc0-web01", :received_at ..., :tags ["metrics" "performance"] } Saturday, October 27, 12
85. message format reporter reporter reporter reporter reporter { :additional_info {:execution_time 0.5469999, :url “http://mysite.com/” }, :type "landing_page_load", :hostname "dc0-web01", :received_at ..., :tags ["metrics" "performance"] } Saturday, October 27, 12
86. Calculated params •internal example is md5 hash, used to ease finding unique events •could be used for grouping within a type •percent of a whole calculation max heap: 4gb, current: 2gb => 50% of a whole •add flags for further pipeline modules Processor + internal broadcast Saturday, October 27, 12
87. message format reporter reporter reporter reporter reporter { :additional_info {:backtrace “...” }, :type "backtrace", :hostname "dc0-web01", :received_at ..., :tags ["metrics" "performance"] } Saturday, October 27, 12
88. message format reporter reporter reporter reporter reporter { :additional_info {:backtrace “...” }, :md5 (calculate-md5 (:backtrace “...”)) :type "backtrace", :hostname "dc0-web01", :received_at ..., :tags ["metrics" "performance"] } Saturday, October 27, 12
89. Internal broadcast •when production system is running, you may want to attach to it in real-time and calculate message rate based on certain rules •throw events to persistence layer, state machine, arbitrary listeners Processor + internal broadcast Saturday, October 27, 12
90. reporter Processor + internal broadcast Persistent Store State Machine Alerts Reduce Engine (ad-hoc queries and analytics) Grouped Graphs Histograms Table Data Real-time graphs Real-time table data Latest Events (capped in-memory store) reporter reporter reporter reporter UDP, TCP, AMQP Health check polling Console (nrepl, custom consumers, real-time analysis) Saturday, October 27, 12
91. •highly depends on your throughput •consider something lightweight and scalable •if you generate fairly large amount of data, don’t try to handle processing it by means of your store •keep it simple •think about Cassandra or HDFS, depending on how you plan to access data Persistent Store Saturday, October 27, 12
92. reporter Processor + internal broadcast Persistent Store State Machine Alerts Reduce Engine (ad-hoc queries and analytics) Grouped Graphs Histograms Table Data Real-time graphs Real-time table data Latest Events (capped in-memory store) reporter reporter reporter reporter UDP, TCP, AMQP Health check polling Console (nrepl, custom consumers, real-time analysis) Saturday, October 27, 12
93. •sometimes you just don’t know in real-time •a new problem pops up, and you have to re-play stuff from a year ago •find trends, that are simply not visible on a day/week/month scale •most likely going to be something hadoop-based (YMMV) •cascalog works quite well for us •ad-hoc queries •scheduled analytics •correlating metrics (hard to do in real time) Reduce Engine (ad-hoc queries and analytics) Saturday, October 27, 12
94. Graphs Saturday, October 27, 12
95. Graphs Saturday, October 27, 12
96. reporter Processor + internal broadcast Persistent Store State Machine Alerts Reduce Engine (ad-hoc queries and analytics) Grouped Graphs Histograms Table Data Real-time graphs Real-time table data Latest Events (capped in-memory store) reporter reporter reporter reporter UDP, TCP, AMQP Health check polling Console (nrepl, custom consumers, real-time analysis) Saturday, October 27, 12
97. •that's basically an event cache. •ring-buffer, whose size is limited by rules •either time-bound (last hour, 6 hours, 24 hours) •or size-bound (max 10K events) •Riemann’s indexes are Cliff Click NonBlockingHashMap backed •you can go with (atom {:keyword (ref [])}) for starters •fire up an nrepl server on collector •have complete clojure access to your live data Latest Events (capped in-memory store) Saturday, October 27, 12
98. •very queriable (100K+ events in collection are processed blazingly fast) •optimize, calculate real-time indexes when needed •for the most complex queries, that turn out to be a bottleneck, use state machine •use for anything: stats, graphs, table data, dashboards. It’s also very, very easy! Latest Events (capped in-memory store) Saturday, October 27, 12
99. •very queriable (100K+ events in collection are processed blazingly fast) •optimize, calculate real-time indexes when needed •for the most complex queries, that turn out to be a bottleneck, use state machine •use for anything: stats, graphs, table data, dashboards. It’s also very, very easy! Latest Events (capped in-memory store) Saturday, October 27, 12
100. reporter Processor + internal broadcast Persistent Store State Machine Alerts Reduce Engine (ad-hoc queries and analytics) Grouped Graphs Histograms Table Data Real-time graphs Real-time table data Latest Events (capped in-memory store) reporter reporter reporter reporter UDP, TCP, AMQP Health check polling Console (nrepl, custom consumers, real-time analysis) Saturday, October 27, 12
101. •counters, gauges, meters •triggers for events and alarms •reducers of any type, actually •if the amount of events of type `T` with md5 `M` (or any md5) exceeds 10, escalate the issue (send alert) •if we received more than `N` events of type `T` in 1 minute, send alert •if reponse of page exceeds `X` ms, send alert State Machine Saturday, October 27, 12
102. •You can measure metrics based on any arbitrary rule and make any arbitrary callback, basically. State Machine Saturday, October 27, 12
103. reporter Processor + internal broadcast Persistent Store State Machine Alerts Reduce Engine (ad-hoc queries and analytics) Grouped Graphs Histograms Table Data Real-time graphs Real-time table data Latest Events (capped in-memory store) reporter reporter reporter reporter UDP, TCP, AMQP Health check polling Console (nrepl, custom consumers, real-time analysis) Saturday, October 27, 12
104. •build processing pipelines in runtime •run nrepl •run queries, get necessary information Console (nrepl, custom consumers, real-time analysis) Saturday, October 27, 12
105. it’s important to know Saturday, October 27, 12
106. what your application does right now Saturday, October 27, 12
107. make it easy to notice failures Saturday, October 27, 12
108. to identify behavior oddities Saturday, October 27, 12
109. for that, again Saturday, October 27, 12
110. you have to run a lot of ad-hoc queries Saturday, October 27, 12
111. make system more stable Saturday, October 27, 12
112. by noticing things that happen Saturday, October 27, 12
113. often but irregular Saturday, October 27, 12
114. see what code doesn’t reveal Saturday, October 27, 12
115. if the host is up, is “up” what you expect by “up”? Saturday, October 27, 12
116. collect everything Saturday, October 27, 12
117. storage is cheap Saturday, October 27, 12
118. YMMV if if you operate on twitter/facebook scale, where you need to seek more global approaches Saturday, October 27, 12
119. let your system talk to you Saturday, October 27, 12
120. extract business metrics from tech data Saturday, October 27, 12
121. improve business metrics by tech data Saturday, October 27, 12
122. make it easy to see what your users see Saturday, October 27, 12
123. plan your growth probably quite hard to do it without knowing your current corellation between performance and number of servers, and extracting trends from historical data Saturday, October 27, 12
124. DYI I’ve set it already quite a few times, but I won’t get tired of saying it: DO IT YOURSELF. it’s possible to find a tool that does most of things for you, but you may not find your own patterns with it. Use libs for visualization, persistency, distribution, transport, processing, but try to avoid coupling. If it’s hard to get data in, or get data out of it, good idea to avoid it. Saturday, October 27, 12
125. also, do you know about Clojure Saturday, October 27, 12
126. Saturday, October 27, 12
127. Werkz Saturday, October 27, 12
128. It just werkz Saturday, October 27, 12
129. A growing collection of open source Clojure libraries that support multiple Clojure & JDK versions, licensed under the EPL, target Clojure 1.3+. Saturday, October 27, 12
130. Saturday, October 27, 12
131. Saturday, October 27, 12
132. ifesdjeen clojurewerkz / / michaelklishin / Also, you should follow Saturday, October 27, 12