Necessary tooling and monitoring for performance critical applications

In a competitive market which is heavily based on trust and confidence, ensuring stable performance becomes important. However, with Continuos Delivery, making sure that something doesn’t break becomes a hard problem to solve.

The audience can look forward to an understanding about the performance and error monitoring setup at https://www.otto.de/, a hugely successful e-commerce application. I plan to share experiences around how the monitoring set up is able to equip the developers with enough knowledge to debug and fix critical performance issues before the users could see and how we can use such metrics to create automated alarms to alert project teams about any performance problems as well as security threats.

Transcript

1. PERFORMANCE CRITICAL APPLICATIONS NECESSARY TOOLING AND MONITORING FOR @mananbharara
2. PERFORMANCE CRITICAL APPLICATIONS NECESSARY TOOLING AND MONITORING FOR @mananbharara
3. OTTO
4. OTTO http://www.forbes.com/sites/adamtanner/2014/03/05/amazons-war-on-germanys-18-billion-patriarch/#7989b6eb4162
5. OTTO
6. OTTO ▸ 1 million visitors / day ▸ ~ 2 orders / second http://dev.otto.de/2016/03/20/why-microservices/#more-2320
7. OTTO ▸ 1 million visitors / day ▸ ~ 2 orders / second http://dev.otto.de/2016/03/20/why-microservices/#more-2320
8. OTTO ▸ 1 million visitors / day ▸ ~ 2 orders / second http://dev.otto.de/2016/03/20/why-microservices/#more-2320
9. OTTO ▸ 1 million visitors / day ▸ ~ 2 orders / second 30 customers X 8 screens X 2 orders = 480 PI’s / second http://dev.otto.de/2016/03/20/why-microservices/#more-2320
10. OTTO ▸ 1 million visitors / day ▸ ~ 2 orders / second 30 customers X 8 screens X 2 orders = 480 PI’s / second ~ 1000 PI’s / second http://dev.otto.de/2016/03/20/why-microservices/#more-2320
11. WHY? WHAT MAKES THIS DIFFERENT
12. WHY? WHAT MAKES THIS DIFFERENT ▸ We’re talking money…
13. WHY? WHAT MAKES THIS DIFFERENT ▸ We’re talking money… ▸ Building more features
14. WHY? WHAT MAKES THIS DIFFERENT ▸ We’re talking money… ▸ Building more features => Adding more value
15. WHY? WHAT MAKES THIS DIFFERENT ▸ We’re talking money… ▸ Building more features => Adding more value => > money
16. WHY? WHAT MAKES THIS DIFFERENT ▸ We’re talking money… ▸ Building more features ▸ Decision making => Adding more value => > money
17. WHY? WHAT MAKES THIS DIFFERENT ▸ We’re talking money… ▸ Building more features ▸ Decision making ▸ Verify business assumptions => Adding more value => > money
18. WHY? WHAT MAKES THIS DIFFERENT ▸ We’re talking money… ▸ Building more features ▸ Decision making ▸ Verify business assumptions => Adding more value => > money 300 billion 3 trillion
19. REQUIREMENTS I NEED…
20. REQUIREMENTS I NEED… ▸ What every other monitoring tool does
21. REQUIREMENTS I NEED… ▸ What every other monitoring tool does - Database monitoring
22. REQUIREMENTS I NEED… ▸ What every other monitoring tool does - Database monitoring - Monitoring standard server metrics - Req/Sec, Response time, Throughput…
23. REQUIREMENTS I NEED… ▸ What every other monitoring tool does - Database monitoring - Monitoring standard server metrics - Req/Sec, Response time, Throughput… - Alerting upon exceptions
24. REQUIREMENTS I NEED… ▸ What every other monitoring tool does - Database monitoring - Monitoring standard server metrics - Req/Sec, Response time, Throughput… - Alerting upon exceptions ▸ Measuring the state of the system
25. REQUIREMENTS I NEED… ▸ What every other monitoring tool does - Database monitoring - Monitoring standard server metrics - Req/Sec, Response time, Throughput… - Alerting upon exceptions ▸ Measuring the state of the system - Orders/sec, Users on website
26. REQUIREMENTS I NEED… ▸ What every other monitoring tool does - Database monitoring - Monitoring standard server metrics - Req/Sec, Response time, Throughput… - Alerting upon exceptions ▸ Measuring the state of the system - Orders/sec, Users on website ▸ Narrow down sources of bottlenecks
27. REQUIREMENTS I NEED… ▸ What every other monitoring tool does - Database monitoring - Monitoring standard server metrics - Req/Sec, Response time, Throughput… - Alerting upon exceptions ▸ Measuring the state of the system - Orders/sec, Users on website ▸ Narrow down sources of bottlenecks ▸ Validate business assumptions
28. LOGGING LOGGING
29. LOGGING
30. LOGGING
31. LOGGING IS ABOUT SCATTERED INCIDENTS IN THE SYSTEM LOGGING
32. METRICS HELP IN UNDERSTANDING THE STATE OF THE SYSTEM AND GAIN VALUABLE INSIGHTS… CAPTURE
33. METRICS HELP IN UNDERSTANDING THE STATE OF THE SYSTEM AND GAIN VALUABLE INSIGHTS… CAPTURE ANYTIME
34. WHY? Collecting data is cheap, but not having it when you need it can be expensive, so you should instrument everything, and collect all the useful data you reasonably can… https://www.datadoghq.com
35. METRICS Measure the behavior of critical components in your production environment… CAPTURE http://metrics.dropwizard.io/3.1.0/
36. METRICS Measure the behavior of critical components in your production environment… CAPTURE http://metrics.dropwizard.io/3.1.0/
37. METRICS Measure the behavior of critical components in your production environment… CAPTURE http://metrics.dropwizard.io/3.1.0/
38. METRICS Measure the behavior of critical components in your production environment… CAPTURE http://metrics.dropwizard.io/3.1.0/
39. METRICS Measure the behavior of critical components in your production environment… CAPTURE http://metrics.dropwizard.io/3.1.0/
40. METRICS Measure the behavior of critical components in your production environment… CAPTURE http://metrics.dropwizard.io/3.1.0/
41. CAPTURE
42. CAPTURE ▸ Counters
43. CAPTURE ▸ Counters - HTTP/any connection pools
44. CAPTURE ▸ Counters - HTTP/any connection pools ▸ Gauges
45. CAPTURE ▸ Counters - HTTP/any connection pools ▸ Gauges - Used disk space
46. CAPTURE ▸ Counters - HTTP/any connection pools ▸ Gauges - Used disk space ▸ Meters
47. CAPTURE ▸ Counters - HTTP/any connection pools ▸ Gauges - Used disk space ▸ Meters - Requests per second
48. CAPTURE ▸ Counters - HTTP/any connection pools ▸ Gauges - Used disk space ▸ Meters - Requests per second - Number of personalized users - Number of cached products - Orders per second
49. CAPTURE ▸ Counters - HTTP/any connection pools ▸ Gauges - Used disk space ▸ Meters - Requests per second - Number of personalized users - Number of cached products - Orders per second
50. CAPTURE
51. CAPTURE ▸ Histograms
52. CAPTURE ▸ Histograms - Mean size of response bodies
53. CAPTURE ▸ Histograms - Mean size of response bodies - Percentile of customers getting 5 personal recommendations or more
54. CAPTURE ▸ Histograms - Mean size of response bodies - Percentile of customers getting 5 personal recommendations or more ▸ Timers
55. CAPTURE ▸ Histograms - Mean size of response bodies - Percentile of customers getting 5 personal recommendations or more ▸ Timers - Page rendering time
56. CAPTURE ▸ Histograms - Mean size of response bodies - Percentile of customers getting 5 personal recommendations or more ▸ Timers - Page rendering time - It took 80ms to give recommendations to 99% customers for 300 req/sec
57. CAPTURE ▸ Histograms - Mean size of response bodies - Percentile of customers getting 5 personal recommendations or more ▸ Timers - Page rendering time - It took 80ms to give recommendations to 99% customers for 300 req/sec - But took 200ms at 800 req/sec
58. CAPTURE ▸ Histograms - Mean size of response bodies - Percentile of customers getting 5 personal recommendations or more ▸ Timers - Page rendering time - It took 80ms to give recommendations to 99% customers for 300 req/sec - But took 200ms at 800 req/sec
59. AGGREGATE MAKE SENSE OF DATA
60. AGGREGATE GRAPHITE GRAPHITE-WEB CARBON CACHE
61. ▸ Graphing AGGREGATE GRAPHITE GRAPHITE-WEB CARBON CACHE
62. ▸ Graphing ▸ Query DSL AGGREGATE GRAPHITE GRAPHITE-WEB CARBON CACHE
63. ▸ Graphing ▸ Query DSL ▸ Running graphite in cluster mode AGGREGATE GRAPHITE GRAPHITE-WEB CARBON CACHE
64. ▸ Graphing ▸ Query DSL ▸ Running graphite in cluster mode ▸ Up to date AGGREGATE GRAPHITE GRAPHITE-WEB CARBON CACHE
65. VISUALIZE GET THE PICTURE…
66. VISUALIZE GRAPHITE WEB
67. VISUALIZE GRAPHITE WEB ‣Difficult to drill down
68. VISUALIZE GRAPHITE WEB ‣Difficult to drill down ‣Non Interactive
69. VISUALIZE GRAPHITE WEB ‣Difficult to drill down ‣Non Interactive ‣Query language and editor is prone to mistakes
70. VISUALIZE GRAPHITE WEB ‣Difficult to drill down ‣Non Interactive ‣Query language and editor is prone to mistakes ‣Difficult to correlate changes in graphs to events
71. VISUALIZE OSCILLATOR
72. VISUALIZE OSCILLATOR
73. VISUALIZE OSCILLATOR
74. VISUALIZE OSCILLATOR
75. VISUALIZE OSCILLATOR DEMO
76. VISUALIZE ANNOTATIONS
77. LEARNINGS LEARNINGS
78. LEARNINGS LEARNINGS ▸ Choose the right metrics
79. LEARNINGS LEARNINGS
80. LEARNINGS LEARNINGS
81. LEARNINGS LEARNINGS
82. LEARNINGS LEARNINGS ▸ Choose the right metrics - Disk utilization, User logins
83. LEARNINGS LEARNINGS ▸ Choose the right metrics - Disk utilization, User logins ▸ Mean can be absurd - Choose wisely
84. LEARNINGS
85. LEARNINGS
86. LEARNINGS LEARNINGS ▸ Choose the right metrics - Disk utilization, User logins ▸ Mean can be absurd - Choose wisely
87. LEARNINGS LEARNINGS ▸ Choose the right metrics - Disk utilization, User logins ▸ Mean can be absurd - Choose wisely ▸ Choose the right tool for the job. Histograms, counters, meters, timers.
88. ALERT ALERTING
89. ALERT ALERTING
90. ALERT ALERTING
91. ALERT ALERTING
92. ALERT ALERTING
93. ALERT ALERTING
94. ALERT ALERTING ▸ Of extreme importance
95. ALERT ALERTING ▸ Of extreme importance ▸ Definition of done
96. ALERT ALERTING ▸ Of extreme importance ▸ Definition of done ▸ Make it visible
97. SUMMING IT UP THE COMPLETE SOLUTION
98. SUMMING IT UP THE COMPLETE SOLUTION CAPTURE
99. SUMMING IT UP THE COMPLETE SOLUTION CAPTURE AGGREGATE
100. SUMMING IT UP THE COMPLETE SOLUTION CAPTURE AGGREGATE VISUALIZE
101. SUMMING IT UP THE COMPLETE SOLUTION CAPTURE AGGREGATE VISUALIZE ALERT
102. CONTINUOS DELIVERY CONTINUOUS DELIVERY & MONITORING
103. SOFTWARE THAT CLAIMS TO MONITOR EVERYTHING FOR YOU CURRENT SETUP
104. SOFTWARE THAT CLAIMS TO MONITOR EVERYTHING FOR YOU CURRENT SETUP
105. SOFTWARE THAT CLAIMS TO MONITOR EVERYTHING FOR YOU ▸ Hooks into individual function calls CURRENT SETUP
106. SOFTWARE THAT CLAIMS TO MONITOR EVERYTHING FOR YOU ▸ Hooks into individual function calls ▸ Could be too expensive CURRENT SETUP
107. SOFTWARE THAT CLAIMS TO MONITOR EVERYTHING FOR YOU ▸ Hooks into individual function calls ▸ Could be too expensive ▸ A great monitoring and alerting setup is possible for free CURRENT SETUP
108. THANKS! @mananbharara @mananbharara READ MORE ▸ Metrics, Metrics, Everywhere - https://www.youtube.com/watch?v=czes-oa0yik ▸ Oscillator - https://github.com/otto-de/oscillator ▸ Xray - https://github.com/otto-de/tesla-xray ▸ OTTO Dev blog - http://dev.otto.de/