Performance Monitoring at Employment Hero

part 1 of a series about Performance Management and Optimisation. Talks are held at Employment Hero to share the mindset of caring about application performance.

Transcript

1. Performance management Part 1: Setting the right expectation Luong Vo - coolie @ Employment hero
2. I am not an expert :)
3. I inspire Not instruct
4. Today’s Agenda 1. Why ? 2. Why measure? 3. Why monitor? 4. What to monitor? 5. How to monitor?
5. Today’s Goals Everyone leaves with at least one idea about performance management. Everyone knows where and what to look for in performance monitoring.
6. 1. Why ? Pinterest increased search engine traffic and sign-ups by 15% when they reduced perceived wait times by 40%. COOK increased conversions by 7%, decreased bounce rates by 7%, and increased pages per session by 10% when they reduced average page load time by 850 milliseconds. The BBC found they lost an additional 10% of users for every additional second their site took to load. DoubleClick by Google found 53% of mobile site visits were abandoned if a page took longer than 3 seconds to load.
7. 1. Why ? It’s about users , expenses, outages Then business, income, adoption, retention…. Source: https://idlewords.com/talks/website_obesity.htm , https://developers.google.com/web/fundamentals/performance/why-performance-matters
8. 2. Why measure? Measure the actual performance of your application, and focus on the 20% of your code that is the worst performance offender. If you're applying equal effort to all areas, you are wasting your time.
9. I do not optimize anything in my application until my metrics tell me so.
10. 2. Why Monitor? Random boss in AU: “The site feels slow” Luong: “Really!? It works like a charm here” Luong’s colleague: “Look good on my machine” Is my API, web or mobile application available? “Yes” Is it functioning correctly? “Yes” Is it fast? “Hm”
11. Monitoring is DEfinitely a good place to start with. (although there are other options)
12. 3. WHat to monitor? External metrics: latency, throughput, error rate. Internal metrics: CPU, Load Average, Memory, Network.
13. Individual Brainstorming: Scenario 1 What will we look for if some of our users tell us that the app is slow ?
14. Individual Brainstorming: Scenario 2 How can we be aware of outage before our users?
15. Individual Brainstorming: Scenario 3 How do we know how many servers we need or how much load we can handle in peak time?
16. 4. How to Monitor? Mindset: - Calculate - Store - Query - Which layer? - How to integrate? Tools: - Stackify - Datadog - New Relic - Prometheus - ...
17. 4. How to Monitor?
18. 4.1. Prometheus overview Model: Push/Pull
19. 4.1. Prometheus overview Push Gateway
20. 4.1. Prometheus overview Exporters
21. 4.1. Prometheus overview Review
22. 4.2. Instrument your application Step 1: Decide which type of metrics you should use: - Gauge - Counter - Histogram - Summary Is it only rising? “Counter” Is it rising and decreasing? “Gauge” Is it based on observations and timeframe? “Histogram and Summary”
23. Individual Brainstorming: Scenario 1 What type of metric will we use for Memory usage?
24. Individual Brainstorming: Scenario 2 What type of metric will we use for number of requests served?
25. Individual Brainstorming: Scenario 3 What type of metric will we use for average latency?
26. 4.2. Instrument your application Step 2: Choose your metric name and labels, start using it
27. 4.2. Instrument your application Or: Integrate with your application - Eh Monitoring - Middleware - Required files - Hooks - Metric container We wrap Ruby Prometheus library to handle labeling and automatic integrations with rails, sinatra, sequel, active record….
28. 4.2. Instrument your application Step 3: Start Querying - PromQL - Metric name http://prometheus.ehrocks.com http://prometheus.staging.ehrock s.com
29. 4.2. Instrument your application Step 4: Start visualizing http://monitor.ehrocks.com
30. 4.2. Instrument your application Step 2.5: Actually....
31. 4.2. Instrument your application If you want to go fast, use cloud monitoring as a service
32. 4.2. Instrument your application Step 4: Start Alerting
34. Why measure? ¯\_(ツ)_/¯ Noi dao ly thi thuong song nhu … ?
35. REsults from Sep 2019
36. REsults Nowadays
37. Always set your Goals. Make sure you got the right metrics to back it up (because reality sucks)