Metrics and Monitoring

Introduction to Metrics and Monitoring including steps - steps that can be implemented to achieve good monitoring.

Transcript

1. Metrics & Monitoring Sleman, January 9th 2016
2. “You cannot manage what you do not measure” - Edward Deming
3. Metrics indicators to find what is on track and what needs to be changed before it is too late
4. Metrics Categories • Business performance • Usage • Health
5. What product guys wants to see
6. What backend guys want to see
7. What mobile guys want to see
8. What we DON’T WANT to see
9. Monitoring Core Principles 1. Identify important topics as many as possible 2. Identify all topics as early as possible 3. Generate alarms as few as possible 4. Do it with as little work as possible
10. Tools • Alert Alarms that wakes us up if something happened • Graph Summarize all of trends. We are visual beings. • Logs Base source of truth and contains all of details
11. DEVELOPMENT ENV
12. Development Practices 1. Naive Implementation 2. Measure Implementation 3. Optimize (if needed) • Measure Everything • Logging is Cheap
13. Continuous Integration Pipeline
14. Insights • Potential thresholds • Consequences for failures • Filtered important resources to be monitored • Where to do improvement or optimization
15. PRODUCTION ENV
16. Where to set our eyes? Layer 0: Application Layer 1: Process • Active connections • Slow processing • Throughput • Warning, Error, Fatal logs, etc • Changes in process status e.g. terminated, stopped, restarted • Uptime • Consumed resources, etc
17. Where to set our eyes? Layer 2: Server Layer 3: Hosting • Number of running processes • System resource ( CPU, network, IO, memory, etc) • Hardware health, etc • Latency • Availability • Maintenance schedules, etc
18. Where to set our eyes? Layer 4: External Dependencies Layer 5: USERS!! • API Changes • SSL-APNS certificates renewals • Policy changes, etc • Behaviours • Crashes • Device types • Social Media oauth logins • Successful responses * • Sessions, etc
19. Four Monitoring Steps • Monitor potential bad things • Monitor actual bad things • Monitor good things • Tune and Improve
20. Monitor potential bad things • Identify resource • Understand the threshold value and consequences • Set alert before the threshold reached • Daily active users reached 70% of PubNub threshold • Increased social login failure in 30 minutes • Increased timeouts in 30 minutes • Increased >= 400 HTTP Codes
21. Monitor actual bad things • INEVITABLE • Identify resource • Understand the failure effects • Ensure alert triggered • Ensure all source of truths exists • Application server restarted • Fatal error or exceptions happened in apps * • Mobile apps crashed • Chats aren’t delivered • Twilio failed to send SMS
22. Monitor Good Things ( before turns into disaster ) • Identify resource • Set alert when changes happened • It’s BETTER to compare to sudden drops/spike rather than gradual changes / threshold reached * • Stores created every hour • Transactions created every hour • Successful payments every hour • Chat delivered every hour, etc
23. Tune and Improve • Add metrics as part of our retrospectives • Asks our teams if any metrics need to be added / changed • Add / remove logs if necessary • Remove noise alerts • Pay close attention to our tools *
24. A metric will tell you that something is happening, while an analysis will tell you why something is happening. - Vince Law
25. maturnuwun :)
26. Source: • Scalyr.com • Fabric.io • Newrelic.com