Slawek Skowron - Monitoring @ Scale

Graphite monitoring in scale and how we done this @ BaseCRM.
Presentation from DevOps KRK.

1. MONITORING @ SCALE CLOUD SCALE Sławomir Skowron Devops @ BaseCRM Devops Kraków 2015
2. OUTLINE • What is Graphite ? • Graphite architecture • Additional components • Current production setup • Writes • Reads • BaseCRM graphite evolution • Data migrations and recovery • Multi region • Dashboards management • Future work
3. Monitoring system focused on: • Simple store metrics time series • Render graphs from time series on demand • API with functions • Dashboards • Huge number of tools and 3rd party products based on graphite WHAT IS GRAPHITE ?
4. WHY ALL THIS ? LET’S LOOK AT EXAMPLE IN GRAFANA
6. GRAPHITE ARCHITECTURE
7. • Graphite-Web - Django web application with JS frontend • Dashboards (DB to save dashboards) • API with functions, server side graphs rendering • Carbon - Twisted daemon • carbon-relay - hash / route metrics • carbon-aggregator - aggregate metrics • carbon-cache - “memory cache” and persist metrics to disk • Whisper - simple time series DB • seconds to point Data points send as: metric name + value + Unix epoch timestamp GRAPHITE ARCHITECTURE
8. GRAPHITE ARCHITECTURE
9. ADDITIONAL COMPONENTS
10. ADDITIONAL COMPONENTS • Diamond - https://github.com/BrightcoveOS/Diamond • Python daemon • over 120 collectors • simple collectors development • used for OS and generic services monitoring
11. ADDITIONAL COMPONENTS • Statsd - https://github.com/etsy/statsd/ • Node.js daemon • counts, sets, gauges, timers aggregates sends to graphite • many clients library’s • used for in app metrics
12. CURRENT PRODUCTION SETUP WRITES
13. ALL PROVISIONED BY ANSIBLE
15. Which components failed to work at scale ? carbon-relay switch to Carbon-c-Relay carbon-cache - switch to PyPy
16. REPLACEMENT • Carbon-C-Relay - https://github.com/grobian/carbon-c-relay • Written in C • replacement for carbon-relay in python • High performance • multi cluster support (traffic replication) • traffic load-balancing • traffic hashing • Aggregation and rewrites
17. IMPROVE Carbon-cache Switch to PyPy (2.4 and current 2.5) 40-50% less CPU usage on carbon-cache
18. CURRENT PRODUCTION SETUP - WRITES
19. CURRENT PRODUCTION SETUP - WRITES VM’s report to ELB Round-Robin to Relay Top Consistent hash with replication 2 Any of Carbon-cache on each store instance Write to local whisper store volume
20. CURRENT PRODUCTION SETUP - WRITES 450+ instances as clients (diamond, statsd, other) report in 30 seconds intervals carbon-c-relay as Top Relay • 3.5 - 4 mln metrics / min • 20% CPU usage on each • batch send (20k metrics) • queue 10 mln metrics carbon-c-relay as Local Relay • 7- 8 mln metrics / min • batch send (20k metrics) • queue 5 mln metrics Carbon-cache with PyPy (50% less CPU) • 7 - 8 mln metrics / min • each point update 0.13 - 0.15 ms 250K-350K Write IOPS 5-6 mln whisper DB files (2 copies)
21. CURRENT PRODUCTION SETUP - WRITES Graphite hashing - max space/performance like weakest host in cluster
22. CURRENT PRODUCTION SETUP - WRITES
23. CURRENT PRODUCTION SETUP - WRITES • Minimise number of other processes and CPU usage • CPU Offload • Carbon-c-relay low cpu, • Batch writes, • Separate webs for clients from store hosts • Focus on carbon-cache (Write) + graphite-web (Read) • Leverage OS memory for carbon-cache • Raid0 for more write performance - we have replica • Focus on IOPS - low service time • Time must be always sync
24. CURRENT PRODUCTION SETUP READS
25. CURRENT PRODUCTION SETUP - READS
26. CURRENT PRODUCTION SETUP - READS Web front dashboard based on Graphite-Web Graphite-web django backend as API for Grafana Couchbase as cache for graphite-web metrics Each store as API via graphite-web Average response <300ms Nginx on top behind ELB Webs calculates functions, stores serves RAW metrics (CPU offload)
27. BASECRM GRAPHITE EVOLUTION
28. BASECRM GRAPHITE EVOLUTION • PoC with external EBS and C3 instances • graphite-web, carbon-relay, carbon-cache, whisper files on EBS • Production started on i2.xlarge • 5 store instances - 800GB SSD, 30GB RAM, 4xCPU’s • 4 carbon-cache’s on each store • Same software as in PoC • Problems with machines replace and migrations to bigger cluster • dash-maker to manage complicated dashboards • Next with i2.4xlarge • 5 store instances - 2x800GB in Raid0, 60GB RAM, 8xCPU’s • 8 carbon-cache’s on each store • carbon-c-relay as Top and Local relay • Recovery tool to recover data from old cluster • Grafana as second dashboard interface • Current with i2.8large - latest bump • 5 store instances - 4x800GB in Raid0, 120GB RAM, 16xCPU’s • 16 carbon-cache’s on each store
29. DATA MIGRATION & RECOVERY
30. DATA MIGRATION & RECOVERY Replicate Traffic Copy old whispers, based on new cluster creates
31. DATA MIGRATION & RECOVERY 5 instances with 1Gbit/s - recovery tops 4.5Gbit/s using HTTP
32. DATA MIGRATION & RECOVERY Switch on ELB
33. DATA MIGRATION & RECOVERY Remove old cluster
34. MULTI REGION METRICS COLLECTING
35. MULTI REGION
36. DASH-MAKER INTERNAL DASHBOARDS MANAGEMENT
37. DASH-MAKER • Manage dashboards like never before • Template everything with Jinja2 (all jinja2 features) • Dashboard config - one YAML with Jinja2 support • Reusable graphs - Json's like in graphite-web with Jinja2 support • Global key=values for Jinja2 • Dynamic Jinja2 vars expanded from graphite (last * in metric name) • Many dashboards options from one config based on loop vars • supports graphite 0.9.12, 0.9.12 (evernote), 0.9.13, 0.10.0
38. DASH-MAKER $ dash-maker -f rabbitmq-server.yml 23:54:55 - dash-maker - main():Line:292 - INFO - Time [templating: 0. 023229 build: 0. 000177 save: 2.546407] Dashboard dm.us-east- 1.production.rabbitmq. server saved with success 23:54:58 - dash-maker - main():Line:292 - INFO - Time [templating: 0. 017746 build: 0. 000057 save: 2.549711] Dashboard dm.us-east- 1.sandbox.rabbitmq. server saved with success
39. FUTURE WORK AND PROBLEMS
40. FUTURE WORK AND PROBLEMS • Dash-maker with Grafana support • Out-o-band fast aggregation with anomaly detection • Graphite with Hashing is not elastic - InfluxDB ? march prod ready ? • In future one dashboard - grafana + influxdb ?