Fullstack monitoring

Monitor any layer of your Ruby application with Yabeda monitoring suite.

Transcript

1. Fullstack monitoring Monitor any layer of your Ruby application Dmitry Salahutdinov Summer Saint P Rubyconf 2021
2. Save people time for something that matters! Is a leading e-grocery company in Russia
3. Me on GitHub
4. Why you do need Ruby-app monitoring? • To know how application performs • Ensure application works as expected (SRE) • What limit of capacity is has • To prevent degradations • Investigate degradations (when happen) • To evaluate results of optimisations
5. Yabeda framework Tooling extended extendable framework!
6. Yabeda DSL
7. Yabeda adapters
8. Yabeda adapters Developer friendly: “get up and running” or “run example & explore”
9. Common Ruby-application Is complicated and has multi-layer stack
10. 1. Puma 2. Rails 3. Rack / Rack-attack 4. Sidekiq 5. Kafka 6. External Services/HTTP requests 7. Non-standard metrics usage (memory profiling) We will cover
11. Puma monitoring With yabeda-puma-plugin ruby gem
12. Metrics
13. Capacity
14. Capacity gap
15. Lack of capacity alert
16. Application Busy metric
18. Rails monitoring With yabeda-rails ruby gem
19. Yabeda::Rails dashbord
20. Slowest queries
21. SLO: serve 95% of requests within 250ms
22. APDEX score
23. APDEX (by endpoint): Yabeda version
24. Error rate
25. Rack monitoring
26. Rack custom logger
27. Rack::Attack monitoring With yabeda-rack-attack ruby gem
28. Rack::Attack monitoring
29. Throttled requests
30. - rule can works incorrect - periodically call outrange? - bad requests produces workload Why to monitor abusive requests?
31. Tracked requests Research the number of specific requests
32. Sidekiq monitoring With yabeda-sidekiq ruby gem
33. Sidekiq dashbord
34. Latency - critical jobs SLO - any other jobs default SLO max(sidekiq_queue_latency{queue=”sms”}) by (queue) > 300
35. Job Error rate (by queue)
36. max (sidekiq_running_job_runtime) by (queue) > 300 Job execution time
37. - prevent capacity lack - investigate bottlenecks Why to monitor Sidekick?
38. Kafka client monitoring With yabeda-kafka ruby gem
39. Kafka metrics With ruby-kafka
40. Kafka metrics With yabeda-kafka ruby gem MVP
41. Kafka metrics
42. External HTTP request monitoring With yabeda-http_requests ruby gem
43. Why to monitor - how good HTTP requests are made - how many times they consume - what is the distribution of HTTP response statuses
44. Metrics
45. Log memory profiling (research) With memory_profiler gem
47. Results
48. Why to monitor? - determine hungry jobs - Its parameters (you can log them)
49. Research execution time (research)
50. Server side rendering execution time
51. Product metrics (for developers)
52. Metrics: - yabeda-rb/yabeda-puma-plugin - Built-in metrics for rack-attack monitoring out of the box! - yabeda-rb/yabeda-rails - Yabeda plugin to collect basic metrics for Rails applications - yabeda-rb/yabeda-graphql - Measure and understand how good your GraphQL-Ruby application works - yabeda-rb/yabeda-sidekiq -Yabeda plugin for complete monitoring of Sidekiq - yabeda-rb/yabeda-faktory - Built-in metrics for monitoring Faktory worker for Ruby out of the box - yabeda-rb/yabeda-http_requests - monitor how many outgoing HTTP calls your application does (uses Sniffer) - dsalahutdinov/yabeda-kafka - metrics for ruby-kafka Kafka client - dsalahutdinov/yabeda-rack_attack - Built-in metrics for rack-attack monitoring out of the box! - efigence/yabeda-grape - Metrics for monitoring Grape endpoints - Placewise/yabeda-gruf - metrics for gRPC Ruby Framework - ianks/yabeda-gc - Built in metrics for monitoring the Ruby garbage collector. Exporters: - yabeda-rb/yabeda-prometheus - Adapter to expose metrics collected by Yabeda plugins to Prometheus - yabeda-rb/yabeda-prometheus-mmap - Yabeda Prometheus exporter based on prometheus-client-mmap Adapters: - yabeda-rb/yabeda-datadog - Yabeda Datadog adapter. Collect and send custom metrics from Ruby apps to Datadog. - yabeda-rb/yabeda-newrelic - NewRelic adapter for Yabeda monitoring framework - asusikov/yabeda-statsd - Yabeda adapter for easy exporting collected custom metrics from your application to the Statsd server. Existing tools
53. Non-yet-existing tools
54. Why to invent new bicycle? - Flexibility (g.e. the APDEX in context of endpoint) - Newrelic does not logs specific metrics and tools (g.e. sidekiq latency, Kafka client) - Yabeda can complement NewRelic (with specific metrics & NewRelic adapter) - Modularity (use what you need) - Cost/Maintenance (self-hosted vs SaaS) - Yabeda is the research tool - Better developer experience - SRE-ready Why not just use Newrelic?
55. Thank you 🧡