Practical Phoenix Monitoring

* what to measure
* how to mesure it
* how to use exometer and elixometer
* how we implemented

Transcript

1. 1 JUN SUMIDA PRACTICAL PHOENIX MONITORING
2. 2 Introduction @junsumida Jun Sumida self()
 |> at( XFLAG™ STUDIO, mixi, inc.) |> develops(game backend servers) |> works(elixir, ruby, javascript, devops) |> github(junsumida)
3. 3 Introduction • What to measure • How to measure • How to use Exo/Elixometer • How we Implemented Table of Contents
4. 4 WHAT TO MEASURE What to measure
5. • Application information • api request/response count, api response time • query count, query exec time • Stats of the Erlang VM • memory usage • process count • garbage collection • reductions • io 5 What to measure What to measure
6. 6 How to measure HOW TO MEASURE
7. 7 Metric libraries • Erlang • exometer • folsom • vmstats • Elixir • Elixometer • Beaker • metrix see also https://github.com/h4cc/awesome-elixir#debugging How to measure
8. 8 Metric libraries • Erlang • exometer • folsom • vmstats • Elixir • Elixometer • Beaker • metrix see also https://github.com/h4cc/awesome-elixir#debugging How to measure
9. 9 Exometer https://github.com/Feuerlabs/exometer • a great measurement tool written in Erlang • developed by Basho (Riak) • easy to use (probably, maybe, possibly…) • multipule backends • InfluxDB, loggers, Graphite, etc… How to measure
10. 10 Elixometer https://github.com/pinterest/elixometer • a light wrapper around exometer • developed by Pinterest • easy to use (i swear!) • no complex configurations How to measure
11. Why Exo/Elixometer? 11 PROS • well-documented • both of them “were” well-maintained • still better than most of other measuring libraries • comparatively easy to find examples How to measure
12. Why Exo/Elixometer? 12 CONS • dependency hell • lager vs Logger • configuring exometer is a little bit confusing • struggling with Erlang due to the reporters anyway How to measure
13. 13 HOW TO USE EXO/ELIXOMETER What to measure
14. 14 ELIXOMETER • counter • histogram • timer • gauge • spiral How to use Elixometer
15. 15 How to use Elixometer • ordinary histograms • stores all values during a configured time span • provides some analysis • min, max • mean, median • percentile histogram (1)
16. 16 histogram (2) defmodule Myapp.Plug.Metrics do import Plug.Conn use Elixometer def call(conn, _default) do before_time = :os.timestamp diff = :timer.now_diff(:os.timestamp, before_time) conn |> register_before_send(fn conn -> update_histogram("response_time", diff) end) end end microseconds How to use Elixometer
17. 3 histogram (3) What to measure
18. • a sort of histogram • useful when measuring the performance of a function 18 timer use Elixometer @timed(key: :"timed.super_heavy") def super_heavy do HeavyModule.call_something end def triple_super_heavy do timed("timed.triple.super_heavy") do HeavyModule.call_something HeavyModule.call_something HeavyModule.call_something end end annotation do block How to use Elixometer
19. 19 • holds one numeric value • always returns the most recently provided one • good for metrics like memory usage or disk space gauge photo credit: Pressure Cooker - All American Model 910 - Pressure gauge How to use Elixometer
20. 20 • increases value over time • when reporting the value, the metric is reset • good for metrics like queries per second or requests per second spiral (1) photo credit: University of Washington How to use Elixometer
21. spiral (2) 21 # config/config.exs config :elixometer, reporter: :exometer_report_lager, env: "webapp", metric_prefix: "myapp", update_frequency: 10_000 config :myapp, Myapp.Repo, adapter: Ecto.Adapters.MySQL, username: "root", password: "", loggers: [{Myapp.Repo.Metrics, :record_metric, []}], hostname: "localhost", database: "myapp", # lib/myapp/repo/metrics.ex defmodule Myapp.Repo.Metrics do use Elixometer def record_metric(entry) do update_spiral("query_count", 1) end end How to use Elixometer
22. 22 EXOMETER • function • counter • gauge • duration • histogram How to use Exometer
23. function (1) 23 How to use Exometer allows to use Erlang function as an exometer
24. function (2) 24 histogram_stats = ~w(min max mean 95 90)a memory_stats = ~w(atom binary ets processes total)a polling_interval = 60_000 config :exometer, predefined: [ { ~w(erlang memory)a, {:function, :erlang, :memory, [], :proplist, memory_stats}, [] }, { ~w(erlang system_info)a, {:function, :erlang, :system_info, [:'$dp'], :value, [:process_count, :port_count, :thread_pool_size]}, [] } ], report: [ reporters: [ {:exometer_report_lager, []}, ], subscribers: [ {:exometer_report_lager, [:erlang, :memory], memory_stats, polling_interval, true}, {:exometer_report_lager, [:erlang, :system_info], [:process_count, :port_count, :thread_pool_size], polling_interval, true}, ] ] How to use Exometer
25. • Application information • api response count, api response time • query count, query exec time • Stats of the Erlang VM • memory usage • process count • garbage collection • reductions • io 25 works well with Exometer (pre-defined components) What to measure (again) What to measure (again) works well with Elixometer
26. 26 How we implemented HOW WE IMPLEMENTED
27. 27 Load Balancer Database sensu client sensu client sensu client Cache Apps Architecture Sensu / influxDB Grafana How we implemented
28. 28 Our requirements • Metrics of the whole system • outputs logs in JSON format, and transfers them to ElasticSearch via fluentd • -> Transforming metrics into JSON • Metrics of each server • uses sensu for monitoring • -> HTTP interface for sensu client How we implemented
29. 29 Load Balancer Database sensu client sensu client sensu client Cache Apps Sensu / influxDB Grafana How we implemented How we implemented -Transforming metrics into JSON- need some magic metrics
30. 30 • a Logger backend which outputs log files in JSON format • transforms metadata into JSON https://github.com/xflagstudio/elixir_logger_json_file_backend {"time":"2016-06-27 12:10:32.023","metric_value": 1443953,"metric_name":"erlang_memory_atom","message":"exomet er_report_lager: erlang_memory_atom: 1443953","level":"info","from":"exometer_report_larger"} {"time":"2016-06-27 12:10:32.023","metric_value": 114784072,"metric_name":"erlang_memory_binary","message":"ex ometer_report_lager: erlang_memory_binary: 114784072","level":"info","from":"exometer_report_larger"} LoggerJSONFileBackend How we implemented -Transforming metrics into JSON-
31. elixometer phoenix app exometer lager Logger lager_logger Logging metrics depends on lager conflicts LoggerJsonFileBackend forked exometer_core

32. 32 Load Balancer Database sensu client sensu client sensu client Cache Apps Sensu / influxDB Grafana How we implemented How we implemented -HTTP interface for sensu client-
33. elixometer phoenix app exometer HTTP Interface for sensu client 3 ets Grafana Sensu / InfluxDB easy to get a subscription list since dynamically defined, a bit tricky to get a subscription list directly from ets!
 (*exometer’s spiral doesn’t work as expected.) 33 How we implemented How we implemented -HTTP interface for sensu client-
34. 34 Other References • Monitoring Phoenix • Monitoring with Exometer: An Ongoing Love Story • ࣌Ӎಊ BOT αʔόʔ (Վ෣ب࠲.tech#5) • A guide to measuring your Elixir app • Feuerlabs/exometer_core References
35. 35 THANK YOU FOR LISTENING
36. 36 AND
37. 37 WE ARE HIRING!