1 JUN SUMIDA PRACTICAL PHOENIX MONITORING

2 Introduction @junsumida Jun Sumida self()
 |> at( XFLAG™ STUDIO, mixi, inc.) |> develops(game backend servers) |> works(elixir, ruby, javascript, devops) |> github(junsumida)

3 Introduction • What to measure • How to measure • How to use Exo/Elixometer • How we Implemented Table of Contents

4 WHAT TO MEASURE What to measure

• Application information • api request/response count, api response time • query count, query exec time • Stats of the Erlang VM • memory usage • process count • garbage collection • reductions • io 5 What to measure What to measure

6 How to measure HOW TO MEASURE

7 Metric libraries • Erlang • exometer • folsom • vmstats • Elixir • Elixometer • Beaker • metrix see also https://github.com/h4cc/awesome-elixir#debugging How to measure

8 Metric libraries • Erlang • exometer • folsom • vmstats • Elixir • Elixometer • Beaker • metrix see also https://github.com/h4cc/awesome-elixir#debugging How to measure

9 Exometer https://github.com/Feuerlabs/exometer • a great measurement tool written in Erlang • developed by Basho (Riak) • easy to use (probably, maybe, possibly…) • multipule backends • InfluxDB, loggers, Graphite, etc… How to measure

10 Elixometer https://github.com/pinterest/elixometer • a light wrapper around exometer • developed by Pinterest • easy to use (i swear!) • no complex configurations How to measure

Why Exo/Elixometer? 11 PROS • well-documented • both of them “were” well-maintained • still better than most of other measuring libraries • comparatively easy to find examples How to measure

Why Exo/Elixometer? 12 CONS • dependency hell • lager vs Logger • configuring exometer is a little bit confusing • struggling with Erlang due to the reporters anyway How to measure

13 HOW TO USE EXO/ELIXOMETER What to measure

14 ELIXOMETER • counter • histogram • timer • gauge • spiral How to use Elixometer

15 How to use Elixometer • ordinary histograms • stores all values during a configured time span • provides some analysis • min, max • mean, median • percentile histogram (1)

16 histogram (2) defmodule Myapp.Plug.Metrics do import Plug.Conn use Elixometer def call(conn, _default) do before_time = :os.timestamp diff = :timer.now_diff(:os.timestamp, before_time) conn |> register_before_send(fn conn -> update_histogram("response_time", diff) end) end end microseconds How to use Elixometer

3 histogram (3) What to measure

• a sort of histogram • useful when measuring the performance of a function 18 timer use Elixometer @timed(key: :"timed.super_heavy") def super_heavy do HeavyModule.call_something end def triple_super_heavy do timed("timed.triple.super_heavy") do HeavyModule.call_something HeavyModule.call_something HeavyModule.call_something end end annotation do block How to use Elixometer

19 • holds one numeric value • always returns the most recently provided one • good for metrics like memory usage or disk space gauge photo credit: Pressure Cooker - All American Model 910 - Pressure gauge How to use Elixometer

20 • increases value over time • when reporting the value, the metric is reset • good for metrics like queries per second or requests per second spiral (1) photo credit: University of Washington How to use Elixometer

spiral (2) 21 # config/config.exs config :elixometer, reporter: :exometer_report_lager, env: "webapp", metric_prefix: "myapp", update_frequency: 10_000 config :myapp, Myapp.Repo, adapter: Ecto.Adapters.MySQL, username: "root", password: "", loggers: [{Myapp.Repo.Metrics, :record_metric, []}], hostname: "localhost", database: "myapp", # lib/myapp/repo/metrics.ex defmodule Myapp.Repo.Metrics do use Elixometer def record_metric(entry) do update_spiral("query_count", 1) end end How to use Elixometer

22 EXOMETER • function • counter • gauge • duration • histogram How to use Exometer

function (1) 23 How to use Exometer allows to use Erlang function as an exometer

function (2) 24 histogram_stats = ~w(min max mean 95 90)a memory_stats = ~w(atom binary ets processes total)a polling_interval = 60_000 config :exometer, predefined: [ { ~w(erlang memory)a, {:function, :erlang, :memory, [], :proplist, memory_stats}, [] }, { ~w(erlang system_info)a, {:function, :erlang, :system_info, [:'$dp'], :value, [:process_count, :port_count, :thread_pool_size]}, [] } ], report: [ reporters: [ {:exometer_report_lager, []}, ], subscribers: [ {:exometer_report_lager, [:erlang, :memory], memory_stats, polling_interval, true}, {:exometer_report_lager, [:erlang, :system_info], [:process_count, :port_count, :thread_pool_size], polling_interval, true}, ] ] How to use Exometer

• Application information • api response count, api response time • query count, query exec time • Stats of the Erlang VM • memory usage • process count • garbage collection • reductions • io 25 works well with Exometer (pre-defined components) What to measure (again) What to measure (again) works well with Elixometer

26 How we implemented HOW WE IMPLEMENTED

27 Load Balancer Database sensu client sensu client sensu client Cache Apps Architecture Sensu / influxDB Grafana How we implemented

28 Our requirements • Metrics of the whole system • outputs logs in JSON format, and transfers them to ElasticSearch via fluentd • -> Transforming metrics into JSON • Metrics of each server • uses sensu for monitoring • -> HTTP interface for sensu client How we implemented

29 Load Balancer Database sensu client sensu client sensu client Cache Apps Sensu / influxDB Grafana How we implemented How we implemented -Transforming metrics into JSON- need some magic metrics

30 • a Logger backend which outputs log files in JSON format • transforms metadata into JSON https://github.com/xflagstudio/elixir_logger_json_file_backend {"time":"2016-06-27 12:10:32.023","metric_value": 1443953,"metric_name":"erlang_memory_atom","message":"exomet er_report_lager: erlang_memory_atom: 1443953","level":"info","from":"exometer_report_larger"} {"time":"2016-06-27 12:10:32.023","metric_value": 114784072,"metric_name":"erlang_memory_binary","message":"ex ometer_report_lager: erlang_memory_binary: 114784072","level":"info","from":"exometer_report_larger"} LoggerJSONFileBackend How we implemented -Transforming metrics into JSON-

elixometer phoenix app exometer lager Logger lager_logger Logging metrics depends on lager conflicts LoggerJsonFileBackend forked exometer_core


32 Load Balancer Database sensu client sensu client sensu client Cache Apps Sensu / influxDB Grafana How we implemented How we implemented -HTTP interface for sensu client-

elixometer phoenix app exometer HTTP Interface for sensu client 3 ets Grafana Sensu / InfluxDB easy to get a subscription list since dynamically defined, a bit tricky to get a subscription list directly from ets!
 (*exometer’s spiral doesn’t work as expected.) 33 How we implemented How we implemented -HTTP interface for sensu client-

34 Other References • Monitoring Phoenix • Monitoring with Exometer: An Ongoing Love Story • ࣌Ӎಊ BOT αʔόʔ (Վ෣ب࠲.tech#5) • A guide to measuring your Elixir app • Feuerlabs/exometer_core References

35 THANK YOU FOR LISTENING

36 AND

37 WE ARE HIRING!