Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Practical Phoenix Monitoring

Practical Phoenix Monitoring

* what to measure
* how to mesure it
* how to use exometer and elixometer
* how we implemented

D4715d085532a801bb2040ce5d9d55ec?s=128

junsumida

June 30, 2016
Tweet

Transcript

  1. 1 JUN SUMIDA PRACTICAL PHOENIX MONITORING

  2. 2 Introduction @junsumida Jun Sumida self()
 |> at( XFLAG™ STUDIO,

    mixi, inc.) |> develops(game backend servers) |> works(elixir, ruby, javascript, devops) |> github(junsumida)
  3. 3 Introduction • What to measure • How to measure

    • How to use Exo/Elixometer • How we Implemented Table of Contents
  4. 4 WHAT TO MEASURE What to measure

  5. • Application information • api request/response count, api response time

    • query count, query exec time • Stats of the Erlang VM • memory usage • process count • garbage collection • reductions • io 5 What to measure What to measure
  6. 6 How to measure HOW TO MEASURE

  7. 7 Metric libraries • Erlang • exometer • folsom •

    vmstats • Elixir • Elixometer • Beaker • metrix see also https://github.com/h4cc/awesome-elixir#debugging How to measure
  8. 8 Metric libraries • Erlang • exometer • folsom •

    vmstats • Elixir • Elixometer • Beaker • metrix see also https://github.com/h4cc/awesome-elixir#debugging How to measure
  9. 9 Exometer https://github.com/Feuerlabs/exometer • a great measurement tool written in

    Erlang • developed by Basho (Riak) • easy to use (probably, maybe, possibly…) • multipule backends • InfluxDB, loggers, Graphite, etc… How to measure
  10. 10 Elixometer https://github.com/pinterest/elixometer • a light wrapper around exometer •

    developed by Pinterest • easy to use (i swear!) • no complex configurations How to measure
  11. Why Exo/Elixometer? 11 PROS • well-documented • both of them

    “were” well-maintained • still better than most of other measuring libraries • comparatively easy to find examples How to measure
  12. Why Exo/Elixometer? 12 CONS • dependency hell • lager vs

    Logger • configuring exometer is a little bit confusing • struggling with Erlang due to the reporters anyway How to measure
  13. 13 HOW TO USE EXO/ELIXOMETER What to measure

  14. 14 ELIXOMETER • counter • histogram • timer • gauge

    • spiral How to use Elixometer
  15. 15 How to use Elixometer • ordinary histograms • stores

    all values during a configured time span • provides some analysis • min, max • mean, median • percentile histogram (1)
  16. 16 histogram (2) defmodule Myapp.Plug.Metrics do import Plug.Conn use Elixometer

    def call(conn, _default) do before_time = :os.timestamp diff = :timer.now_diff(:os.timestamp, before_time) conn |> register_before_send(fn conn -> update_histogram("response_time", diff) end) end end microseconds How to use Elixometer
  17. 3 histogram (3) What to measure

  18. • a sort of histogram • useful when measuring the

    performance of a function 18 timer use Elixometer @timed(key: :"timed.super_heavy") def super_heavy do HeavyModule.call_something end def triple_super_heavy do timed("timed.triple.super_heavy") do HeavyModule.call_something HeavyModule.call_something HeavyModule.call_something end end annotation do block How to use Elixometer
  19. 19 • holds one numeric value • always returns the

    most recently provided one • good for metrics like memory usage or disk space gauge photo credit: Pressure Cooker - All American Model 910 - Pressure gauge How to use Elixometer
  20. 20 • increases value over time • when reporting the

    value, the metric is reset • good for metrics like queries per second or requests per second spiral (1) photo credit: University of Washington How to use Elixometer
  21. spiral (2) 21 # config/config.exs config :elixometer, reporter: :exometer_report_lager, env:

    "webapp", metric_prefix: "myapp", update_frequency: 10_000 config :myapp, Myapp.Repo, adapter: Ecto.Adapters.MySQL, username: "root", password: "", loggers: [{Myapp.Repo.Metrics, :record_metric, []}], hostname: "localhost", database: "myapp", # lib/myapp/repo/metrics.ex defmodule Myapp.Repo.Metrics do use Elixometer def record_metric(entry) do update_spiral("query_count", 1) end end How to use Elixometer
  22. 22 EXOMETER • function • counter • gauge • duration

    • histogram How to use Exometer
  23. function (1) 23 How to use Exometer allows to use

    Erlang function as an exometer
  24. function (2) 24 histogram_stats = ~w(min max mean 95 90)a

    memory_stats = ~w(atom binary ets processes total)a polling_interval = 60_000 config :exometer, predefined: [ { ~w(erlang memory)a, {:function, :erlang, :memory, [], :proplist, memory_stats}, [] }, { ~w(erlang system_info)a, {:function, :erlang, :system_info, [:'$dp'], :value, [:process_count, :port_count, :thread_pool_size]}, [] } ], report: [ reporters: [ {:exometer_report_lager, []}, ], subscribers: [ {:exometer_report_lager, [:erlang, :memory], memory_stats, polling_interval, true}, {:exometer_report_lager, [:erlang, :system_info], [:process_count, :port_count, :thread_pool_size], polling_interval, true}, ] ] How to use Exometer
  25. • Application information • api response count, api response time

    • query count, query exec time • Stats of the Erlang VM • memory usage • process count • garbage collection • reductions • io 25 works well with Exometer (pre-defined components) What to measure (again) What to measure (again) works well with Elixometer
  26. 26 How we implemented HOW WE IMPLEMENTED

  27. 27 Load Balancer Database sensu client sensu client sensu client

    Cache Apps Architecture Sensu / influxDB Grafana How we implemented
  28. 28 Our requirements • Metrics of the whole system •

    outputs logs in JSON format, and transfers them to ElasticSearch via fluentd • -> Transforming metrics into JSON • Metrics of each server • uses sensu for monitoring • -> HTTP interface for sensu client How we implemented
  29. 29 Load Balancer Database sensu client sensu client sensu client

    Cache Apps Sensu / influxDB Grafana How we implemented How we implemented -Transforming metrics into JSON- need some magic metrics
  30. 30 • a Logger backend which outputs log files in

    JSON format • transforms metadata into JSON https://github.com/xflagstudio/elixir_logger_json_file_backend {"time":"2016-06-27 12:10:32.023","metric_value": 1443953,"metric_name":"erlang_memory_atom","message":"exomet er_report_lager: erlang_memory_atom: 1443953","level":"info","from":"exometer_report_larger"} {"time":"2016-06-27 12:10:32.023","metric_value": 114784072,"metric_name":"erlang_memory_binary","message":"ex ometer_report_lager: erlang_memory_binary: 114784072","level":"info","from":"exometer_report_larger"} LoggerJSONFileBackend How we implemented -Transforming metrics into JSON-
  31. elixometer phoenix app exometer lager Logger lager_logger Logging metrics depends

    on lager conflicts LoggerJsonFileBackend forked exometer_core

  32. 32 Load Balancer Database sensu client sensu client sensu client

    Cache Apps Sensu / influxDB Grafana How we implemented How we implemented -HTTP interface for sensu client-
  33. elixometer phoenix app exometer HTTP Interface for sensu client 3

    ets Grafana Sensu / InfluxDB easy to get a subscription list since dynamically defined, a bit tricky to get a subscription list directly from ets!
 (*exometer’s spiral doesn’t work as expected.) 33 How we implemented How we implemented -HTTP interface for sensu client-
  34. 34 Other References • Monitoring Phoenix • Monitoring with Exometer:

    An Ongoing Love Story • ࣌Ӎಊ BOT αʔόʔ (Վ෣ب࠲.tech#5) • A guide to measuring your Elixir app • Feuerlabs/exometer_core References
  35. 35 THANK YOU FOR LISTENING

  36. 36 AND

  37. 37 WE ARE HIRING!