Zen and the Art of Elixir Applications Monitoring

Application monitoring is one of the most crucial aspects that facilitates application's success.
In this talk, we will have a journey from metrics to alerting, along the way visiting major highlights including the aggregation and downsampling concerns, and understanding how this all is tied together.

Transcript

1. Zen and the Art of Elixir Applications Monitoring
2. lexmag
6. Why it is important How it works What to measure and react on
7. Why it is important
8. Why it is important
9. Preventing and shortening incidents
10. Personal growth
11. High-level overview
12. Host agents
13. Host agents
14. Retention and downsampling Before measurement value time temperature 21 9:11 temperature 24 9:12 temperature 21 9:13 temperature 23 9:14 temperature 22 9:15 temperature 23 9:16 After measurement value time temperature 22 9:13 temperature 23 9:16
15. Alerting measurement value time temperature 22 9:31 temperature 25 9:32 temperature 31 9:33 temperature 35 9:34 temperature 37 9:35 If temperature > 33 ⾠ The temperature value 35 is beyond the unacceptable limit
16. beam-telemetry/telemetry # Measurements %{ queue_time: 14, query_time: 57, decode_time: 4 } # Metadata %{ repo: "foo" }
17. Metric anatomy measurement labels value timestamp ecto_repo_queue_time repo=foo 14 123 ecto_repo_query_time repo=foo 57 123 ecto_repo_decode_time repo=foo 4 124
18. Metric anatomy measurement labels fields timestamp ecto_repo_exec repo=foo queue_time=14 query_time=57 decode_time=4 123
19. def handle_event([:my_app, :repo, :query], measurements, metadata, %{}) do fields = [] |> Fluxter.put_request_id() |> include_if_present(measurements, :queue_time) |> include_if_present(measurements, :query_time) |> include_if_present(measurements, :decode_time) MyApp.Fluxter.write("ecto_repo_exec", [repo: metadata.repo], fields) end defp include_if_present(fields, measurements, key) do case measurements do %{^key => value} -> microseconds = System.convert_time_unit(value, :native, :microsecond) [{key, microseconds} | fields] _ -> fields end end lexmag/fluxter
21. VM metrics :erlang.system_info/1 ‣ atom_count ‣ process_count ‣ port_count :erlang.process_info/2 ‣ message_queue_len :erlang.statistics/1 ‣ run_queue ‣ garbage_collection
24. If count(vm_port_limit, 1 minute) < 10 Outage detection
25. Reporting app start # We do this at compile-time as we don't have Mix available in releases. @version Mix.Project.config().version def start(_type, _args) do #... with {:ok, _} = result <- Supervisor.start_link(children, options) do MyApp.Fluxter.write("start", version: @version) result end end
27. App start notification If last_value(app_start.version) changed
28. App start anomaly If count(app_start, 30 seconds) > 10
29. Host metrics CPU (usage, idle, iowait, etc.)
 Memory (used, swap, etc.) Disks (free_space, utilization, etc.) Network (bytes in/out, drops, errors, etc.)
30. Request tracing measurement labels fields timestamp resp_time path=/home value=18 request_id=YpTiV 123 X-Request-ID: YpTiV
33. gist.github.com/lexmag Thank you