Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Monitoring Containerized Elixir

Monitoring Containerized Elixir

#ElixirConfJP #CastleConf

4b071f90c5d9c0a58e2d9076460b7be4?s=128

さっちゃん

September 07, 2019
Tweet

Transcript

  1. Monitoring Containerized Elixir

  2. .。oO(さっちゃんですよヾ(〃l _ l)ノ゙☆) @ Mackerel.io / Hatena.ne.jp

  3. Imagine, you have a great Elixir app.

  4. Imagine, you have a great Elixir app. And will be

    working it on Docker
  5. Imagine, you have a great Elixir app. And will be

    working it on Docker on Kubernetes.
  6. Why container, Docker? Container is an abstract layer over infrastructure.

    It runs everywhere. It easy to create, share, transfer & run.
  7. Why Kubernetes? K8s is a common language. K8s can run

    every types of applications.
  8. Elixir on Containers https://speakerdeck.com/ne_sachirou/elixir-on- containers s/:distillery/Mix.Release/

  9. When you've deployed the app on K8s. How to monitor

    it?
  10. Why Monitoring? Is your app works well? Know how your

    app works. What caused when your app is broken. Keep your app observable.
  11. Liveness check Is your app is living or not?

  12. K8s has liveness probe. --- apiVersion: apps/v1 kind: Deployment spec:

    template: spec: containers: - name: app livenessProbe: httpGet: path: /ops/heartbeat port: 4000
  13. How to check your app is live or not?

  14. KomachiHeartbeat https://hex.pm/packages/komachi_heartbeat Vital monitoring Elixir Web application.

  15. KomachiHeartbeat The name is originate from a great Rubygems :

    https://rubygems.org/gems/komachi_heartbeat
  16. Add KomachiHeartbeat. defmodule Example.Router do use Plug.Router plug(:match) plug(:dispatch) forward("/ops",

    to: KomachiHeartbeat) end defmodule ExampleWeb.Router do use ExampleWeb, :router forward("/ops", KomachiHeartbeat) end
  17. Check it. % curl http://localhost/ops/heartbeat heartbeat:ok Also available from Zabbix,

    ALB health check or Mackerel's "External Http monitor (外形監視)".
  18. Error collection Listening canary song.

  19. Sentry https://sentry.io/ Collect & sort errors. Show the error environments:

    stacktrace, HTTP params, app host, release ver… Assign & resolve issue. Integrate with Sass: Slack, GitHub…
  20. Add https://hex.pm/packages/sentry in your app. defmodule Example.Application do use Application

    def start(_type, _args) do Logger.add_backend(Sentry.LoggerBackend) … end defmodule Example.Router do use Sentry.Plug … end & add many metadata, by yourself (・﹏・)
  21. Metrics monitoring Know the app behaviour & detect anomary.

  22. Prometheus + Grafana + deadtrickster/beam-dashboards Beautiful. But to keep Prometheus

    servers have many toils.
  23. Mackerel https://mackerel.io

  24. Add mackerel-container-agent as a sidecar. --- apiVersion: apps/v1 kind: Deployment

    spec: template: spec: containers: - name: app … - name: mackerel-container-agent image: mackerel/mackerel-container-agent:latest imagePullPolicy: Always resources: limits: {memory: 128Mi} env: - {name: MACKEREL_CONTAINER_PLATFORM, value: kubernetes} - name: MACKEREL_KUBERNETES_KUBELET_HOST valueFrom: fieldRef: fieldPath: status.hostIP - name: MACKEREL_KUBERNETES_NAMESPACE valueFrom: fieldRef: {fieldPath: metadata.namespace} - name: MACKEREL_KUBERNETES_POD_NAME valueFrom: fieldRef: {fieldPath: metadata.name} - {name: MACKEREL_ROLES, value: example:app} envFrom: - secretRef: {name: mackerel}
  25. apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization … secretGenerator: - name: mackerel type:

    Opaque files: [MACKEREL_APIKEY=secret/MACKEREL_APIKEY]
  26. None
  27. OK. I see container's metrics. How about BEAM metrics?

  28. KomachiHeartbeat + mackerel- plugin-json https://hex.pm/packages/komachi_heartbeat Vital monitoring Elixir Web application.

    https://github.com/mackerelio/mackerel-plugin-json Json custom metrics plugin for mackerel.io agent.
  29. KomachiHeartbeat has /stats endpoint that returns app's stats in JSON.

    mackerel-plugin-json posts any data gotten from JSON API as Mackerel custome metrics.
  30. We have KomachiHeartbeat.BeamVital . https://github.com/ne- sachirou/ex_komachi_heartbeat/pull/29 (Sorry for under development.)

  31. Add KomachiHeartbeat.BeamVital . defmodule TeijiBot.Router do use Plug.Router plug(:match) plug(:dispatch)

    forward( "/ops", to: KomachiHeartbeat, init_opts: [vitals: [KomachiHeartbeat.BeamVital]] ) end
  32. Configure mackerel-plugin-json. --- apiVersion: apps/v1 kind: Deployment spec: template: spec:

    volumes: - name: mackerel-agent-config configMap: name: mackerel-agent-config containers: - name: app … - name: mackerel-container-agent image: mackerel/mackerel-container-agent:plugins … env: - {name: MACKEREL_AGENT_CONFIG, value: /etc/mackerel/mackerel.yaml} … volumeMounts: - name: mackerel-agent-config mountPath: /etc/mackerel readOnly: true
  33. --- apiVersion: v1 kind: ConfigMap metadata: name: mackerel-agent-config data: mackerel.yaml:

    | plugin: metrics: json: command: | mackerel-plugin-json \ -url="http://localhost:4000/ops/stats" \ -prefix='beam'
  34. None
  35. Current metrics : atom_count memory_ets_tables memory_system_bytes_total atom, binary, code, ets,

    other port_count process_count (More, under development.)