Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Monitoring Containerized Elixir

Monitoring Containerized Elixir

#ElixirConfJP #CastleConf


September 07, 2019

More Decks by さっちゃん

Other Decks in Programming


  1. Imagine, you have a great Elixir app. And will be

    working it on Docker on Kubernetes.
  2. Why container, Docker? Container is an abstract layer over infrastructure.

    It runs everywhere. It easy to create, share, transfer & run.
  3. Why Monitoring? Is your app works well? Know how your

    app works. What caused when your app is broken. Keep your app observable.
  4. K8s has liveness probe. --- apiVersion: apps/v1 kind: Deployment spec:

    template: spec: containers: - name: app livenessProbe: httpGet: path: /ops/heartbeat port: 4000
  5. KomachiHeartbeat The name is originate from a great Rubygems :

  6. Add KomachiHeartbeat. defmodule Example.Router do use Plug.Router plug(:match) plug(:dispatch) forward("/ops",

    to: KomachiHeartbeat) end defmodule ExampleWeb.Router do use ExampleWeb, :router forward("/ops", KomachiHeartbeat) end
  7. Check it. % curl http://localhost/ops/heartbeat heartbeat:ok Also available from Zabbix,

    ALB health check or Mackerel's "External Http monitor (外形監視)".
  8. Sentry https://sentry.io/ Collect & sort errors. Show the error environments:

    stacktrace, HTTP params, app host, release ver… Assign & resolve issue. Integrate with Sass: Slack, GitHub…
  9. Add https://hex.pm/packages/sentry in your app. defmodule Example.Application do use Application

    def start(_type, _args) do Logger.add_backend(Sentry.LoggerBackend) … end defmodule Example.Router do use Sentry.Plug … end & add many metadata, by yourself (・﹏・)
  10. Add mackerel-container-agent as a sidecar. --- apiVersion: apps/v1 kind: Deployment

    spec: template: spec: containers: - name: app … - name: mackerel-container-agent image: mackerel/mackerel-container-agent:latest imagePullPolicy: Always resources: limits: {memory: 128Mi} env: - {name: MACKEREL_CONTAINER_PLATFORM, value: kubernetes} - name: MACKEREL_KUBERNETES_KUBELET_HOST valueFrom: fieldRef: fieldPath: status.hostIP - name: MACKEREL_KUBERNETES_NAMESPACE valueFrom: fieldRef: {fieldPath: metadata.namespace} - name: MACKEREL_KUBERNETES_POD_NAME valueFrom: fieldRef: {fieldPath: metadata.name} - {name: MACKEREL_ROLES, value: example:app} envFrom: - secretRef: {name: mackerel}
  11. KomachiHeartbeat + mackerel- plugin-json https://hex.pm/packages/komachi_heartbeat Vital monitoring Elixir Web application.

    https://github.com/mackerelio/mackerel-plugin-json Json custom metrics plugin for mackerel.io agent.
  12. KomachiHeartbeat has /stats endpoint that returns app's stats in JSON.

    mackerel-plugin-json posts any data gotten from JSON API as Mackerel custome metrics.
  13. Add KomachiHeartbeat.BeamVital . defmodule TeijiBot.Router do use Plug.Router plug(:match) plug(:dispatch)

    forward( "/ops", to: KomachiHeartbeat, init_opts: [vitals: [KomachiHeartbeat.BeamVital]] ) end
  14. Configure mackerel-plugin-json. --- apiVersion: apps/v1 kind: Deployment spec: template: spec:

    volumes: - name: mackerel-agent-config configMap: name: mackerel-agent-config containers: - name: app … - name: mackerel-container-agent image: mackerel/mackerel-container-agent:plugins … env: - {name: MACKEREL_AGENT_CONFIG, value: /etc/mackerel/mackerel.yaml} … volumeMounts: - name: mackerel-agent-config mountPath: /etc/mackerel readOnly: true
  15. --- apiVersion: v1 kind: ConfigMap metadata: name: mackerel-agent-config data: mackerel.yaml:

    | plugin: metrics: json: command: | mackerel-plugin-json \ -url="http://localhost:4000/ops/stats" \ -prefix='beam'