Monitoring Containerized Elixir

Monitoring Containerized Elixir

#ElixirConfJP #CastleConf

4b071f90c5d9c0a58e2d9076460b7be4?s=128

さっちゃん

September 07, 2019
Tweet

Transcript

  1. 5.

    Imagine, you have a great Elixir app. And will be

    working it on Docker on Kubernetes.
  2. 6.

    Why container, Docker? Container is an abstract layer over infrastructure.

    It runs everywhere. It easy to create, share, transfer & run.
  3. 10.

    Why Monitoring? Is your app works well? Know how your

    app works. What caused when your app is broken. Keep your app observable.
  4. 12.

    K8s has liveness probe. --- apiVersion: apps/v1 kind: Deployment spec:

    template: spec: containers: - name: app livenessProbe: httpGet: path: /ops/heartbeat port: 4000
  5. 15.

    KomachiHeartbeat The name is originate from a great Rubygems :

    https://rubygems.org/gems/komachi_heartbeat
  6. 16.

    Add KomachiHeartbeat. defmodule Example.Router do use Plug.Router plug(:match) plug(:dispatch) forward("/ops",

    to: KomachiHeartbeat) end defmodule ExampleWeb.Router do use ExampleWeb, :router forward("/ops", KomachiHeartbeat) end
  7. 17.

    Check it. % curl http://localhost/ops/heartbeat heartbeat:ok Also available from Zabbix,

    ALB health check or Mackerel's "External Http monitor (外形監視)".
  8. 19.

    Sentry https://sentry.io/ Collect & sort errors. Show the error environments:

    stacktrace, HTTP params, app host, release ver… Assign & resolve issue. Integrate with Sass: Slack, GitHub…
  9. 20.

    Add https://hex.pm/packages/sentry in your app. defmodule Example.Application do use Application

    def start(_type, _args) do Logger.add_backend(Sentry.LoggerBackend) … end defmodule Example.Router do use Sentry.Plug … end & add many metadata, by yourself (・﹏・)
  10. 24.

    Add mackerel-container-agent as a sidecar. --- apiVersion: apps/v1 kind: Deployment

    spec: template: spec: containers: - name: app … - name: mackerel-container-agent image: mackerel/mackerel-container-agent:latest imagePullPolicy: Always resources: limits: {memory: 128Mi} env: - {name: MACKEREL_CONTAINER_PLATFORM, value: kubernetes} - name: MACKEREL_KUBERNETES_KUBELET_HOST valueFrom: fieldRef: fieldPath: status.hostIP - name: MACKEREL_KUBERNETES_NAMESPACE valueFrom: fieldRef: {fieldPath: metadata.namespace} - name: MACKEREL_KUBERNETES_POD_NAME valueFrom: fieldRef: {fieldPath: metadata.name} - {name: MACKEREL_ROLES, value: example:app} envFrom: - secretRef: {name: mackerel}
  11. 26.
  12. 28.

    KomachiHeartbeat + mackerel- plugin-json https://hex.pm/packages/komachi_heartbeat Vital monitoring Elixir Web application.

    https://github.com/mackerelio/mackerel-plugin-json Json custom metrics plugin for mackerel.io agent.
  13. 29.

    KomachiHeartbeat has /stats endpoint that returns app's stats in JSON.

    mackerel-plugin-json posts any data gotten from JSON API as Mackerel custome metrics.
  14. 31.

    Add KomachiHeartbeat.BeamVital . defmodule TeijiBot.Router do use Plug.Router plug(:match) plug(:dispatch)

    forward( "/ops", to: KomachiHeartbeat, init_opts: [vitals: [KomachiHeartbeat.BeamVital]] ) end
  15. 32.

    Configure mackerel-plugin-json. --- apiVersion: apps/v1 kind: Deployment spec: template: spec:

    volumes: - name: mackerel-agent-config configMap: name: mackerel-agent-config containers: - name: app … - name: mackerel-container-agent image: mackerel/mackerel-container-agent:plugins … env: - {name: MACKEREL_AGENT_CONFIG, value: /etc/mackerel/mackerel.yaml} … volumeMounts: - name: mackerel-agent-config mountPath: /etc/mackerel readOnly: true
  16. 33.

    --- apiVersion: v1 kind: ConfigMap metadata: name: mackerel-agent-config data: mackerel.yaml:

    | plugin: metrics: json: command: | mackerel-plugin-json \ -url="http://localhost:4000/ops/stats" \ -prefix='beam'
  17. 34.