Slide 1

Slide 1 text

Monitoring Containerized Elixir

Slide 2

Slide 2 text

.。oO(さっちゃんですよヾ(〃l _ l)ノ゙☆) @ Mackerel.io / Hatena.ne.jp

Slide 3

Slide 3 text

Imagine, you have a great Elixir app.

Slide 4

Slide 4 text

Imagine, you have a great Elixir app. And will be working it on Docker

Slide 5

Slide 5 text

Imagine, you have a great Elixir app. And will be working it on Docker on Kubernetes.

Slide 6

Slide 6 text

Why container, Docker? Container is an abstract layer over infrastructure. It runs everywhere. It easy to create, share, transfer & run.

Slide 7

Slide 7 text

Why Kubernetes? K8s is a common language. K8s can run every types of applications.

Slide 8

Slide 8 text

Elixir on Containers https://speakerdeck.com/ne_sachirou/elixir-on- containers s/:distillery/Mix.Release/

Slide 9

Slide 9 text

When you've deployed the app on K8s. How to monitor it?

Slide 10

Slide 10 text

Why Monitoring? Is your app works well? Know how your app works. What caused when your app is broken. Keep your app observable.

Slide 11

Slide 11 text

Liveness check Is your app is living or not?

Slide 12

Slide 12 text

K8s has liveness probe. --- apiVersion: apps/v1 kind: Deployment spec: template: spec: containers: - name: app livenessProbe: httpGet: path: /ops/heartbeat port: 4000

Slide 13

Slide 13 text

How to check your app is live or not?

Slide 14

Slide 14 text

KomachiHeartbeat https://hex.pm/packages/komachi_heartbeat Vital monitoring Elixir Web application.

Slide 15

Slide 15 text

KomachiHeartbeat The name is originate from a great Rubygems : https://rubygems.org/gems/komachi_heartbeat

Slide 16

Slide 16 text

Add KomachiHeartbeat. defmodule Example.Router do use Plug.Router plug(:match) plug(:dispatch) forward("/ops", to: KomachiHeartbeat) end defmodule ExampleWeb.Router do use ExampleWeb, :router forward("/ops", KomachiHeartbeat) end

Slide 17

Slide 17 text

Check it. % curl http://localhost/ops/heartbeat heartbeat:ok Also available from Zabbix, ALB health check or Mackerel's "External Http monitor (外形監視)".

Slide 18

Slide 18 text

Error collection Listening canary song.

Slide 19

Slide 19 text

Sentry https://sentry.io/ Collect & sort errors. Show the error environments: stacktrace, HTTP params, app host, release ver… Assign & resolve issue. Integrate with Sass: Slack, GitHub…

Slide 20

Slide 20 text

Add https://hex.pm/packages/sentry in your app. defmodule Example.Application do use Application def start(_type, _args) do Logger.add_backend(Sentry.LoggerBackend) … end defmodule Example.Router do use Sentry.Plug … end & add many metadata, by yourself (・﹏・)

Slide 21

Slide 21 text

Metrics monitoring Know the app behaviour & detect anomary.

Slide 22

Slide 22 text

Prometheus + Grafana + deadtrickster/beam-dashboards Beautiful. But to keep Prometheus servers have many toils.

Slide 23

Slide 23 text

Mackerel https://mackerel.io

Slide 24

Slide 24 text

Add mackerel-container-agent as a sidecar. --- apiVersion: apps/v1 kind: Deployment spec: template: spec: containers: - name: app … - name: mackerel-container-agent image: mackerel/mackerel-container-agent:latest imagePullPolicy: Always resources: limits: {memory: 128Mi} env: - {name: MACKEREL_CONTAINER_PLATFORM, value: kubernetes} - name: MACKEREL_KUBERNETES_KUBELET_HOST valueFrom: fieldRef: fieldPath: status.hostIP - name: MACKEREL_KUBERNETES_NAMESPACE valueFrom: fieldRef: {fieldPath: metadata.namespace} - name: MACKEREL_KUBERNETES_POD_NAME valueFrom: fieldRef: {fieldPath: metadata.name} - {name: MACKEREL_ROLES, value: example:app} envFrom: - secretRef: {name: mackerel}

Slide 25

Slide 25 text

apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization … secretGenerator: - name: mackerel type: Opaque files: [MACKEREL_APIKEY=secret/MACKEREL_APIKEY]

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

OK. I see container's metrics. How about BEAM metrics?

Slide 28

Slide 28 text

KomachiHeartbeat + mackerel- plugin-json https://hex.pm/packages/komachi_heartbeat Vital monitoring Elixir Web application. https://github.com/mackerelio/mackerel-plugin-json Json custom metrics plugin for mackerel.io agent.

Slide 29

Slide 29 text

KomachiHeartbeat has /stats endpoint that returns app's stats in JSON. mackerel-plugin-json posts any data gotten from JSON API as Mackerel custome metrics.

Slide 30

Slide 30 text

We have KomachiHeartbeat.BeamVital . https://github.com/ne- sachirou/ex_komachi_heartbeat/pull/29 (Sorry for under development.)

Slide 31

Slide 31 text

Add KomachiHeartbeat.BeamVital . defmodule TeijiBot.Router do use Plug.Router plug(:match) plug(:dispatch) forward( "/ops", to: KomachiHeartbeat, init_opts: [vitals: [KomachiHeartbeat.BeamVital]] ) end

Slide 32

Slide 32 text

Configure mackerel-plugin-json. --- apiVersion: apps/v1 kind: Deployment spec: template: spec: volumes: - name: mackerel-agent-config configMap: name: mackerel-agent-config containers: - name: app … - name: mackerel-container-agent image: mackerel/mackerel-container-agent:plugins … env: - {name: MACKEREL_AGENT_CONFIG, value: /etc/mackerel/mackerel.yaml} … volumeMounts: - name: mackerel-agent-config mountPath: /etc/mackerel readOnly: true

Slide 33

Slide 33 text

--- apiVersion: v1 kind: ConfigMap metadata: name: mackerel-agent-config data: mackerel.yaml: | plugin: metrics: json: command: | mackerel-plugin-json \ -url="http://localhost:4000/ops/stats" \ -prefix='beam'

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

Current metrics : atom_count memory_ets_tables memory_system_bytes_total atom, binary, code, ets, other port_count process_count (More, under development.)