Monitoring Containerized Elixir

#ElixirConfJP #CastleConf

Transcript

1. Monitoring
   Containerized Elixir
   

   View Slide

2. .｡oO(さっちゃんですよヾ(〃l _ l)ﾉﾞ☆)
   @ Mackerel.io / Hatena.ne.jp
   

   View Slide

3. Imagine, you have a great Elixir app.
   

   View Slide

4. Imagine, you have a great Elixir app.
   And will be working it on Docker
   

   View Slide

5. Imagine, you have a great Elixir app.
   And will be working it on Docker on Kubernetes.
   

   View Slide

6. Why container, Docker?
   Container is an abstract layer over infrastructure.
   It runs everywhere.
   It easy to create, share, transfer & run.
   

   View Slide

7. Why Kubernetes?
   K8s is a common language.
   K8s can run every types of applications.
   

   View Slide

8. Elixir on Containers
   https://speakerdeck.com/ne_sachirou/elixir-on-
   containers
   s/:distillery/Mix.Release/
   

   View Slide

9. When you've deployed the app on K8s.
   How to monitor it?
   

   View Slide

10. Why Monitoring?
    Is your app works well?
    Know how your app works.
    What caused when your app is broken.
    Keep your app observable.
    

    View Slide

11. Liveness check
    Is your app is living or not?
    

    View Slide

12. K8s has liveness probe.
    ---
    apiVersion: apps/v1
    kind: Deployment
    spec:
    template:
    spec:
    containers:
    - name: app
    livenessProbe:
    httpGet:
    path: /ops/heartbeat
    port: 4000
    

    View Slide

13. How to check your app is live or not?
    

    View Slide

14. KomachiHeartbeat
    https://hex.pm/packages/komachi_heartbeat
    Vital monitoring Elixir Web application.
    

    View Slide

15. KomachiHeartbeat
    The name is originate from a great Rubygems :
    https://rubygems.org/gems/komachi_heartbeat
    

    View Slide

16. Add KomachiHeartbeat.
    defmodule Example.Router do
    use Plug.Router
    plug(:match)
    plug(:dispatch)
    forward("/ops", to: KomachiHeartbeat)
    end
    defmodule ExampleWeb.Router do
    use ExampleWeb, :router
    forward("/ops", KomachiHeartbeat)
    end
    

    View Slide

17. Check it.
    % curl http://localhost/ops/heartbeat
    heartbeat:ok
    Also available from Zabbix, ALB health check or
    Mackerel's "External Http monitor (外形監視)".
    

    View Slide

18. Error collection
    Listening canary song.
    

    View Slide

19. Sentry
    https://sentry.io/
    Collect & sort errors.
    Show the error environments: stacktrace, HTTP
    params, app host, release ver…
    Assign & resolve issue.
    Integrate with Sass: Slack, GitHub…
    

    View Slide

20. Add https://hex.pm/packages/sentry in your app.
    defmodule Example.Application do
    use Application
    def start(_type, _args) do
    Logger.add_backend(Sentry.LoggerBackend)
    …
    end
    defmodule Example.Router do
    use Sentry.Plug
    …
    end
    & add many metadata, by yourself (・﹏・)
    

    View Slide

21. Metrics monitoring
    Know the app behaviour & detect anomary.
    

    View Slide

22. Prometheus + Grafana
    +
    deadtrickster/beam-dashboards
    Beautiful. But to keep Prometheus servers have
    many toils.
    

    View Slide

23. Mackerel
    https://mackerel.io
    

    View Slide

24. Add mackerel-container-agent as a sidecar.
    ---
    apiVersion: apps/v1
    kind: Deployment
    spec:
    template:
    spec:
    containers:
    - name: app
    …
    - name: mackerel-container-agent
    image: mackerel/mackerel-container-agent:latest
    imagePullPolicy: Always
    resources:
    limits: {memory: 128Mi}
    env:
    - {name: MACKEREL_CONTAINER_PLATFORM, value: kubernetes}
    - name: MACKEREL_KUBERNETES_KUBELET_HOST
    valueFrom:
    fieldRef:
    fieldPath: status.hostIP
    - name: MACKEREL_KUBERNETES_NAMESPACE
    valueFrom:
    fieldRef: {fieldPath: metadata.namespace}
    - name: MACKEREL_KUBERNETES_POD_NAME
    valueFrom:
    fieldRef: {fieldPath: metadata.name}
    - {name: MACKEREL_ROLES, value: example:app}
    envFrom:
    - secretRef: {name: mackerel}
    

    View Slide

25. apiVersion: kustomize.config.k8s.io/v1beta1
    kind: Kustomization
    …
    secretGenerator:
    - name: mackerel
    type: Opaque
    files: [MACKEREL_APIKEY=secret/MACKEREL_APIKEY]
    

    View Slide

26. View Slide

27. OK. I see container's metrics.
    How about BEAM metrics?
    

    View Slide

28. KomachiHeartbeat + mackerel-
    plugin-json
    https://hex.pm/packages/komachi_heartbeat
    Vital monitoring Elixir Web application.
    https://github.com/mackerelio/mackerel-plugin-json
    Json custom metrics plugin for mackerel.io agent.
    

    View Slide

29. KomachiHeartbeat has /stats endpoint that
    returns app's stats in JSON.
    mackerel-plugin-json posts any data gotten from
    JSON API as Mackerel custome metrics.
    

    View Slide

30. We have KomachiHeartbeat.BeamVital .
    https://github.com/ne-
    sachirou/ex_komachi_heartbeat/pull/29
    (Sorry for under development.)
    

    View Slide

31. Add KomachiHeartbeat.BeamVital .
    defmodule TeijiBot.Router do
    use Plug.Router
    plug(:match)
    plug(:dispatch)
    forward(
    "/ops",
    to: KomachiHeartbeat,
    init_opts: [vitals: [KomachiHeartbeat.BeamVital]]
    )
    end
    

    View Slide

32. Configure mackerel-plugin-json.
    ---
    apiVersion: apps/v1
    kind: Deployment
    spec:
    template:
    spec:
    volumes:
    - name: mackerel-agent-config
    configMap:
    name: mackerel-agent-config
    containers:
    - name: app
    …
    - name: mackerel-container-agent
    image: mackerel/mackerel-container-agent:plugins
    …
    env:
    - {name: MACKEREL_AGENT_CONFIG, value: /etc/mackerel/mackerel.yaml}
    …
    volumeMounts:
    - name: mackerel-agent-config
    mountPath: /etc/mackerel
    readOnly: true
    

    View Slide

33. ---
    apiVersion: v1
    kind: ConfigMap
    metadata:
    name: mackerel-agent-config
    data:
    mackerel.yaml: |
    plugin:
    metrics:
    json:
    command: |
    mackerel-plugin-json \
    -url="http://localhost:4000/ops/stats" \
    -prefix='beam'
    

    View Slide

34. View Slide

35. Current metrics :
    atom_count
    memory_ets_tables
    memory_system_bytes_total
    atom, binary, code, ets, other
    port_count
    process_count
    (More, under development.)
    

    View Slide