Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Monitoring Containerized Elixir

Monitoring Containerized Elixir

#ElixirConfJP #CastleConf

さっちゃん

September 07, 2019
Tweet

More Decks by さっちゃん

Other Decks in Programming

Transcript

  1. Monitoring
    Containerized Elixir

    View Slide

  2. .。oO(さっちゃんですよヾ(〃l _ l)ノ゙☆)
    @ Mackerel.io / Hatena.ne.jp

    View Slide

  3. Imagine, you have a great Elixir app.

    View Slide

  4. Imagine, you have a great Elixir app.
    And will be working it on Docker

    View Slide

  5. Imagine, you have a great Elixir app.
    And will be working it on Docker on Kubernetes.

    View Slide

  6. Why container, Docker?
    Container is an abstract layer over infrastructure.
    It runs everywhere.
    It easy to create, share, transfer & run.

    View Slide

  7. Why Kubernetes?
    K8s is a common language.
    K8s can run every types of applications.

    View Slide

  8. Elixir on Containers
    https://speakerdeck.com/ne_sachirou/elixir-on-
    containers
    s/:distillery/Mix.Release/

    View Slide

  9. When you've deployed the app on K8s.
    How to monitor it?

    View Slide

  10. Why Monitoring?
    Is your app works well?
    Know how your app works.
    What caused when your app is broken.
    Keep your app observable.

    View Slide

  11. Liveness check
    Is your app is living or not?

    View Slide

  12. K8s has liveness probe.
    ---
    apiVersion: apps/v1
    kind: Deployment
    spec:
    template:
    spec:
    containers:
    - name: app
    livenessProbe:
    httpGet:
    path: /ops/heartbeat
    port: 4000

    View Slide

  13. How to check your app is live or not?

    View Slide

  14. KomachiHeartbeat
    https://hex.pm/packages/komachi_heartbeat
    Vital monitoring Elixir Web application.

    View Slide

  15. KomachiHeartbeat
    The name is originate from a great Rubygems :
    https://rubygems.org/gems/komachi_heartbeat

    View Slide

  16. Add KomachiHeartbeat.
    defmodule Example.Router do
    use Plug.Router
    plug(:match)
    plug(:dispatch)
    forward("/ops", to: KomachiHeartbeat)
    end
    defmodule ExampleWeb.Router do
    use ExampleWeb, :router
    forward("/ops", KomachiHeartbeat)
    end

    View Slide

  17. Check it.
    % curl http://localhost/ops/heartbeat
    heartbeat:ok
    Also available from Zabbix, ALB health check or
    Mackerel's "External Http monitor (外形監視)".

    View Slide

  18. Error collection
    Listening canary song.

    View Slide

  19. Sentry
    https://sentry.io/
    Collect & sort errors.
    Show the error environments: stacktrace, HTTP
    params, app host, release ver…
    Assign & resolve issue.
    Integrate with Sass: Slack, GitHub…

    View Slide

  20. Add https://hex.pm/packages/sentry in your app.
    defmodule Example.Application do
    use Application
    def start(_type, _args) do
    Logger.add_backend(Sentry.LoggerBackend)

    end
    defmodule Example.Router do
    use Sentry.Plug

    end
    & add many metadata, by yourself (・﹏・)

    View Slide

  21. Metrics monitoring
    Know the app behaviour & detect anomary.

    View Slide

  22. Prometheus + Grafana
    +
    deadtrickster/beam-dashboards
    Beautiful. But to keep Prometheus servers have
    many toils.

    View Slide

  23. Mackerel
    https://mackerel.io

    View Slide

  24. Add mackerel-container-agent as a sidecar.
    ---
    apiVersion: apps/v1
    kind: Deployment
    spec:
    template:
    spec:
    containers:
    - name: app

    - name: mackerel-container-agent
    image: mackerel/mackerel-container-agent:latest
    imagePullPolicy: Always
    resources:
    limits: {memory: 128Mi}
    env:
    - {name: MACKEREL_CONTAINER_PLATFORM, value: kubernetes}
    - name: MACKEREL_KUBERNETES_KUBELET_HOST
    valueFrom:
    fieldRef:
    fieldPath: status.hostIP
    - name: MACKEREL_KUBERNETES_NAMESPACE
    valueFrom:
    fieldRef: {fieldPath: metadata.namespace}
    - name: MACKEREL_KUBERNETES_POD_NAME
    valueFrom:
    fieldRef: {fieldPath: metadata.name}
    - {name: MACKEREL_ROLES, value: example:app}
    envFrom:
    - secretRef: {name: mackerel}

    View Slide

  25. apiVersion: kustomize.config.k8s.io/v1beta1
    kind: Kustomization

    secretGenerator:
    - name: mackerel
    type: Opaque
    files: [MACKEREL_APIKEY=secret/MACKEREL_APIKEY]

    View Slide

  26. View Slide

  27. OK. I see container's metrics.
    How about BEAM metrics?

    View Slide

  28. KomachiHeartbeat + mackerel-
    plugin-json
    https://hex.pm/packages/komachi_heartbeat
    Vital monitoring Elixir Web application.
    https://github.com/mackerelio/mackerel-plugin-json
    Json custom metrics plugin for mackerel.io agent.

    View Slide

  29. KomachiHeartbeat has /stats endpoint that
    returns app's stats in JSON.
    mackerel-plugin-json posts any data gotten from
    JSON API as Mackerel custome metrics.

    View Slide

  30. We have KomachiHeartbeat.BeamVital .
    https://github.com/ne-
    sachirou/ex_komachi_heartbeat/pull/29
    (Sorry for under development.)

    View Slide

  31. Add KomachiHeartbeat.BeamVital .
    defmodule TeijiBot.Router do
    use Plug.Router
    plug(:match)
    plug(:dispatch)
    forward(
    "/ops",
    to: KomachiHeartbeat,
    init_opts: [vitals: [KomachiHeartbeat.BeamVital]]
    )
    end

    View Slide

  32. Configure mackerel-plugin-json.
    ---
    apiVersion: apps/v1
    kind: Deployment
    spec:
    template:
    spec:
    volumes:
    - name: mackerel-agent-config
    configMap:
    name: mackerel-agent-config
    containers:
    - name: app

    - name: mackerel-container-agent
    image: mackerel/mackerel-container-agent:plugins

    env:
    - {name: MACKEREL_AGENT_CONFIG, value: /etc/mackerel/mackerel.yaml}

    volumeMounts:
    - name: mackerel-agent-config
    mountPath: /etc/mackerel
    readOnly: true

    View Slide

  33. ---
    apiVersion: v1
    kind: ConfigMap
    metadata:
    name: mackerel-agent-config
    data:
    mackerel.yaml: |
    plugin:
    metrics:
    json:
    command: |
    mackerel-plugin-json \
    -url="http://localhost:4000/ops/stats" \
    -prefix='beam'

    View Slide

  34. View Slide

  35. Current metrics :
    atom_count
    memory_ets_tables
    memory_system_bytes_total
    atom, binary, code, ets, other
    port_count
    process_count
    (More, under development.)

    View Slide