Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Monitoring Containerized Elixir
さっちゃん
September 07, 2019
Programming
1
520
Monitoring Containerized Elixir
#ElixirConfJP #CastleConf
さっちゃん
September 07, 2019
Tweet
Share
More Decks by さっちゃん
See All by さっちゃん
名實一致
ne_sachirou
0
470
まかれるあなとみあ ―Mackerel のしくみを理解する 30 分― @ Hatena Engineer Seminar #16
ne_sachirou
0
2.3k
tacit programming : Point-free, Concatenatives & J
ne_sachirou
0
330
Let's create stateful systems, by Elixir
ne_sachirou
1
520
Phoenix LiveReact
ne_sachirou
1
280
Phoenix LiveView チュートリアル
ne_sachirou
1
110
DDD: Data Driven Development
ne_sachirou
6
4.4k
Elixir on Containers
ne_sachirou
1
620
發言の超越論的な根拠
ne_sachirou
1
170
Other Decks in Programming
See All in Programming
Why Airflow? & What's new in Airflow 2.3?
kaxil
0
110
即、New Relic / New Relic NOW!
uzulla
0
270
Beyond Micro Frontends: Frontend Moduliths for the Enterprise @wad2022
manfredsteyer
PRO
0
130
Gitlab CIでMRを自動生成する
forcia_dev_pr
0
110
Baseline Profilesでアプリのパフォーマンスを向上させる / Improve app performance with Baseline Profiles
numeroanddev
0
230
1時間半で克服するJavaScriptの非同期処理/async_javascript_kokufuku
marchin1989
2
600
Value and Record Types
hschwentner
0
550
Beyond Micro Frontends: Frontend Moduliths for the Enterprise @enterjs2022
manfredsteyer
PRO
0
120
Jetpack Compose, 어디까지 알고 있을까?
jisungbin
0
100
Reactアプリケーションのテスト戦略
0906koki
10
4.6k
JSのウェブフレームワークで高速なルーターを実装する方法
usualoma
1
1.7k
言語処理ライブラリ開発における失敗談 / NLPHacks
taishii
1
430
Featured
See All Featured
Java REST API Framework Comparison - PWX 2021
mraible
PRO
11
4.7k
How New CSS Is Changing Everything About Graphic Design on the Web
jensimmons
213
11k
Building a Scalable Design System with Sketch
lauravandoore
447
30k
Docker and Python
trallard
27
1.6k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
15
940
Put a Button on it: Removing Barriers to Going Fast.
kastner
56
2.3k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
655
120k
The Cult of Friendly URLs
andyhume
68
4.8k
Learning to Love Humans: Emotional Interface Design
aarron
261
37k
From Idea to $5000 a Month in 5 Months
shpigford
373
44k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
236
1M
Imperfection Machines: The Place of Print at Facebook
scottboms
253
12k
Transcript
Monitoring Containerized Elixir
.。oO(さっちゃんですよヾ(〃l _ l)ノ゙☆) @ Mackerel.io / Hatena.ne.jp
Imagine, you have a great Elixir app.
Imagine, you have a great Elixir app. And will be
working it on Docker
Imagine, you have a great Elixir app. And will be
working it on Docker on Kubernetes.
Why container, Docker? Container is an abstract layer over infrastructure.
It runs everywhere. It easy to create, share, transfer & run.
Why Kubernetes? K8s is a common language. K8s can run
every types of applications.
Elixir on Containers https://speakerdeck.com/ne_sachirou/elixir-on- containers s/:distillery/Mix.Release/
When you've deployed the app on K8s. How to monitor
it?
Why Monitoring? Is your app works well? Know how your
app works. What caused when your app is broken. Keep your app observable.
Liveness check Is your app is living or not?
K8s has liveness probe. --- apiVersion: apps/v1 kind: Deployment spec:
template: spec: containers: - name: app livenessProbe: httpGet: path: /ops/heartbeat port: 4000
How to check your app is live or not?
KomachiHeartbeat https://hex.pm/packages/komachi_heartbeat Vital monitoring Elixir Web application.
KomachiHeartbeat The name is originate from a great Rubygems :
https://rubygems.org/gems/komachi_heartbeat
Add KomachiHeartbeat. defmodule Example.Router do use Plug.Router plug(:match) plug(:dispatch) forward("/ops",
to: KomachiHeartbeat) end defmodule ExampleWeb.Router do use ExampleWeb, :router forward("/ops", KomachiHeartbeat) end
Check it. % curl http://localhost/ops/heartbeat heartbeat:ok Also available from Zabbix,
ALB health check or Mackerel's "External Http monitor (外形監視)".
Error collection Listening canary song.
Sentry https://sentry.io/ Collect & sort errors. Show the error environments:
stacktrace, HTTP params, app host, release ver… Assign & resolve issue. Integrate with Sass: Slack, GitHub…
Add https://hex.pm/packages/sentry in your app. defmodule Example.Application do use Application
def start(_type, _args) do Logger.add_backend(Sentry.LoggerBackend) … end defmodule Example.Router do use Sentry.Plug … end & add many metadata, by yourself (・﹏・)
Metrics monitoring Know the app behaviour & detect anomary.
Prometheus + Grafana + deadtrickster/beam-dashboards Beautiful. But to keep Prometheus
servers have many toils.
Mackerel https://mackerel.io
Add mackerel-container-agent as a sidecar. --- apiVersion: apps/v1 kind: Deployment
spec: template: spec: containers: - name: app … - name: mackerel-container-agent image: mackerel/mackerel-container-agent:latest imagePullPolicy: Always resources: limits: {memory: 128Mi} env: - {name: MACKEREL_CONTAINER_PLATFORM, value: kubernetes} - name: MACKEREL_KUBERNETES_KUBELET_HOST valueFrom: fieldRef: fieldPath: status.hostIP - name: MACKEREL_KUBERNETES_NAMESPACE valueFrom: fieldRef: {fieldPath: metadata.namespace} - name: MACKEREL_KUBERNETES_POD_NAME valueFrom: fieldRef: {fieldPath: metadata.name} - {name: MACKEREL_ROLES, value: example:app} envFrom: - secretRef: {name: mackerel}
apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization … secretGenerator: - name: mackerel type:
Opaque files: [MACKEREL_APIKEY=secret/MACKEREL_APIKEY]
None
OK. I see container's metrics. How about BEAM metrics?
KomachiHeartbeat + mackerel- plugin-json https://hex.pm/packages/komachi_heartbeat Vital monitoring Elixir Web application.
https://github.com/mackerelio/mackerel-plugin-json Json custom metrics plugin for mackerel.io agent.
KomachiHeartbeat has /stats endpoint that returns app's stats in JSON.
mackerel-plugin-json posts any data gotten from JSON API as Mackerel custome metrics.
We have KomachiHeartbeat.BeamVital . https://github.com/ne- sachirou/ex_komachi_heartbeat/pull/29 (Sorry for under development.)
Add KomachiHeartbeat.BeamVital . defmodule TeijiBot.Router do use Plug.Router plug(:match) plug(:dispatch)
forward( "/ops", to: KomachiHeartbeat, init_opts: [vitals: [KomachiHeartbeat.BeamVital]] ) end
Configure mackerel-plugin-json. --- apiVersion: apps/v1 kind: Deployment spec: template: spec:
volumes: - name: mackerel-agent-config configMap: name: mackerel-agent-config containers: - name: app … - name: mackerel-container-agent image: mackerel/mackerel-container-agent:plugins … env: - {name: MACKEREL_AGENT_CONFIG, value: /etc/mackerel/mackerel.yaml} … volumeMounts: - name: mackerel-agent-config mountPath: /etc/mackerel readOnly: true
--- apiVersion: v1 kind: ConfigMap metadata: name: mackerel-agent-config data: mackerel.yaml:
| plugin: metrics: json: command: | mackerel-plugin-json \ -url="http://localhost:4000/ops/stats" \ -prefix='beam'
None
Current metrics : atom_count memory_ets_tables memory_system_bytes_total atom, binary, code, ets,
other port_count process_count (More, under development.)