Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Mackerelコンテナエージェントによる コンテナ監視について / Mackerel Meetup #13 Tokyo
Search
Hayato Imai
March 01, 2019
Technology
1
8.1k
Mackerelコンテナエージェントによる コンテナ監視について / Mackerel Meetup #13 Tokyo
2019/03/01
https://mackerelio.connpass.com/event/118995/
Hayato Imai
March 01, 2019
Tweet
Share
More Decks by Hayato Imai
See All by Hayato Imai
Kubernetes撤退、 その後のはてなの取り組み / kubernetes meetup tokyo number 52
hayajo
9
7k
Mackerelにおける Cloud Nativeへの取り組みと チームへ与えた変化 / CloudNative Days Tokyo 2020
hayajo
2
1.3k
MackerelにおけるKubernetes利用の取組みとこれから / Kubernetes Meetup Tokyo #22
hayajo
20
9.2k
Mackerelチームのコンテナ開発における戦略とこれから / 190722-cndt2019
hayajo
1
1.7k
AWSコンテナサービス入門 / nds60-jaws-ug
hayajo
0
2.7k
コンテナのメトリクスと モニタリングパターン / 190320-sakura-event
hayajo
6
1.5k
Docker for Mac/Windows ではじめる Kubernetes / NDS55 Docker with Kubernetes
hayajo
16
14k
Terrafromで構築するマルチクラウドプラットフォームインフラストラクチャ / NDS53 Terraform
hayajo
0
340
Ncatをつかおう / Use Ncat
hayajo
1
3.5k
Other Decks in Technology
See All in Technology
Rustで「プリズモイダル法」を利用して「土量計算」をガチでやる
nokonoko1203
1
300
コードファーストの考え方。 Amplify Gen2から学ぶAWS次世代のWeb開発体験
yoshiitaka
2
340
Python と Snowflake はズッ友だょ!~ Snowflake の Python 関連機能をふりかえる ~
__allllllllez__
2
140
Tellus の衛星データを見てみよう #mf_fukuoka
kongmingstrap
0
270
Cloud Service Mesh に触れ合う
phaya72
1
180
今年のRubyKaigiはProfiler Year🤘
osyoyu
0
360
成長をサポートするピープルマネジメントのやり方
sioncojp
8
1.1k
推しは推せるときに推せ! プロダクトにフィードバックしていこう
nakasho
0
460
.NET Profiler in 2024.
kkamegawa
2
1.2k
Gradle Build Scanを使ってビルドのことを知ろう potatotips #87
tomorrowkey
2
150
AOAI をきっかけに 社内の Azure 管理を見直した話
recruitengineers
PRO
1
450
require(ESM)とECMAScript仕様
uhyo
4
960
Featured
See All Featured
ParisWeb 2013: Learning to Love: Crash Course in Emotional UX Design
dotmariusz
104
6.6k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
352
28k
Teambox: Starting and Learning
jrom
128
8.4k
Bash Introduction
62gerente
605
210k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
21
1.4k
Making the Leap to Tech Lead
cromwellryan
125
8.5k
Being A Developer After 40
akosma
66
580k
Building a Modern Day E-commerce SEO Strategy
aleyda
21
6.4k
YesSQL, Process and Tooling at Scale
rocio
165
13k
Why You Should Never Use an ORM
jnunemaker
PRO
51
8.7k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
33
6k
Designing Experiences People Love
moore
136
23k
Transcript
MackerelίϯςφΤʔδΣϯτʹΑΔ ίϯςφࢹʹ͍ͭͯ .BDLFSFM.FFUVQ JEIBZBKP@
ࣗݾհ • ࠓҪ൏ਓ(id:hayajo_77) • MackerelνʔϜ SRE • 201711݄ೖࣾ • ίϯςφཁૉٕज़ɺपลٕज़
ίϯςφٕज़ೖԾԽͱͷҧ͍ΛΓɺཁૉٕज़Λ৮ֶͬͯ΅͏ IUUQTFNQMPZNFOUFOKBQBODPNFOHJOFFSIVCFOUSZ
ΞδΣϯμ • MackerelίϯςφΤʔδΣϯτͷհ • ϝτϦοΫऔಘઓུղઆ • ࠓޙͷίϯςφΤʔδΣϯτ • ·ͱΊ
MackerelίϯςφΤʔδΣϯτͷհ
MackerelίϯςφΤʔδΣϯτ ʢύϒϦοΫϕʔλʣΛެ։͠·ͨ͠ https://mackerel.io/ja/blog/entry/weekly/20190218
ύϒϦοΫϕʔλͱ • কདྷతʹਖ਼ࣜͳαʔϏεϦϦʔεΛߦ͏༧ఆ͕͋ΔػೳΛઌߦͯ͠ ެ։ • ׆ൃʹมߋΛߦ͍ɺ৽͍͠όʔδϣϯΛఏڙ • ࣄલࠂͷ্Ͱඇޓमਖ਼Λ࣮ࢪ
OSSͱͯ͠ެ։ • DockerHub • mackerel/mackerel-container-agent • GitHub • mackerelio/mackerel-container-agent •
IssuePRݪଇӳޠͰ͓ئ͍͠·͢
ίϯςφઐ༻ͷܰྔΤʔδΣϯτ • ίϯςφઐ༻ͱͯ͠࠶ઃܭ • Amazon ECS, AWS Fargate, KubernetesʹରԠ •
1λεΫ/Podʹ͖ͭ1ϗετͱͯ͠Χϯτ • ʰγεςϜΛߏ͢Δ࠷খ୯Ґ(ͩͱγεςϜΛཧ͢Δ্ͰΈͳ ͖͢ͷ)ʱ • see. ʮFAQɾϗετͷܭࢉํ๏ʹ͍ͭͯ - Mackerel ϔϧϓʯ
αΠυΧʔίϯςφͱͯ͠σϓϩΠ • ϓϥοτϑΥʔϜʹΑͬͯઃఆ͕ҟͳΔ • ECS(EC2/Bridge, EC2/Host) • ECS(EC2/awsvpc, Fargate) •
Kubernetes • see. ʮίϯςφΛࢹ͢Δ - Mackerel ϔϧϓʯ
Web UI/ϗετҰཡ
Web UI/ϗετৄࡉ
Web UI/ϩʔϧάϥϑ
Δ͜ͱ • λεΫ/Podͷͯ͢ͷίϯςφͷϝτϦοΫΛγεςϜϝτϦοΫ ͱͯ͠ߘ • CPUɺϝϞϦɺωοτϫʔΫ • αʔϏε/ϩʔϧͷׂΓͯ • ϓϥάΠϯͷར༻
• ϓϥάΠϯΛΠϯετʔϧͨ͠DockerΠϝʔδͷ४උ͕ඞཁ
Βͳ͍͜ͱ • ΫϥελϊʔυͷϝτϦοΫऩू͠ͳ͍ • ϊʔυɺλεΫ/Pod, etc. • ϊʔυͷCPUɺϝϞϦɺωοτϫʔΫɺσΟεΫͳͲ • mackerel-agentΛར༻
• αϙʔτ͢ΔϓϥοτϑΥʔϜҎ֎ͷίϯςφͷࢹ • DockerʮDockerΛϞχλϦϯά͢Δ - Mackerel ϔϧϓʯ
αΠυΧʔύλʔϯͷ࠾༻ • FargateͷΑ͏ͳɺΠϯϑϥετϥΫνϟͷཧ͕ෆཁͳίϯςφ ར༻͕ओྲྀʹͳΔͱߟ͍͑ͯΔ • Mackerelʹ͓͚ΔϗετͷఆٛͱλεΫ/PodͷϥΠϑαΠΫϧ • ίϯςφؒͷϦιʔεڞ༗ • ࢹରͷΞϓϦέʔγϣϯ༷ʹ͍ۙࢹઃఆ͕Մೳ
ϝτϦοΫऔಘઓུղઆ
ϝτϦοΫऔಘΞʔΩςΫνϟ
ECS/Fargate
ECS/FargateͷAPI &$#SJEHF &$)PTU &$BXTWQD 'BSHBUF *OUSPTQFDUJPO "1* ˔ ˔ ☓
☓ 5BTL.FUBEBUB &OEQPJOUW ☓ ☓ ˔ ˔ 5BTL.FUBEBUB &OEQPJOUW ˔ ˔ ˔ ☓
Introspection API • λεΫͷϝλσʔλΛฦ͢ • λεΫͷARNεςʔλεɺ֤ίϯςφͷDockerIDͳͲ • ίϯςφͷϝτϦοΫCPUɺϝϞϦLIMITऔಘͰ͖ͳ͍ • Docker
stats API(docker.sock)cgroupfsͰΧόʔ • EC2/Bridge, EC2/HostͰར༻ • Task Metadata Endpoint v3ʹରԠ༧ఆ
Task Metadata Endpoint v2/v3 • λεΫϝλσʔλɺίϯςφϝτϦοΫΛฦ͢ • ϝτϦοΫDocker stats APIͦͷͷΛฦ͢
• v2/v3ͰऔಘͰ͖Δσʔλʹେ͖ͳҧ͍ͳ͍ • EC2/awsvpc, FargateͰv2Λར༻ • EC2/awsvpcv3ʹҠߦ༧ఆ
ECS/FargateͷωοτϫʔΫϝτϦοΫ • ωοτϫʔΫϝτϦοΫ͕औಘͰ͖ΔͷEC2/Bridge͚ͩ...... • Docker stats API(libnetwork)bridgeϞʔυͷͱ͖͔͠ϝτϦοΫ ͕औΕͳ͍ͬΆ͍ • ࣮ʹৄ͍͠ํɺͥͻ࠙ձͰ͓͠·͠ΐ͏ʂ
root@ebb5c8c90634:/# curl -s $ {ECS_CONTAINER_METADATA_URI}/stats | jq .networks { "eth0":
{ "rx_bytes": 17331985, "tx_packets": 932, "rx_packets": 1353, "tx_bytes": 77755 } } e.g. EC2/Bridge w/ TMEv3
root@ip-10-0-10-144:/# curl -s ${ECS_CONTAINER_METADATA_URI}/stats | \ > jq .networks null
e.g. EC2/Host, EC2/awsvpc w/ TMEv3 root@9ea93ec5d92b:/# curl -s ${ECS_CONTAINER_METADATA_URI}/stats | \ > jq .networks null
-bash-4.2# CID=$(basename $(head -n1 /proc/self/ cgroup | cut -d: -f3))
-bash-4.2# curl -s 169.254.170.2/v2/stats/${CID} | jq .networks null e.g. Fargate w/ TMEv2
EC2/BridgeͷωοτϫʔΫελοΫ • ωοτϫʔΫελοΫλεΫͷίϯςφ͝ͱʹҟͳΔ
EC2/HostωοτϫʔΫελοΫ • ϗετͷωοτϫʔΫελοΫΛλεΫͷίϯςφؒͰڞ༗
EC2/awsvpc, FargateωοτϫʔΫελοΫ • ωοτϫʔΫελοΫΛλεΫͷίϯςφؒͰڞ༗
ίϯςφΤʔδΣϯτʹ͓͚ΔECS/Fargateͷ ωοτϫʔΫϝτϦοΫͷऔಘ • EC2/BridgeϞʔυͷ߹ɺDocker stats API͔Β֤ίϯςφͷωο τϫʔΫϝτϦοΫΛऔಘͯ͠λεΫͷϝτϦοΫͱͯ͠ߘ • EC2/Host, EC2/awsvpc,
FargateͰɺίϯςφಉ࢜ωοτϫʔ ΫελοΫΛڞ༗͢ΔͷͰɺίϯςφΤʔδΣϯτࣗͷωοτϫʔ ΫϝτϦοΫΛऔಘͯ͠λεΫͷϝτϦοΫͱͯ͠ߘ
• ಉ͡ίϯςφఆٛͰɺىಈλΠϓɺωοτϫʔΫϞʔυͰΠϯλʔ ϑΣʔεͷݟ͑ํ͕ҟͳΔ λεΫͷInterfaceάϥϑ &$#SJEHF &$)PTU &$BXTWQD 'BSHBUF
• ىಈλΠϓɺωοτϫʔΫϞʔυʹΑͬͯར༻Ͱ͖ΔAPI͕ҟͳΔ • EC2ىಈλΠϓTask Metadata Endpoint v3ʹରԠத • ωοτϫʔΫϞʔυʹΑͬͯωοτϫʔΫϝτϦοΫͷऔಘํ๏͕ ҟͳΔ
ECS/Fargateʹ͓͚ΔϝτϦοΫऔಘ·ͱΊ
Kubernetes
KubernetesͷCore metrics pipeline
kubelet API • PodͷϝλσʔλϝτϦοΫɺϩάΛऔಘɺίϚϯυͷϩʔΧϧ ࣮ߦͷͨΊͷAPI • kubelet port(10250/HTTPS)ͱread-only port(10255/HTTP)Ͱ LISTEN
• read-only port͕ແޮͳڥ͋Δ
kubelet APIͷAuthN/AuthZ • kubelet portͰೝূ/ೝՄΛઃఆͰ͖Δ • Authentication • ಗ໊ΞΫηεɺΫϥΠΞϯτূ໌ॻೝূɺτʔΫϯೝূ •
Authorization • AlwaysAllow, Webhook • SubjectAccessReview APIʹΑΔݖݶνΣοΫ • see. ʮKubelet authentication/authorization - Kubernetesʯ
ίϯςφΤʔδΣϯτͱkubelet API • σϑΥϧτͰread-only portΛར༻ • ઃఆͰkubelet portʹΓସ͑Մೳ • τʔΫϯೝূΛαϙʔτ
• "nodes/proxy", "nodes/stats", "nodes/spec"ʹgetΞΫηε • ͏·͍͔͘ͳ͍߹automountServiceAccountTokenઃఆνΣοΫ • see. ʮKubernetesʹmackerel-container-agentΛηοτΞοϓ͢Δ - Mackerel ϔϧϓʯ
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: mackerel-container-agent rules: - apiGroups:
[""] resources: ["nodes/proxy", "nodes/stats", "nodes/spec"] verbs: ["get"] e.g. RBACઃఆ(ClusterRole)
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: mackerel-container-agent-binding roleRef: apiGroup: rbac.authorization.k8s.io
kind: ClusterRole name: mackerel-container-agent subjects: - kind: ServiceAccount name: my-service-account namespace: default e.g. RBACઃఆ(ClusterRoleBinding)
• kubelet API͔ΒϝτϦοΫΛऔಘ͢Δ • σϑΥϧτread-only PortΛར༻ • kubelet portͰτʔΫϯೝূʹରԠ •
ඞཁʹԠͯ͡RBACΛઃఆ Kubernetesʹ͓͚ΔϝτϦοΫऔಘ·ͱΊ
ࠓޙͷίϯςφΤʔδΣϯτ
ۙରԠ༧ఆ • Task Metadata Endpoint v3ରԠ • EC2/Bridge, EC2/Host, EC2/awspvc
• docker.sockcgroupfsͷґଘΛͳ͘͢ • rootϢʔβඞཁͳ͠ • cgroupfsͷϚϯτϙΠϯτͷҧ͍Λؾʹ͠ͳͯ͘Α͍
లͱߏ • ϓϥάΠϯར༻ͷརศੑ্ • ϓϥάΠϯಉࠝΠϝʔδͷఏڙͳͲ • ΧελϜϝτϦοΫͷѻ͍ • ϗετಉ༷ʹୀޙɺҰఆظؒܦաͰඇදࣔͱͳΔ •
PrometheusServiceMeshͱͷ࿈ܞ
·ͱΊ
·ͱΊ • ίϯςφΤʔδΣϯτʢύϒϦοΫϕʔλʣΛϦϦʔε • λεΫ/PodΛࢹ͢ΔαΠυΧʔίϯςφ • ϓϥοτϑΥʔϜ͕ఏڙ͢ΔAPI͔ΒϝτϦοΫΛऩू • see. ʮίϯςφΛࢹ͢Δ
- Mackerel ϔϧϓʯ
ϑΟʔυόοΫΛ͓͍ͪͯ͠·͢
None