Mackerelコンテナエージェントによるコンテナ監視について / Mackerel Meetup #13 Tokyo

by Hayato Imai

Slide 1

Slide 1 text

MackerelίϯςφΤʔδΣϯτʹΑΔ ίϯςφ؂ࢹʹ͍ͭͯ .BDLFSFM.FFUVQ JEIBZBKP@

Slide 2

Slide 2 text

ࣗݾ঺հ • ࠓҪ൏ਓ(id:hayajo_77) • MackerelνʔϜ SRE • 2017೥11݄ೖࣾ • ίϯςφཁૉٕज़ɺपลٕज़ ίϯςφٕज़ೖ໳Ծ૝Խͱͷҧ͍Λ஌Γɺཁૉٕज़Λ৮ֶͬͯ΅͏ IUUQTFNQMPZNFOUFOKBQBODPNFOHJOFFSIVCFOUSZ

Slide 3

Slide 3 text

ΞδΣϯμ • MackerelίϯςφΤʔδΣϯτͷ঺հ • ϝτϦοΫऔಘઓུղઆ • ࠓޙͷίϯςφΤʔδΣϯτ • ·ͱΊ

Slide 4

Slide 4 text

MackerelίϯςφΤʔδΣϯτͷ঺հ

Slide 5

Slide 5 text

MackerelίϯςφΤʔδΣϯτ ʢύϒϦοΫϕʔλʣΛެ։͠·ͨ͠ https://mackerel.io/ja/blog/entry/weekly/20190218

Slide 6

Slide 6 text

ύϒϦοΫϕʔλͱ͸ • কདྷతʹਖ਼ࣜͳαʔϏεϦϦʔεΛߦ͏༧ఆ͕͋ΔػೳΛઌߦͯ͠ ެ։ • ׆ൃʹมߋΛߦ͍ɺ৽͍͠όʔδϣϯΛఏڙ • ࣄલࠂ஌ͷ্Ͱඇޓ׵मਖ਼Λ࣮ࢪ

Slide 7

Slide 7 text

OSSͱͯ͠ެ։ • DockerHub • mackerel/mackerel-container-agent • GitHub • mackerelio/mackerel-container-agent • Issue΍PR͸ݪଇӳޠͰ͓ئ͍͠·͢

Slide 8

Slide 8 text

ίϯςφઐ༻ͷܰྔΤʔδΣϯτ • ίϯςφઐ༻ͱͯ͠࠶ઃܭ • Amazon ECS, AWS Fargate, KubernetesʹରԠ • 1λεΫ/Podʹ͖ͭ1ϗετͱͯ͠Χ΢ϯτ • ʰγεςϜΛߏ੒͢Δ࠷খ୯Ґ(ͩͱγεςϜΛ؅ཧ͢Δ্ͰΈͳ ͢΂͖΋ͷ)ʱ • see. ʮFAQɾϗετ਺ͷܭࢉํ๏ʹ͍ͭͯ - Mackerel ϔϧϓʯ

Slide 9

Slide 9 text

αΠυΧʔίϯςφͱͯ͠σϓϩΠ • ϓϥοτϑΥʔϜʹΑͬͯઃఆ͕ҟͳΔ • ECS(EC2/Bridge, EC2/Host) • ECS(EC2/awsvpc, Fargate) • Kubernetes • see. ʮίϯςφΛ؂ࢹ͢Δ - Mackerel ϔϧϓʯ

Slide 10

Slide 10 text

Web UI/ϗετҰཡ

Slide 11

Slide 11 text

Web UI/ϗετৄࡉ

Slide 12

Slide 12 text

Web UI/ϩʔϧάϥϑ

Slide 13

Slide 13 text

΍Δ͜ͱ • λεΫ/Podͷ͢΂ͯͷίϯςφͷϝτϦοΫΛγεςϜϝτϦοΫ ͱͯ͠౤ߘ • CPUɺϝϞϦɺωοτϫʔΫ • αʔϏε/ϩʔϧͷׂΓ౰ͯ • ϓϥάΠϯͷར༻ • ϓϥάΠϯΛΠϯετʔϧͨ͠DockerΠϝʔδͷ४උ͕ඞཁ

Slide 14

Slide 14 text

΍Βͳ͍͜ͱ • Ϋϥελ΍ϊʔυͷϝτϦοΫ͸ऩू͠ͳ͍ • ϊʔυ਺ɺλεΫ/Pod਺, etc. • ϊʔυͷCPUɺϝϞϦɺωοτϫʔΫɺσΟεΫͳͲ • mackerel-agentΛར༻ • αϙʔτ͢ΔϓϥοτϑΥʔϜҎ֎ͷίϯςφͷ؂ࢹ • Docker͸ʮDockerΛϞχλϦϯά͢Δ - Mackerel ϔϧϓʯ

Slide 15

Slide 15 text

αΠυΧʔύλʔϯͷ࠾༻ • FargateͷΑ͏ͳɺΠϯϑϥετϥΫνϟͷ؅ཧ͕ෆཁͳίϯςφ ར༻͕ओྲྀʹͳΔͱߟ͍͑ͯΔ • Mackerelʹ͓͚ΔϗετͷఆٛͱλεΫ/PodͷϥΠϑαΠΫϧ • ίϯςφؒͷϦιʔεڞ༗ • ؂ࢹର৅ͷΞϓϦέʔγϣϯ࢓༷ʹ͍ۙ؂ࢹઃఆ͕Մೳ

Slide 16

Slide 16 text

ϝτϦοΫऔಘઓུղઆ

Slide 17

Slide 17 text

ϝτϦοΫऔಘΞʔΩςΫνϟ

Slide 18

Slide 18 text

ECS/Fargate

Slide 19

Slide 19 text

ECS/FargateͷAPI &$#SJEHF &$)PTU &$BXTWQD 'BSHBUF *OUSPTQFDUJPO "1* ˔ ˔ ☓ ☓ 5BTL.FUBEBUB &OEQPJOUW ☓ ☓ ˔ ˔ 5BTL.FUBEBUB &OEQPJOUW ˔ ˔ ˔ ☓

Slide 20

Slide 20 text

Introspection API • λεΫͷϝλσʔλΛฦ͢ • λεΫͷARN΍εςʔλεɺ֤ίϯςφͷDockerIDͳͲ • ίϯςφͷϝτϦοΫ΍CPUɺϝϞϦLIMIT͸औಘͰ͖ͳ͍ • Docker stats API(docker.sock)΍cgroupfsͰΧόʔ • EC2/Bridge, EC2/HostͰར༻ • Task Metadata Endpoint v3ʹରԠ༧ఆ

Slide 21

Slide 21 text

Task Metadata Endpoint v2/v3 • λεΫϝλσʔλɺίϯςφϝτϦοΫΛฦ͢ • ϝτϦοΫ͸Docker stats APIͦͷ΋ͷΛฦ͢ • v2/v3ͰऔಘͰ͖Δσʔλʹେ͖ͳҧ͍͸ͳ͍ • EC2/awsvpc, FargateͰv2Λར༻ • EC2/awsvpc͸v3ʹҠߦ༧ఆ

Slide 22

Slide 22 text

ECS/FargateͷωοτϫʔΫϝτϦοΫ • ωοτϫʔΫϝτϦοΫ͕औಘͰ͖Δͷ͸EC2/Bridge͚ͩ...... • Docker stats API(libnetwork)͸bridgeϞʔυͷͱ͖͔͠ϝτϦοΫ ͕औΕͳ͍ͬΆ͍ • ࣮૷ʹৄ͍͠ํɺͥͻ࠙਌ձͰ͓࿩͠·͠ΐ͏ʂ

Slide 23

Slide 23 text

root@ebb5c8c90634:/# curl -s $ {ECS_CONTAINER_METADATA_URI}/stats | jq .networks { "eth0": { "rx_bytes": 17331985, "tx_packets": 932, "rx_packets": 1353, "tx_bytes": 77755 } } e.g. EC2/Bridge w/ TMEv3

Slide 24

Slide 24 text

root@ip-10-0-10-144:/# curl -s ${ECS_CONTAINER_METADATA_URI}/stats | \ > jq .networks null e.g. EC2/Host, EC2/awsvpc w/ TMEv3 root@9ea93ec5d92b:/# curl -s ${ECS_CONTAINER_METADATA_URI}/stats | \ > jq .networks null

Slide 25

Slide 25 text

-bash-4.2# CID=$(basename $(head -n1 /proc/self/ cgroup | cut -d: -f3)) -bash-4.2# curl -s 169.254.170.2/v2/stats/${CID} | jq .networks null e.g. Fargate w/ TMEv2

Slide 26

Slide 26 text

EC2/BridgeͷωοτϫʔΫελοΫ • ωοτϫʔΫελοΫ͸λεΫͷίϯςφ͝ͱʹҟͳΔ

Slide 27

Slide 27 text

EC2/HostωοτϫʔΫελοΫ • ϗετͷωοτϫʔΫελοΫΛλεΫͷίϯςφؒͰڞ༗

Slide 28

Slide 28 text

EC2/awsvpc, FargateωοτϫʔΫελοΫ • ωοτϫʔΫελοΫΛλεΫͷίϯςφؒͰڞ༗

Slide 29

Slide 29 text

ίϯςφΤʔδΣϯτʹ͓͚ΔECS/Fargateͷ ωοτϫʔΫϝτϦοΫͷऔಘ • EC2/BridgeϞʔυͷ৔߹ɺDocker stats API͔Β֤ίϯςφͷωο τϫʔΫϝτϦοΫΛऔಘͯ͠λεΫͷϝτϦοΫͱͯ͠౤ߘ • EC2/Host, EC2/awsvpc, FargateͰ͸ɺίϯςφಉ࢜͸ωοτϫʔ ΫελοΫΛڞ༗͢ΔͷͰɺίϯςφΤʔδΣϯτࣗ਎ͷωοτϫʔ ΫϝτϦοΫΛऔಘͯ͠λεΫͷϝτϦοΫͱͯ͠౤ߘ

Slide 30

Slide 30 text

• ಉ͡ίϯςφఆٛͰ΋ɺىಈλΠϓɺωοτϫʔΫϞʔυͰΠϯλʔ ϑΣʔεͷݟ͑ํ͕ҟͳΔ λεΫͷInterfaceάϥϑ &$#SJEHF &$)PTU &$BXTWQD 'BSHBUF

Slide 31

Slide 31 text

• ىಈλΠϓɺωοτϫʔΫϞʔυʹΑͬͯར༻Ͱ͖ΔAPI͕ҟͳΔ • EC2ىಈλΠϓ͸Task Metadata Endpoint v3ʹରԠத • ωοτϫʔΫϞʔυʹΑͬͯωοτϫʔΫϝτϦοΫͷऔಘํ๏͕ ҟͳΔ ECS/Fargateʹ͓͚ΔϝτϦοΫऔಘ·ͱΊ

Slide 32

Slide 32 text

Kubernetes

Slide 33

Slide 33 text

KubernetesͷCore metrics pipeline

Slide 34

Slide 34 text

kubelet API • Podͷϝλσʔλ΍ϝτϦοΫɺϩάΛऔಘɺίϚϯυͷϩʔΧϧ ࣮ߦͷͨΊͷAPI • kubelet port(10250/HTTPS)ͱread-only port(10255/HTTP)Ͱ LISTEN • read-only port͕ແޮͳ؀ڥ΋͋Δ

Slide 35

Slide 35 text

kubelet APIͷAuthN/AuthZ • kubelet portͰ͸ೝূ/ೝՄΛઃఆͰ͖Δ • Authentication • ಗ໊ΞΫηεɺΫϥΠΞϯτূ໌ॻೝূɺτʔΫϯೝূ • Authorization • AlwaysAllow, Webhook • SubjectAccessReview APIʹΑΔݖݶνΣοΫ • see. ʮKubelet authentication/authorization - Kubernetesʯ

Slide 36

Slide 36 text

ίϯςφΤʔδΣϯτͱkubelet API • σϑΥϧτͰ͸read-only portΛར༻ • ઃఆͰkubelet portʹ੾Γସ͑Մೳ • τʔΫϯೝূΛαϙʔτ • "nodes/proxy", "nodes/stats", "nodes/spec"ʹgetΞΫηε • ͏·͍͔͘ͳ͍৔߹͸automountServiceAccountTokenઃఆ΋νΣοΫ • see. ʮKubernetesʹmackerel-container-agentΛηοτΞοϓ͢Δ - Mackerel ϔϧϓʯ

Slide 37

Slide 37 text

apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: mackerel-container-agent rules: - apiGroups: [""] resources: ["nodes/proxy", "nodes/stats", "nodes/spec"] verbs: ["get"] e.g. RBACઃఆ(ClusterRole)

Slide 38

Slide 38 text

apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: mackerel-container-agent-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: mackerel-container-agent subjects: - kind: ServiceAccount name: my-service-account namespace: default e.g. RBACઃఆ(ClusterRoleBinding)

Slide 39

Slide 39 text

• kubelet API͔ΒϝτϦοΫΛऔಘ͢Δ • σϑΥϧτ͸read-only PortΛར༻ • kubelet portͰ͸τʔΫϯೝূʹରԠ • ඞཁʹԠͯ͡RBACΛઃఆ Kubernetesʹ͓͚ΔϝτϦοΫऔಘ·ͱΊ

Slide 40

Slide 40 text

ࠓޙͷίϯςφΤʔδΣϯτ

Slide 41

Slide 41 text

ۙ೔ରԠ༧ఆ • Task Metadata Endpoint v3ରԠ • EC2/Bridge, EC2/Host, EC2/awspvc • docker.sock΍cgroupfs΁ͷґଘΛͳ͘͢ • rootϢʔβඞཁͳ͠ • cgroupfsͷϚ΢ϯτϙΠϯτͷҧ͍Λؾʹ͠ͳͯ͘Α͍

Slide 42

Slide 42 text

ల๬ͱߏ૝ • ϓϥάΠϯར༻ͷརศੑ޲্ • ϓϥάΠϯಉࠝΠϝʔδͷఏڙͳͲ • ΧελϜϝτϦοΫͷѻ͍ • ϗετಉ༷ʹୀ໾ޙɺҰఆظؒܦաͰඇදࣔͱͳΔ • Prometheus΍ServiceMeshͱͷ࿈ܞ

Slide 43

Slide 43 text

·ͱΊ

Slide 44

Slide 44 text

·ͱΊ • ίϯςφΤʔδΣϯτʢύϒϦοΫϕʔλʣΛϦϦʔε • λεΫ/PodΛ؂ࢹ͢ΔαΠυΧʔίϯςφ • ϓϥοτϑΥʔϜ͕ఏڙ͢ΔAPI͔ΒϝτϦοΫΛऩू • see. ʮίϯςφΛ؂ࢹ͢Δ - Mackerel ϔϧϓʯ