Mackerelコンテナエージェントによる コンテナ監視について / Mackerel Meetup #13 Tokyo

Transcript

1. MackerelίϯςφΤʔδΣϯτʹΑΔ ίϯςφ؂ࢹʹ͍ͭͯ .BDLFSFM
   .FFUVQ
   
   JEIBZBKP@

2. ࣗݾ঺հ • ࠓҪ൏ਓ(id:hayajo_77) • MackerelνʔϜ SRE • 2017೥11݄ೖࣾ • ίϯςφཁૉٕज़ɺपลٕज़

   ίϯςφٕज़ೖ໳
   
   Ծ૝Խͱͷҧ͍Λ஌Γɺཁૉٕज़Λ৮ֶͬͯ΅͏
   IUUQTFNQMPZNFOUFOKBQBODPNFOHJOFFSIVCFOUSZ 

3. ΞδΣϯμ • MackerelίϯςφΤʔδΣϯτͷ঺հ • ϝτϦοΫऔಘઓུղઆ • ࠓޙͷίϯςφΤʔδΣϯτ • ·ͱΊ

4. MackerelίϯςφΤʔδΣϯτͷ঺հ

5. MackerelίϯςφΤʔδΣϯτ ʢύϒϦοΫϕʔλʣΛެ։͠·ͨ͠ https://mackerel.io/ja/blog/entry/weekly/20190218

6. ύϒϦοΫϕʔλͱ͸ • কདྷతʹਖ਼ࣜͳαʔϏεϦϦʔεΛߦ͏༧ఆ͕͋ΔػೳΛઌߦͯ͠ ެ։ • ׆ൃʹมߋΛߦ͍ɺ৽͍͠όʔδϣϯΛఏڙ • ࣄલࠂ஌ͷ্Ͱඇޓ׵मਖ਼Λ࣮ࢪ

7. OSSͱͯ͠ެ։ • DockerHub • mackerel/mackerel-container-agent • GitHub • mackerelio/mackerel-container-agent •

   Issue΍PR͸ݪଇӳޠͰ͓ئ͍͠·͢

8. ίϯςφઐ༻ͷܰྔΤʔδΣϯτ • ίϯςφઐ༻ͱͯ͠࠶ઃܭ • Amazon ECS, AWS Fargate, KubernetesʹରԠ •

   1λεΫ/Podʹ͖ͭ1ϗετͱͯ͠Χ΢ϯτ • ʰγεςϜΛߏ੒͢Δ࠷খ୯Ґ(ͩͱγεςϜΛ؅ཧ͢Δ্ͰΈͳ ͢΂͖΋ͷ)ʱ • see. ʮFAQɾϗετ਺ͷܭࢉํ๏ʹ͍ͭͯ - Mackerel ϔϧϓʯ

9. αΠυΧʔίϯςφͱͯ͠σϓϩΠ • ϓϥοτϑΥʔϜʹΑͬͯઃఆ͕ҟͳΔ • ECS(EC2/Bridge, EC2/Host) • ECS(EC2/awsvpc, Fargate) •

   Kubernetes • see. ʮίϯςφΛ؂ࢹ͢Δ - Mackerel ϔϧϓʯ

10. Web UI/ϗετҰཡ

11. Web UI/ϗετৄࡉ

12. Web UI/ϩʔϧάϥϑ

13. ΍Δ͜ͱ • λεΫ/Podͷ͢΂ͯͷίϯςφͷϝτϦοΫΛγεςϜϝτϦοΫ ͱͯ͠౤ߘ • CPUɺϝϞϦɺωοτϫʔΫ • αʔϏε/ϩʔϧͷׂΓ౰ͯ • ϓϥάΠϯͷར༻

    • ϓϥάΠϯΛΠϯετʔϧͨ͠DockerΠϝʔδͷ४උ͕ඞཁ

14. ΍Βͳ͍͜ͱ • Ϋϥελ΍ϊʔυͷϝτϦοΫ͸ऩू͠ͳ͍ • ϊʔυ਺ɺλεΫ/Pod਺, etc. • ϊʔυͷCPUɺϝϞϦɺωοτϫʔΫɺσΟεΫͳͲ • mackerel-agentΛར༻

    • αϙʔτ͢ΔϓϥοτϑΥʔϜҎ֎ͷίϯςφͷ؂ࢹ • Docker͸ʮDockerΛϞχλϦϯά͢Δ - Mackerel ϔϧϓʯ

15. αΠυΧʔύλʔϯͷ࠾༻ • FargateͷΑ͏ͳɺΠϯϑϥετϥΫνϟͷ؅ཧ͕ෆཁͳίϯςφ ར༻͕ओྲྀʹͳΔͱߟ͍͑ͯΔ • Mackerelʹ͓͚ΔϗετͷఆٛͱλεΫ/PodͷϥΠϑαΠΫϧ • ίϯςφؒͷϦιʔεڞ༗ • ؂ࢹର৅ͷΞϓϦέʔγϣϯ࢓༷ʹ͍ۙ؂ࢹઃఆ͕Մೳ

16. ϝτϦοΫऔಘઓུղઆ

17. ϝτϦοΫऔಘΞʔΩςΫνϟ

18. ECS/Fargate

19. ECS/FargateͷAPI &$#SJEHF &$)PTU &$BXTWQD 'BSHBUF *OUSPTQFDUJPO
    "1* ˔ ˔ ☓

    ☓ 5BTL
    .FUBEBUB
    &OEQPJOU
    W ☓ ☓ ˔ ˔ 5BTL
    .FUBEBUB
    &OEQPJOU
    W ˔ ˔ ˔ ☓

20. Introspection API • λεΫͷϝλσʔλΛฦ͢ • λεΫͷARN΍εςʔλεɺ֤ίϯςφͷDockerIDͳͲ • ίϯςφͷϝτϦοΫ΍CPUɺϝϞϦLIMIT͸औಘͰ͖ͳ͍ • Docker

    stats API(docker.sock)΍cgroupfsͰΧόʔ • EC2/Bridge, EC2/HostͰར༻ • Task Metadata Endpoint v3ʹରԠ༧ఆ

21. Task Metadata Endpoint v2/v3 • λεΫϝλσʔλɺίϯςφϝτϦοΫΛฦ͢ • ϝτϦοΫ͸Docker stats APIͦͷ΋ͷΛฦ͢

    • v2/v3ͰऔಘͰ͖Δσʔλʹେ͖ͳҧ͍͸ͳ͍ • EC2/awsvpc, FargateͰv2Λར༻ • EC2/awsvpc͸v3ʹҠߦ༧ఆ

22. ECS/FargateͷωοτϫʔΫϝτϦοΫ • ωοτϫʔΫϝτϦοΫ͕औಘͰ͖Δͷ͸EC2/Bridge͚ͩ...... • Docker stats API(libnetwork)͸bridgeϞʔυͷͱ͖͔͠ϝτϦοΫ ͕औΕͳ͍ͬΆ͍ • ࣮૷ʹৄ͍͠ํɺͥͻ࠙਌ձͰ͓࿩͠·͠ΐ͏ʂ

23. root@ebb5c8c90634:/# curl -s $ {ECS_CONTAINER_METADATA_URI}/stats | jq .networks { "eth0":

    { "rx_bytes": 17331985, "tx_packets": 932, "rx_packets": 1353, "tx_bytes": 77755 } } e.g. EC2/Bridge w/ TMEv3

24. root@ip-10-0-10-144:/# curl -s ${ECS_CONTAINER_METADATA_URI}/stats | \ > jq .networks null

    e.g. EC2/Host, EC2/awsvpc w/ TMEv3 root@9ea93ec5d92b:/# curl -s ${ECS_CONTAINER_METADATA_URI}/stats | \ > jq .networks null

25. -bash-4.2# CID=$(basename $(head -n1 /proc/self/ cgroup | cut -d: -f3))

    -bash-4.2# curl -s 169.254.170.2/v2/stats/${CID} | jq .networks null e.g. Fargate w/ TMEv2

26. EC2/BridgeͷωοτϫʔΫελοΫ • ωοτϫʔΫελοΫ͸λεΫͷίϯςφ͝ͱʹҟͳΔ

27. EC2/HostωοτϫʔΫελοΫ • ϗετͷωοτϫʔΫελοΫΛλεΫͷίϯςφؒͰڞ༗

28. EC2/awsvpc, FargateωοτϫʔΫελοΫ • ωοτϫʔΫελοΫΛλεΫͷίϯςφؒͰڞ༗

29. ίϯςφΤʔδΣϯτʹ͓͚ΔECS/Fargateͷ ωοτϫʔΫϝτϦοΫͷऔಘ • EC2/BridgeϞʔυͷ৔߹ɺDocker stats API͔Β֤ίϯςφͷωο τϫʔΫϝτϦοΫΛऔಘͯ͠λεΫͷϝτϦοΫͱͯ͠౤ߘ • EC2/Host, EC2/awsvpc,

    FargateͰ͸ɺίϯςφಉ࢜͸ωοτϫʔ ΫελοΫΛڞ༗͢ΔͷͰɺίϯςφΤʔδΣϯτࣗ਎ͷωοτϫʔ ΫϝτϦοΫΛऔಘͯ͠λεΫͷϝτϦοΫͱͯ͠౤ߘ

30. • ಉ͡ίϯςφఆٛͰ΋ɺىಈλΠϓɺωοτϫʔΫϞʔυͰΠϯλʔ ϑΣʔεͷݟ͑ํ͕ҟͳΔ λεΫͷInterfaceάϥϑ &$#SJEHF &$)PTU &$BXTWQD
    'BSHBUF

31. • ىಈλΠϓɺωοτϫʔΫϞʔυʹΑͬͯར༻Ͱ͖ΔAPI͕ҟͳΔ • EC2ىಈλΠϓ͸Task Metadata Endpoint v3ʹରԠத • ωοτϫʔΫϞʔυʹΑͬͯωοτϫʔΫϝτϦοΫͷऔಘํ๏͕ ҟͳΔ

    ECS/Fargateʹ͓͚ΔϝτϦοΫऔಘ·ͱΊ

32. Kubernetes

33. KubernetesͷCore metrics pipeline

34. kubelet API • Podͷϝλσʔλ΍ϝτϦοΫɺϩάΛऔಘɺίϚϯυͷϩʔΧϧ ࣮ߦͷͨΊͷAPI • kubelet port(10250/HTTPS)ͱread-only port(10255/HTTP)Ͱ LISTEN

    • read-only port͕ແޮͳ؀ڥ΋͋Δ

35. kubelet APIͷAuthN/AuthZ • kubelet portͰ͸ೝূ/ೝՄΛઃఆͰ͖Δ • Authentication • ಗ໊ΞΫηεɺΫϥΠΞϯτূ໌ॻೝূɺτʔΫϯೝূ •

    Authorization • AlwaysAllow, Webhook • SubjectAccessReview APIʹΑΔݖݶνΣοΫ • see. ʮKubelet authentication/authorization - Kubernetesʯ

36. ίϯςφΤʔδΣϯτͱkubelet API • σϑΥϧτͰ͸read-only portΛར༻ • ઃఆͰkubelet portʹ੾Γସ͑Մೳ • τʔΫϯೝূΛαϙʔτ

    • "nodes/proxy", "nodes/stats", "nodes/spec"ʹgetΞΫηε • ͏·͍͔͘ͳ͍৔߹͸automountServiceAccountTokenઃఆ΋νΣοΫ • see. ʮKubernetesʹmackerel-container-agentΛηοτΞοϓ͢Δ - Mackerel ϔϧϓʯ

37. apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: mackerel-container-agent rules: - apiGroups:

    [""] resources: ["nodes/proxy", "nodes/stats", "nodes/spec"] verbs: ["get"] e.g. RBACઃఆ(ClusterRole)

38. apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: mackerel-container-agent-binding roleRef: apiGroup: rbac.authorization.k8s.io

    kind: ClusterRole name: mackerel-container-agent subjects: - kind: ServiceAccount name: my-service-account namespace: default e.g. RBACઃఆ(ClusterRoleBinding)

39. • kubelet API͔ΒϝτϦοΫΛऔಘ͢Δ • σϑΥϧτ͸read-only PortΛར༻ • kubelet portͰ͸τʔΫϯೝূʹରԠ •

    ඞཁʹԠͯ͡RBACΛઃఆ Kubernetesʹ͓͚ΔϝτϦοΫऔಘ·ͱΊ

40. ࠓޙͷίϯςφΤʔδΣϯτ

41. ۙ೔ରԠ༧ఆ • Task Metadata Endpoint v3ରԠ • EC2/Bridge, EC2/Host, EC2/awspvc

    • docker.sock΍cgroupfs΁ͷґଘΛͳ͘͢ • rootϢʔβඞཁͳ͠ • cgroupfsͷϚ΢ϯτϙΠϯτͷҧ͍Λؾʹ͠ͳͯ͘Α͍

42. ల๬ͱߏ૝ • ϓϥάΠϯར༻ͷརศੑ޲্ • ϓϥάΠϯಉࠝΠϝʔδͷఏڙͳͲ • ΧελϜϝτϦοΫͷѻ͍ • ϗετಉ༷ʹୀ໾ޙɺҰఆظؒܦաͰඇදࣔͱͳΔ •

    Prometheus΍ServiceMeshͱͷ࿈ܞ

43. ·ͱΊ

44. ·ͱΊ • ίϯςφΤʔδΣϯτʢύϒϦοΫϕʔλʣΛϦϦʔε • λεΫ/PodΛ؂ࢹ͢ΔαΠυΧʔίϯςφ • ϓϥοτϑΥʔϜ͕ఏڙ͢ΔAPI͔ΒϝτϦοΫΛऩू • see. ʮίϯςφΛ؂ࢹ͢Δ

    - Mackerel ϔϧϓʯ

45. ϑΟʔυόοΫΛ͓଴͍ͪͯ͠·͢

46. None