Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mackerelコンテナエージェントによる コンテナ監視について / Mackerel Meetup #13 Tokyo

Mackerelコンテナエージェントによる コンテナ監視について / Mackerel Meetup #13 Tokyo

87425b9ed1c97009802d66c6aebbfcdb?s=128

Hayato Imai

March 01, 2019
Tweet

Transcript

  1. MackerelίϯςφΤʔδΣϯτʹΑΔ ίϯςφ؂ࢹʹ͍ͭͯ .BDLFSFM.FFUVQ JEIBZBKP@

  2. ࣗݾ঺հ • ࠓҪ൏ਓ(id:hayajo_77) • MackerelνʔϜ SRE • 2017೥11݄ೖࣾ • ίϯςφཁૉٕज़ɺपลٕज़

    ίϯςφٕज़ೖ໳Ծ૝Խͱͷҧ͍Λ஌Γɺཁૉٕज़Λ৮ֶͬͯ΅͏ IUUQTFNQMPZNFOUFOKBQBODPNFOHJOFFSIVCFOUSZ 
  3. ΞδΣϯμ • MackerelίϯςφΤʔδΣϯτͷ঺հ • ϝτϦοΫऔಘઓུղઆ • ࠓޙͷίϯςφΤʔδΣϯτ • ·ͱΊ

  4. MackerelίϯςφΤʔδΣϯτͷ঺հ

  5. MackerelίϯςφΤʔδΣϯτ ʢύϒϦοΫϕʔλʣΛެ։͠·ͨ͠ https://mackerel.io/ja/blog/entry/weekly/20190218

  6. ύϒϦοΫϕʔλͱ͸ • কདྷతʹਖ਼ࣜͳαʔϏεϦϦʔεΛߦ͏༧ఆ͕͋ΔػೳΛઌߦͯ͠ ެ։ • ׆ൃʹมߋΛߦ͍ɺ৽͍͠όʔδϣϯΛఏڙ • ࣄલࠂ஌ͷ্Ͱඇޓ׵मਖ਼Λ࣮ࢪ

  7. OSSͱͯ͠ެ։ • DockerHub • mackerel/mackerel-container-agent • GitHub • mackerelio/mackerel-container-agent •

    Issue΍PR͸ݪଇӳޠͰ͓ئ͍͠·͢
  8. ίϯςφઐ༻ͷܰྔΤʔδΣϯτ • ίϯςφઐ༻ͱͯ͠࠶ઃܭ • Amazon ECS, AWS Fargate, KubernetesʹରԠ •

    1λεΫ/Podʹ͖ͭ1ϗετͱͯ͠Χ΢ϯτ • ʰγεςϜΛߏ੒͢Δ࠷খ୯Ґ(ͩͱγεςϜΛ؅ཧ͢Δ্ͰΈͳ ͢΂͖΋ͷ)ʱ • see. ʮFAQɾϗετ਺ͷܭࢉํ๏ʹ͍ͭͯ - Mackerel ϔϧϓʯ
  9. αΠυΧʔίϯςφͱͯ͠σϓϩΠ • ϓϥοτϑΥʔϜʹΑͬͯઃఆ͕ҟͳΔ • ECS(EC2/Bridge, EC2/Host) • ECS(EC2/awsvpc, Fargate) •

    Kubernetes • see. ʮίϯςφΛ؂ࢹ͢Δ - Mackerel ϔϧϓʯ
  10. Web UI/ϗετҰཡ

  11. Web UI/ϗετৄࡉ

  12. Web UI/ϩʔϧάϥϑ

  13. ΍Δ͜ͱ • λεΫ/Podͷ͢΂ͯͷίϯςφͷϝτϦοΫΛγεςϜϝτϦοΫ ͱͯ͠౤ߘ • CPUɺϝϞϦɺωοτϫʔΫ • αʔϏε/ϩʔϧͷׂΓ౰ͯ • ϓϥάΠϯͷར༻

    • ϓϥάΠϯΛΠϯετʔϧͨ͠DockerΠϝʔδͷ४උ͕ඞཁ
  14. ΍Βͳ͍͜ͱ • Ϋϥελ΍ϊʔυͷϝτϦοΫ͸ऩू͠ͳ͍ • ϊʔυ਺ɺλεΫ/Pod਺, etc. • ϊʔυͷCPUɺϝϞϦɺωοτϫʔΫɺσΟεΫͳͲ • mackerel-agentΛར༻

    • αϙʔτ͢ΔϓϥοτϑΥʔϜҎ֎ͷίϯςφͷ؂ࢹ • Docker͸ʮDockerΛϞχλϦϯά͢Δ - Mackerel ϔϧϓʯ
  15. αΠυΧʔύλʔϯͷ࠾༻ • FargateͷΑ͏ͳɺΠϯϑϥετϥΫνϟͷ؅ཧ͕ෆཁͳίϯςφ ར༻͕ओྲྀʹͳΔͱߟ͍͑ͯΔ • Mackerelʹ͓͚ΔϗετͷఆٛͱλεΫ/PodͷϥΠϑαΠΫϧ • ίϯςφؒͷϦιʔεڞ༗ • ؂ࢹର৅ͷΞϓϦέʔγϣϯ࢓༷ʹ͍ۙ؂ࢹઃఆ͕Մೳ

  16. ϝτϦοΫऔಘઓུղઆ

  17. ϝτϦοΫऔಘΞʔΩςΫνϟ

  18. ECS/Fargate

  19. ECS/FargateͷAPI &$#SJEHF &$)PTU &$BXTWQD 'BSHBUF *OUSPTQFDUJPO "1* ˔ ˔ ☓

    ☓ 5BTL.FUBEBUB &OEQPJOUW ☓ ☓ ˔ ˔ 5BTL.FUBEBUB &OEQPJOUW ˔ ˔ ˔ ☓
  20. Introspection API • λεΫͷϝλσʔλΛฦ͢ • λεΫͷARN΍εςʔλεɺ֤ίϯςφͷDockerIDͳͲ • ίϯςφͷϝτϦοΫ΍CPUɺϝϞϦLIMIT͸औಘͰ͖ͳ͍ • Docker

    stats API(docker.sock)΍cgroupfsͰΧόʔ • EC2/Bridge, EC2/HostͰར༻ • Task Metadata Endpoint v3ʹରԠ༧ఆ
  21. Task Metadata Endpoint v2/v3 • λεΫϝλσʔλɺίϯςφϝτϦοΫΛฦ͢ • ϝτϦοΫ͸Docker stats APIͦͷ΋ͷΛฦ͢

    • v2/v3ͰऔಘͰ͖Δσʔλʹେ͖ͳҧ͍͸ͳ͍ • EC2/awsvpc, FargateͰv2Λར༻ • EC2/awsvpc͸v3ʹҠߦ༧ఆ
  22. ECS/FargateͷωοτϫʔΫϝτϦοΫ • ωοτϫʔΫϝτϦοΫ͕औಘͰ͖Δͷ͸EC2/Bridge͚ͩ...... • Docker stats API(libnetwork)͸bridgeϞʔυͷͱ͖͔͠ϝτϦοΫ ͕औΕͳ͍ͬΆ͍ • ࣮૷ʹৄ͍͠ํɺͥͻ࠙਌ձͰ͓࿩͠·͠ΐ͏ʂ

  23. root@ebb5c8c90634:/# curl -s $ {ECS_CONTAINER_METADATA_URI}/stats | jq .networks { "eth0":

    { "rx_bytes": 17331985, "tx_packets": 932, "rx_packets": 1353, "tx_bytes": 77755 } } e.g. EC2/Bridge w/ TMEv3
  24. root@ip-10-0-10-144:/# curl -s ${ECS_CONTAINER_METADATA_URI}/stats | \ > jq .networks null

    e.g. EC2/Host, EC2/awsvpc w/ TMEv3 root@9ea93ec5d92b:/# curl -s ${ECS_CONTAINER_METADATA_URI}/stats | \ > jq .networks null
  25. -bash-4.2# CID=$(basename $(head -n1 /proc/self/ cgroup | cut -d: -f3))

    -bash-4.2# curl -s 169.254.170.2/v2/stats/${CID} | jq .networks null e.g. Fargate w/ TMEv2
  26. EC2/BridgeͷωοτϫʔΫελοΫ • ωοτϫʔΫελοΫ͸λεΫͷίϯςφ͝ͱʹҟͳΔ

  27. EC2/HostωοτϫʔΫελοΫ • ϗετͷωοτϫʔΫελοΫΛλεΫͷίϯςφؒͰڞ༗

  28. EC2/awsvpc, FargateωοτϫʔΫελοΫ • ωοτϫʔΫελοΫΛλεΫͷίϯςφؒͰڞ༗

  29. ίϯςφΤʔδΣϯτʹ͓͚ΔECS/Fargateͷ ωοτϫʔΫϝτϦοΫͷऔಘ • EC2/BridgeϞʔυͷ৔߹ɺDocker stats API͔Β֤ίϯςφͷωο τϫʔΫϝτϦοΫΛऔಘͯ͠λεΫͷϝτϦοΫͱͯ͠౤ߘ • EC2/Host, EC2/awsvpc,

    FargateͰ͸ɺίϯςφಉ࢜͸ωοτϫʔ ΫελοΫΛڞ༗͢ΔͷͰɺίϯςφΤʔδΣϯτࣗ਎ͷωοτϫʔ ΫϝτϦοΫΛऔಘͯ͠λεΫͷϝτϦοΫͱͯ͠౤ߘ
  30. • ಉ͡ίϯςφఆٛͰ΋ɺىಈλΠϓɺωοτϫʔΫϞʔυͰΠϯλʔ ϑΣʔεͷݟ͑ํ͕ҟͳΔ λεΫͷInterfaceάϥϑ &$#SJEHF &$)PTU &$BXTWQD 'BSHBUF

  31. • ىಈλΠϓɺωοτϫʔΫϞʔυʹΑͬͯར༻Ͱ͖ΔAPI͕ҟͳΔ • EC2ىಈλΠϓ͸Task Metadata Endpoint v3ʹରԠத • ωοτϫʔΫϞʔυʹΑͬͯωοτϫʔΫϝτϦοΫͷऔಘํ๏͕ ҟͳΔ

    ECS/Fargateʹ͓͚ΔϝτϦοΫऔಘ·ͱΊ
  32. Kubernetes

  33. KubernetesͷCore metrics pipeline

  34. kubelet API • Podͷϝλσʔλ΍ϝτϦοΫɺϩάΛऔಘɺίϚϯυͷϩʔΧϧ ࣮ߦͷͨΊͷAPI • kubelet port(10250/HTTPS)ͱread-only port(10255/HTTP)Ͱ LISTEN

    • read-only port͕ແޮͳ؀ڥ΋͋Δ
  35. kubelet APIͷAuthN/AuthZ • kubelet portͰ͸ೝূ/ೝՄΛઃఆͰ͖Δ • Authentication • ಗ໊ΞΫηεɺΫϥΠΞϯτূ໌ॻೝূɺτʔΫϯೝূ •

    Authorization • AlwaysAllow, Webhook • SubjectAccessReview APIʹΑΔݖݶνΣοΫ • see. ʮKubelet authentication/authorization - Kubernetesʯ
  36. ίϯςφΤʔδΣϯτͱkubelet API • σϑΥϧτͰ͸read-only portΛར༻ • ઃఆͰkubelet portʹ੾Γସ͑Մೳ • τʔΫϯೝূΛαϙʔτ

    • "nodes/proxy", "nodes/stats", "nodes/spec"ʹgetΞΫηε • ͏·͍͔͘ͳ͍৔߹͸automountServiceAccountTokenઃఆ΋νΣοΫ • see. ʮKubernetesʹmackerel-container-agentΛηοτΞοϓ͢Δ - Mackerel ϔϧϓʯ
  37. apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: mackerel-container-agent rules: - apiGroups:

    [""] resources: ["nodes/proxy", "nodes/stats", "nodes/spec"] verbs: ["get"] e.g. RBACઃఆ(ClusterRole)
  38. apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: mackerel-container-agent-binding roleRef: apiGroup: rbac.authorization.k8s.io

    kind: ClusterRole name: mackerel-container-agent subjects: - kind: ServiceAccount name: my-service-account namespace: default e.g. RBACઃఆ(ClusterRoleBinding)
  39. • kubelet API͔ΒϝτϦοΫΛऔಘ͢Δ • σϑΥϧτ͸read-only PortΛར༻ • kubelet portͰ͸τʔΫϯೝূʹରԠ •

    ඞཁʹԠͯ͡RBACΛઃఆ Kubernetesʹ͓͚ΔϝτϦοΫऔಘ·ͱΊ
  40. ࠓޙͷίϯςφΤʔδΣϯτ

  41. ۙ೔ରԠ༧ఆ • Task Metadata Endpoint v3ରԠ • EC2/Bridge, EC2/Host, EC2/awspvc

    • docker.sock΍cgroupfs΁ͷґଘΛͳ͘͢ • rootϢʔβඞཁͳ͠ • cgroupfsͷϚ΢ϯτϙΠϯτͷҧ͍Λؾʹ͠ͳͯ͘Α͍
  42. ల๬ͱߏ૝ • ϓϥάΠϯར༻ͷརศੑ޲্ • ϓϥάΠϯಉࠝΠϝʔδͷఏڙͳͲ • ΧελϜϝτϦοΫͷѻ͍ • ϗετಉ༷ʹୀ໾ޙɺҰఆظؒܦաͰඇදࣔͱͳΔ •

    Prometheus΍ServiceMeshͱͷ࿈ܞ
  43. ·ͱΊ

  44. ·ͱΊ • ίϯςφΤʔδΣϯτʢύϒϦοΫϕʔλʣΛϦϦʔε • λεΫ/PodΛ؂ࢹ͢ΔαΠυΧʔίϯςφ • ϓϥοτϑΥʔϜ͕ఏڙ͢ΔAPI͔ΒϝτϦοΫΛऩू • see. ʮίϯςφΛ؂ࢹ͢Δ

    - Mackerel ϔϧϓʯ
  45. ϑΟʔυόοΫΛ͓଴͍ͪͯ͠·͢

  46. None