Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mackerelコンテナエージェントによる コンテナ監視について / Mackerel Meetup #13 Tokyo

Mackerelコンテナエージェントによる コンテナ監視について / Mackerel Meetup #13 Tokyo

Hayato Imai

March 01, 2019
Tweet

More Decks by Hayato Imai

Other Decks in Technology

Transcript

  1. MackerelίϯςφΤʔδΣϯτʹΑΔ
    ίϯςφ؂ࢹʹ͍ͭͯ
    .BDLFSFM.FFUVQ
    JEIBZBKP@

    View full-size slide

  2. ࣗݾ঺հ
    • ࠓҪ൏ਓ(id:hayajo_77)
    • MackerelνʔϜ SRE
    • 2017೥11݄ೖࣾ
    • ίϯςφཁૉٕज़ɺपลٕज़
    ίϯςφٕज़ೖ໳Ծ૝Խͱͷҧ͍Λ஌Γɺཁૉٕज़Λ৮ֶͬͯ΅͏
    IUUQTFNQMPZNFOUFOKBQBODPNFOHJOFFSIVCFOUSZ

    View full-size slide

  3. ΞδΣϯμ
    • MackerelίϯςφΤʔδΣϯτͷ঺հ
    • ϝτϦοΫऔಘઓུղઆ
    • ࠓޙͷίϯςφΤʔδΣϯτ
    • ·ͱΊ

    View full-size slide

  4. MackerelίϯςφΤʔδΣϯτͷ঺հ

    View full-size slide

  5. MackerelίϯςφΤʔδΣϯτ
    ʢύϒϦοΫϕʔλʣΛެ։͠·ͨ͠
    https://mackerel.io/ja/blog/entry/weekly/20190218

    View full-size slide

  6. ύϒϦοΫϕʔλͱ͸
    • কདྷతʹਖ਼ࣜͳαʔϏεϦϦʔεΛߦ͏༧ఆ͕͋ΔػೳΛઌߦͯ͠
    ެ։
    • ׆ൃʹมߋΛߦ͍ɺ৽͍͠όʔδϣϯΛఏڙ
    • ࣄલࠂ஌ͷ্Ͱඇޓ׵मਖ਼Λ࣮ࢪ

    View full-size slide

  7. OSSͱͯ͠ެ։
    • DockerHub
    • mackerel/mackerel-container-agent
    • GitHub
    • mackerelio/mackerel-container-agent
    • Issue΍PR͸ݪଇӳޠͰ͓ئ͍͠·͢

    View full-size slide

  8. ίϯςφઐ༻ͷܰྔΤʔδΣϯτ
    • ίϯςφઐ༻ͱͯ͠࠶ઃܭ
    • Amazon ECS, AWS Fargate, KubernetesʹରԠ
    • 1λεΫ/Podʹ͖ͭ1ϗετͱͯ͠Χ΢ϯτ
    • ʰγεςϜΛߏ੒͢Δ࠷খ୯Ґ(ͩͱγεςϜΛ؅ཧ͢Δ্ͰΈͳ
    ͢΂͖΋ͷ)ʱ
    • see. ʮFAQɾϗετ਺ͷܭࢉํ๏ʹ͍ͭͯ - Mackerel ϔϧϓʯ

    View full-size slide

  9. αΠυΧʔίϯςφͱͯ͠σϓϩΠ
    • ϓϥοτϑΥʔϜʹΑͬͯઃఆ͕ҟͳΔ
    • ECS(EC2/Bridge, EC2/Host)
    • ECS(EC2/awsvpc, Fargate)
    • Kubernetes
    • see. ʮίϯςφΛ؂ࢹ͢Δ - Mackerel ϔϧϓʯ

    View full-size slide

  10. Web UI/ϗετҰཡ

    View full-size slide

  11. Web UI/ϗετৄࡉ

    View full-size slide

  12. Web UI/ϩʔϧάϥϑ

    View full-size slide

  13. ΍Δ͜ͱ
    • λεΫ/Podͷ͢΂ͯͷίϯςφͷϝτϦοΫΛγεςϜϝτϦοΫ
    ͱͯ͠౤ߘ
    • CPUɺϝϞϦɺωοτϫʔΫ
    • αʔϏε/ϩʔϧͷׂΓ౰ͯ
    • ϓϥάΠϯͷར༻
    • ϓϥάΠϯΛΠϯετʔϧͨ͠DockerΠϝʔδͷ४උ͕ඞཁ

    View full-size slide

  14. ΍Βͳ͍͜ͱ
    • Ϋϥελ΍ϊʔυͷϝτϦοΫ͸ऩू͠ͳ͍
    • ϊʔυ਺ɺλεΫ/Pod਺, etc.
    • ϊʔυͷCPUɺϝϞϦɺωοτϫʔΫɺσΟεΫͳͲ
    • mackerel-agentΛར༻
    • αϙʔτ͢ΔϓϥοτϑΥʔϜҎ֎ͷίϯςφͷ؂ࢹ
    • Docker͸ʮDockerΛϞχλϦϯά͢Δ - Mackerel ϔϧϓʯ

    View full-size slide

  15. αΠυΧʔύλʔϯͷ࠾༻
    • FargateͷΑ͏ͳɺΠϯϑϥετϥΫνϟͷ؅ཧ͕ෆཁͳίϯςφ
    ར༻͕ओྲྀʹͳΔͱߟ͍͑ͯΔ
    • Mackerelʹ͓͚ΔϗετͷఆٛͱλεΫ/PodͷϥΠϑαΠΫϧ
    • ίϯςφؒͷϦιʔεڞ༗
    • ؂ࢹର৅ͷΞϓϦέʔγϣϯ࢓༷ʹ͍ۙ؂ࢹઃఆ͕Մೳ

    View full-size slide

  16. ϝτϦοΫऔಘઓུղઆ

    View full-size slide

  17. ϝτϦοΫऔಘΞʔΩςΫνϟ

    View full-size slide

  18. ECS/FargateͷAPI
    &$#SJEHF &$)PTU &$BXTWQD 'BSHBUF
    *OUSPTQFDUJPO
    "1*
    ˔ ˔ ☓ ☓
    5BTL.FUBEBUB
    &OEQPJOUW
    ☓ ☓ ˔ ˔
    5BTL.FUBEBUB
    &OEQPJOUW
    ˔ ˔ ˔ ☓

    View full-size slide

  19. Introspection API
    • λεΫͷϝλσʔλΛฦ͢
    • λεΫͷARN΍εςʔλεɺ֤ίϯςφͷDockerIDͳͲ
    • ίϯςφͷϝτϦοΫ΍CPUɺϝϞϦLIMIT͸औಘͰ͖ͳ͍
    • Docker stats API(docker.sock)΍cgroupfsͰΧόʔ
    • EC2/Bridge, EC2/HostͰར༻
    • Task Metadata Endpoint v3ʹରԠ༧ఆ

    View full-size slide

  20. Task Metadata Endpoint v2/v3
    • λεΫϝλσʔλɺίϯςφϝτϦοΫΛฦ͢
    • ϝτϦοΫ͸Docker stats APIͦͷ΋ͷΛฦ͢
    • v2/v3ͰऔಘͰ͖Δσʔλʹେ͖ͳҧ͍͸ͳ͍
    • EC2/awsvpc, FargateͰv2Λར༻
    • EC2/awsvpc͸v3ʹҠߦ༧ఆ

    View full-size slide

  21. ECS/FargateͷωοτϫʔΫϝτϦοΫ
    • ωοτϫʔΫϝτϦοΫ͕औಘͰ͖Δͷ͸EC2/Bridge͚ͩ......
    • Docker stats API(libnetwork)͸bridgeϞʔυͷͱ͖͔͠ϝτϦοΫ
    ͕औΕͳ͍ͬΆ͍
    • ࣮૷ʹৄ͍͠ํɺͥͻ࠙਌ձͰ͓࿩͠·͠ΐ͏ʂ

    View full-size slide

  22. root@ebb5c8c90634:/# curl -s $
    {ECS_CONTAINER_METADATA_URI}/stats | jq .networks
    {
    "eth0": {
    "rx_bytes": 17331985,
    "tx_packets": 932,
    "rx_packets": 1353,
    "tx_bytes": 77755
    }
    }
    e.g. EC2/Bridge w/ TMEv3

    View full-size slide

  23. root@ip-10-0-10-144:/# curl -s ${ECS_CONTAINER_METADATA_URI}/stats | \
    > jq .networks
    null
    e.g. EC2/Host, EC2/awsvpc w/ TMEv3
    root@9ea93ec5d92b:/# curl -s ${ECS_CONTAINER_METADATA_URI}/stats | \
    > jq .networks
    null

    View full-size slide

  24. -bash-4.2# CID=$(basename $(head -n1 /proc/self/
    cgroup | cut -d: -f3))
    -bash-4.2# curl -s 169.254.170.2/v2/stats/${CID} |
    jq .networks
    null
    e.g. Fargate w/ TMEv2

    View full-size slide

  25. EC2/BridgeͷωοτϫʔΫελοΫ
    • ωοτϫʔΫελοΫ͸λεΫͷίϯςφ͝ͱʹҟͳΔ

    View full-size slide

  26. EC2/HostωοτϫʔΫελοΫ
    • ϗετͷωοτϫʔΫελοΫΛλεΫͷίϯςφؒͰڞ༗

    View full-size slide

  27. EC2/awsvpc, FargateωοτϫʔΫελοΫ
    • ωοτϫʔΫελοΫΛλεΫͷίϯςφؒͰڞ༗

    View full-size slide

  28. ίϯςφΤʔδΣϯτʹ͓͚ΔECS/Fargateͷ
    ωοτϫʔΫϝτϦοΫͷऔಘ
    • EC2/BridgeϞʔυͷ৔߹ɺDocker stats API͔Β֤ίϯςφͷωο
    τϫʔΫϝτϦοΫΛऔಘͯ͠λεΫͷϝτϦοΫͱͯ͠౤ߘ
    • EC2/Host, EC2/awsvpc, FargateͰ͸ɺίϯςφಉ࢜͸ωοτϫʔ
    ΫελοΫΛڞ༗͢ΔͷͰɺίϯςφΤʔδΣϯτࣗ਎ͷωοτϫʔ
    ΫϝτϦοΫΛऔಘͯ͠λεΫͷϝτϦοΫͱͯ͠౤ߘ

    View full-size slide

  29. • ಉ͡ίϯςφఆٛͰ΋ɺىಈλΠϓɺωοτϫʔΫϞʔυͰΠϯλʔ
    ϑΣʔεͷݟ͑ํ͕ҟͳΔ
    λεΫͷInterfaceάϥϑ
    &$#SJEHF &$)PTU &$BXTWQD 'BSHBUF

    View full-size slide

  30. • ىಈλΠϓɺωοτϫʔΫϞʔυʹΑͬͯར༻Ͱ͖ΔAPI͕ҟͳΔ
    • EC2ىಈλΠϓ͸Task Metadata Endpoint v3ʹରԠத
    • ωοτϫʔΫϞʔυʹΑͬͯωοτϫʔΫϝτϦοΫͷऔಘํ๏͕
    ҟͳΔ
    ECS/Fargateʹ͓͚ΔϝτϦοΫऔಘ·ͱΊ

    View full-size slide

  31. KubernetesͷCore metrics pipeline

    View full-size slide

  32. kubelet API
    • Podͷϝλσʔλ΍ϝτϦοΫɺϩάΛऔಘɺίϚϯυͷϩʔΧϧ
    ࣮ߦͷͨΊͷAPI
    • kubelet port(10250/HTTPS)ͱread-only port(10255/HTTP)Ͱ
    LISTEN
    • read-only port͕ແޮͳ؀ڥ΋͋Δ

    View full-size slide

  33. kubelet APIͷAuthN/AuthZ
    • kubelet portͰ͸ೝূ/ೝՄΛઃఆͰ͖Δ
    • Authentication
    • ಗ໊ΞΫηεɺΫϥΠΞϯτূ໌ॻೝূɺτʔΫϯೝূ
    • Authorization
    • AlwaysAllow, Webhook
    • SubjectAccessReview APIʹΑΔݖݶνΣοΫ
    • see. ʮKubelet authentication/authorization - Kubernetesʯ

    View full-size slide

  34. ίϯςφΤʔδΣϯτͱkubelet API
    • σϑΥϧτͰ͸read-only portΛར༻
    • ઃఆͰkubelet portʹ੾Γସ͑Մೳ
    • τʔΫϯೝূΛαϙʔτ
    • "nodes/proxy", "nodes/stats", "nodes/spec"ʹgetΞΫηε
    • ͏·͍͔͘ͳ͍৔߹͸automountServiceAccountTokenઃఆ΋νΣοΫ
    • see. ʮKubernetesʹmackerel-container-agentΛηοτΞοϓ͢Δ -
    Mackerel ϔϧϓʯ

    View full-size slide

  35. apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
    name: mackerel-container-agent
    rules:
    - apiGroups: [""]
    resources: ["nodes/proxy", "nodes/stats", "nodes/spec"]
    verbs: ["get"]
    e.g. RBACઃఆ(ClusterRole)

    View full-size slide

  36. apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
    name: mackerel-container-agent-binding
    roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: mackerel-container-agent
    subjects:
    - kind: ServiceAccount
    name: my-service-account
    namespace: default
    e.g. RBACઃఆ(ClusterRoleBinding)

    View full-size slide

  37. • kubelet API͔ΒϝτϦοΫΛऔಘ͢Δ
    • σϑΥϧτ͸read-only PortΛར༻
    • kubelet portͰ͸τʔΫϯೝূʹରԠ
    • ඞཁʹԠͯ͡RBACΛઃఆ
    Kubernetesʹ͓͚ΔϝτϦοΫऔಘ·ͱΊ

    View full-size slide

  38. ࠓޙͷίϯςφΤʔδΣϯτ

    View full-size slide

  39. ۙ೔ରԠ༧ఆ
    • Task Metadata Endpoint v3ରԠ
    • EC2/Bridge, EC2/Host, EC2/awspvc
    • docker.sock΍cgroupfs΁ͷґଘΛͳ͘͢
    • rootϢʔβඞཁͳ͠
    • cgroupfsͷϚ΢ϯτϙΠϯτͷҧ͍Λؾʹ͠ͳͯ͘Α͍

    View full-size slide

  40. ల๬ͱߏ૝
    • ϓϥάΠϯར༻ͷརศੑ޲্
    • ϓϥάΠϯಉࠝΠϝʔδͷఏڙͳͲ
    • ΧελϜϝτϦοΫͷѻ͍
    • ϗετಉ༷ʹୀ໾ޙɺҰఆظؒܦաͰඇදࣔͱͳΔ
    • Prometheus΍ServiceMeshͱͷ࿈ܞ

    View full-size slide

  41. ·ͱΊ
    • ίϯςφΤʔδΣϯτʢύϒϦοΫϕʔλʣΛϦϦʔε
    • λεΫ/PodΛ؂ࢹ͢ΔαΠυΧʔίϯςφ
    • ϓϥοτϑΥʔϜ͕ఏڙ͢ΔAPI͔ΒϝτϦοΫΛऩू
    • see. ʮίϯςφΛ؂ࢹ͢Δ - Mackerel ϔϧϓʯ

    View full-size slide

  42. ϑΟʔυόοΫΛ͓଴͍ͪͯ͠·͢

    View full-size slide