Mackerelコンテナエージェントによる コンテナ監視について / Mackerel Meetup #13 Tokyo

2019/03/01 https://mackerelio.connpass.com/event/118995/

Transcript

1. MackerelίϯςφΤʔδΣϯτʹΑΔ
   ίϯςφ؂ࢹʹ͍ͭͯ
   .BDLFSFM.FFUVQ
   JEIBZBKP@
   

   View Slide

2. ࣗݾ঺հ
   • ࠓҪ൏ਓ(id:hayajo_77)
   • MackerelνʔϜ SRE
   • 2017೥11݄ೖࣾ
   • ίϯςφཁૉٕज़ɺपลٕज़
   ίϯςφٕज़ೖ໳Ծ૝Խͱͷҧ͍Λ஌Γɺཁૉٕज़Λ৮ֶͬͯ΅͏
   IUUQTFNQMPZNFOUFOKBQBODPNFOHJOFFSIVCFOUSZ
   
   

   View Slide

3. ΞδΣϯμ
   • MackerelίϯςφΤʔδΣϯτͷ঺հ
   • ϝτϦοΫऔಘઓུղઆ
   • ࠓޙͷίϯςφΤʔδΣϯτ
   • ·ͱΊ
   

   View Slide

4. MackerelίϯςφΤʔδΣϯτͷ঺հ
   

   View Slide

5. MackerelίϯςφΤʔδΣϯτ
   ʢύϒϦοΫϕʔλʣΛެ։͠·ͨ͠
   https://mackerel.io/ja/blog/entry/weekly/20190218
   

   View Slide

6. ύϒϦοΫϕʔλͱ͸
   • কདྷతʹਖ਼ࣜͳαʔϏεϦϦʔεΛߦ͏༧ఆ͕͋ΔػೳΛઌߦͯ͠
   ެ։
   • ׆ൃʹมߋΛߦ͍ɺ৽͍͠όʔδϣϯΛఏڙ
   • ࣄલࠂ஌ͷ্Ͱඇޓ׵मਖ਼Λ࣮ࢪ
   

   View Slide

7. OSSͱͯ͠ެ։
   • DockerHub
   • mackerel/mackerel-container-agent
   • GitHub
   • mackerelio/mackerel-container-agent
   • Issue΍PR͸ݪଇӳޠͰ͓ئ͍͠·͢
   

   View Slide

8. ίϯςφઐ༻ͷܰྔΤʔδΣϯτ
   • ίϯςφઐ༻ͱͯ͠࠶ઃܭ
   • Amazon ECS, AWS Fargate, KubernetesʹରԠ
   • 1λεΫ/Podʹ͖ͭ1ϗετͱͯ͠Χ΢ϯτ
   • ʰγεςϜΛߏ੒͢Δ࠷খ୯Ґ(ͩͱγεςϜΛ؅ཧ͢Δ্ͰΈͳ
   ͢΂͖΋ͷ)ʱ
   • see. ʮFAQɾϗετ਺ͷܭࢉํ๏ʹ͍ͭͯ - Mackerel ϔϧϓʯ
   

   View Slide

9. αΠυΧʔίϯςφͱͯ͠σϓϩΠ
   • ϓϥοτϑΥʔϜʹΑͬͯઃఆ͕ҟͳΔ
   • ECS(EC2/Bridge, EC2/Host)
   • ECS(EC2/awsvpc, Fargate)
   • Kubernetes
   • see. ʮίϯςφΛ؂ࢹ͢Δ - Mackerel ϔϧϓʯ
   

   View Slide

10. Web UI/ϗετҰཡ
    

    View Slide

11. Web UI/ϗετৄࡉ
    

    View Slide

12. Web UI/ϩʔϧάϥϑ
    

    View Slide

13. ΍Δ͜ͱ
    • λεΫ/Podͷ͢΂ͯͷίϯςφͷϝτϦοΫΛγεςϜϝτϦοΫ
    ͱͯ͠౤ߘ
    • CPUɺϝϞϦɺωοτϫʔΫ
    • αʔϏε/ϩʔϧͷׂΓ౰ͯ
    • ϓϥάΠϯͷར༻
    • ϓϥάΠϯΛΠϯετʔϧͨ͠DockerΠϝʔδͷ४උ͕ඞཁ
    

    View Slide

14. ΍Βͳ͍͜ͱ
    • Ϋϥελ΍ϊʔυͷϝτϦοΫ͸ऩू͠ͳ͍
    • ϊʔυ਺ɺλεΫ/Pod਺, etc.
    • ϊʔυͷCPUɺϝϞϦɺωοτϫʔΫɺσΟεΫͳͲ
    • mackerel-agentΛར༻
    • αϙʔτ͢ΔϓϥοτϑΥʔϜҎ֎ͷίϯςφͷ؂ࢹ
    • Docker͸ʮDockerΛϞχλϦϯά͢Δ - Mackerel ϔϧϓʯ
    

    View Slide

15. αΠυΧʔύλʔϯͷ࠾༻
    • FargateͷΑ͏ͳɺΠϯϑϥετϥΫνϟͷ؅ཧ͕ෆཁͳίϯςφ
    ར༻͕ओྲྀʹͳΔͱߟ͍͑ͯΔ
    • Mackerelʹ͓͚ΔϗετͷఆٛͱλεΫ/PodͷϥΠϑαΠΫϧ
    • ίϯςφؒͷϦιʔεڞ༗
    • ؂ࢹର৅ͷΞϓϦέʔγϣϯ࢓༷ʹ͍ۙ؂ࢹઃఆ͕Մೳ
    

    View Slide

16. ϝτϦοΫऔಘઓུղઆ
    

    View Slide

17. ϝτϦοΫऔಘΞʔΩςΫνϟ
    

    View Slide

18. ECS/Fargate
    

    View Slide

19. ECS/FargateͷAPI
    &$#SJEHF &$)PTU &$BXTWQD 'BSHBUF
    *OUSPTQFDUJPO
    "1*
    ˔ ˔ ☓ ☓
    5BTL.FUBEBUB
    &OEQPJOUW
    ☓ ☓ ˔ ˔
    5BTL.FUBEBUB
    &OEQPJOUW
    ˔ ˔ ˔ ☓
    

    View Slide

20. Introspection API
    • λεΫͷϝλσʔλΛฦ͢
    • λεΫͷARN΍εςʔλεɺ֤ίϯςφͷDockerIDͳͲ
    • ίϯςφͷϝτϦοΫ΍CPUɺϝϞϦLIMIT͸औಘͰ͖ͳ͍
    • Docker stats API(docker.sock)΍cgroupfsͰΧόʔ
    • EC2/Bridge, EC2/HostͰར༻
    • Task Metadata Endpoint v3ʹରԠ༧ఆ
    

    View Slide

21. Task Metadata Endpoint v2/v3
    • λεΫϝλσʔλɺίϯςφϝτϦοΫΛฦ͢
    • ϝτϦοΫ͸Docker stats APIͦͷ΋ͷΛฦ͢
    • v2/v3ͰऔಘͰ͖Δσʔλʹେ͖ͳҧ͍͸ͳ͍
    • EC2/awsvpc, FargateͰv2Λར༻
    • EC2/awsvpc͸v3ʹҠߦ༧ఆ
    

    View Slide

22. ECS/FargateͷωοτϫʔΫϝτϦοΫ
    • ωοτϫʔΫϝτϦοΫ͕औಘͰ͖Δͷ͸EC2/Bridge͚ͩ......
    • Docker stats API(libnetwork)͸bridgeϞʔυͷͱ͖͔͠ϝτϦοΫ
    ͕औΕͳ͍ͬΆ͍
    • ࣮૷ʹৄ͍͠ํɺͥͻ࠙਌ձͰ͓࿩͠·͠ΐ͏ʂ
    

    View Slide

23. root@ebb5c8c90634:/# curl -s $
    {ECS_CONTAINER_METADATA_URI}/stats | jq .networks
    {
    "eth0": {
    "rx_bytes": 17331985,
    "tx_packets": 932,
    "rx_packets": 1353,
    "tx_bytes": 77755
    }
    }
    e.g. EC2/Bridge w/ TMEv3
    

    View Slide

24. root@ip-10-0-10-144:/# curl -s ${ECS_CONTAINER_METADATA_URI}/stats | \
    > jq .networks
    null
    e.g. EC2/Host, EC2/awsvpc w/ TMEv3
    root@9ea93ec5d92b:/# curl -s ${ECS_CONTAINER_METADATA_URI}/stats | \
    > jq .networks
    null
    

    View Slide

25. -bash-4.2# CID=$(basename $(head -n1 /proc/self/
    cgroup | cut -d: -f3))
    -bash-4.2# curl -s 169.254.170.2/v2/stats/${CID} |
    jq .networks
    null
    e.g. Fargate w/ TMEv2
    

    View Slide

26. EC2/BridgeͷωοτϫʔΫελοΫ
    • ωοτϫʔΫελοΫ͸λεΫͷίϯςφ͝ͱʹҟͳΔ
    

    View Slide

27. EC2/HostωοτϫʔΫελοΫ
    • ϗετͷωοτϫʔΫελοΫΛλεΫͷίϯςφؒͰڞ༗
    

    View Slide

28. EC2/awsvpc, FargateωοτϫʔΫελοΫ
    • ωοτϫʔΫελοΫΛλεΫͷίϯςφؒͰڞ༗
    

    View Slide

29. ίϯςφΤʔδΣϯτʹ͓͚ΔECS/Fargateͷ
    ωοτϫʔΫϝτϦοΫͷऔಘ
    • EC2/BridgeϞʔυͷ৔߹ɺDocker stats API͔Β֤ίϯςφͷωο
    τϫʔΫϝτϦοΫΛऔಘͯ͠λεΫͷϝτϦοΫͱͯ͠౤ߘ
    • EC2/Host, EC2/awsvpc, FargateͰ͸ɺίϯςφಉ࢜͸ωοτϫʔ
    ΫελοΫΛڞ༗͢ΔͷͰɺίϯςφΤʔδΣϯτࣗ਎ͷωοτϫʔ
    ΫϝτϦοΫΛऔಘͯ͠λεΫͷϝτϦοΫͱͯ͠౤ߘ
    

    View Slide

30. • ಉ͡ίϯςφఆٛͰ΋ɺىಈλΠϓɺωοτϫʔΫϞʔυͰΠϯλʔ
    ϑΣʔεͷݟ͑ํ͕ҟͳΔ
    λεΫͷInterfaceάϥϑ
    &$#SJEHF &$)PTU &$BXTWQD 'BSHBUF
    

    View Slide

31. • ىಈλΠϓɺωοτϫʔΫϞʔυʹΑͬͯར༻Ͱ͖ΔAPI͕ҟͳΔ
    • EC2ىಈλΠϓ͸Task Metadata Endpoint v3ʹରԠத
    • ωοτϫʔΫϞʔυʹΑͬͯωοτϫʔΫϝτϦοΫͷऔಘํ๏͕
    ҟͳΔ
    ECS/Fargateʹ͓͚ΔϝτϦοΫऔಘ·ͱΊ
    

    View Slide

32. Kubernetes
    

    View Slide

33. KubernetesͷCore metrics pipeline
    

    View Slide

34. kubelet API
    • Podͷϝλσʔλ΍ϝτϦοΫɺϩάΛऔಘɺίϚϯυͷϩʔΧϧ
    ࣮ߦͷͨΊͷAPI
    • kubelet port(10250/HTTPS)ͱread-only port(10255/HTTP)Ͱ
    LISTEN
    • read-only port͕ແޮͳ؀ڥ΋͋Δ
    

    View Slide

35. kubelet APIͷAuthN/AuthZ
    • kubelet portͰ͸ೝূ/ೝՄΛઃఆͰ͖Δ
    • Authentication
    • ಗ໊ΞΫηεɺΫϥΠΞϯτূ໌ॻೝূɺτʔΫϯೝূ
    • Authorization
    • AlwaysAllow, Webhook
    • SubjectAccessReview APIʹΑΔݖݶνΣοΫ
    • see. ʮKubelet authentication/authorization - Kubernetesʯ
    

    View Slide

36. ίϯςφΤʔδΣϯτͱkubelet API
    • σϑΥϧτͰ͸read-only portΛར༻
    • ઃఆͰkubelet portʹ੾Γସ͑Մೳ
    • τʔΫϯೝূΛαϙʔτ
    • "nodes/proxy", "nodes/stats", "nodes/spec"ʹgetΞΫηε
    • ͏·͍͔͘ͳ͍৔߹͸automountServiceAccountTokenઃఆ΋νΣοΫ
    • see. ʮKubernetesʹmackerel-container-agentΛηοτΞοϓ͢Δ -
    Mackerel ϔϧϓʯ
    

    View Slide

37. apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
    name: mackerel-container-agent
    rules:
    - apiGroups: [""]
    resources: ["nodes/proxy", "nodes/stats", "nodes/spec"]
    verbs: ["get"]
    e.g. RBACઃఆ(ClusterRole)
    

    View Slide

38. apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
    name: mackerel-container-agent-binding
    roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: mackerel-container-agent
    subjects:
    - kind: ServiceAccount
    name: my-service-account
    namespace: default
    e.g. RBACઃఆ(ClusterRoleBinding)
    

    View Slide

39. • kubelet API͔ΒϝτϦοΫΛऔಘ͢Δ
    • σϑΥϧτ͸read-only PortΛར༻
    • kubelet portͰ͸τʔΫϯೝূʹରԠ
    • ඞཁʹԠͯ͡RBACΛઃఆ
    Kubernetesʹ͓͚ΔϝτϦοΫऔಘ·ͱΊ
    

    View Slide

40. ࠓޙͷίϯςφΤʔδΣϯτ
    

    View Slide

41. ۙ೔ରԠ༧ఆ
    • Task Metadata Endpoint v3ରԠ
    • EC2/Bridge, EC2/Host, EC2/awspvc
    • docker.sock΍cgroupfs΁ͷґଘΛͳ͘͢
    • rootϢʔβඞཁͳ͠
    • cgroupfsͷϚ΢ϯτϙΠϯτͷҧ͍Λؾʹ͠ͳͯ͘Α͍
    

    View Slide

42. ల๬ͱߏ૝
    • ϓϥάΠϯར༻ͷརศੑ޲্
    • ϓϥάΠϯಉࠝΠϝʔδͷఏڙͳͲ
    • ΧελϜϝτϦοΫͷѻ͍
    • ϗετಉ༷ʹୀ໾ޙɺҰఆظؒܦաͰඇදࣔͱͳΔ
    • Prometheus΍ServiceMeshͱͷ࿈ܞ
    

    View Slide

43. ·ͱΊ
    

    View Slide

44. ·ͱΊ
    • ίϯςφΤʔδΣϯτʢύϒϦοΫϕʔλʣΛϦϦʔε
    • λεΫ/PodΛ؂ࢹ͢ΔαΠυΧʔίϯςφ
    • ϓϥοτϑΥʔϜ͕ఏڙ͢ΔAPI͔ΒϝτϦοΫΛऩू
    • see. ʮίϯςφΛ؂ࢹ͢Δ - Mackerel ϔϧϓʯ
    

    View Slide

45. ϑΟʔυόοΫΛ͓଴͍ͪͯ͠·͢
    

    View Slide

46. View Slide