mackerel-container-agentを用いたマルチテナントKubernetesのモニタリング

社内向けコンテナ基盤として構築中のマルチテナントKubernetesに対して、mackerel-container-agentを導入した際の知見の共有。

Transcript

1. mackerel-container-agentΛ༻͍ͨ ϚϧνςφϯτKubernetesͷ ϞχλϦϯά
2. ൃද಺༰ • ࣗݾ঺հ • ͜Ε·Ͱͷ؂ࢹͱ͜Ε͔Βͷ؂ࢹ • Ϛϧνςφϯτkubernetesͷߏ੒ʹ͍ͭͯ • σϓϩΠํ๏ͷ঺հ SideCarͰར༻͢Δ৔߹ MutatingAdmissionWebhookΛར༻ͨ͠৔߹ ServiceCatalog (TSB)Λར༻ͨ͠৔߹ • ·ͱΊ
3. ࣗݾ঺հ • ໊લɿޙ౻ ल৴ • ॴଐɿٕज़ຊ෦ PrivateCloud ։ൃνʔϜ • ग़਎ɿେ෼ • ܦྺɿ - ( 2014 / 04 ~ ) SIerʹ৽ଔೖࣾ - ( 2016 / 09 ~ ) ݱࡏͷձࣾʹத్ೖࣾ • ۀ຿ɿPrivateCloudʹ͍ͭͯॾʑ
4. ͜Ε·Ͱͷ؂ࢹͱ͜Ε͔Βͷ؂ࢹ αʔϏεͷ଍ճΓ 
 ݹ͘͸શͯ෺ཧαʔόͷ্ʹ৐͍ͬͯͨ 2013೥ࠒ͔ΒϓϥΠϕʔτΫϥ΢υͷఏڙ͢ΔԾ૝αʔό΁Ҡߦ
 ਺೥લ͔ΒɺAWS΍GCPͷ༷ͳύϒϦοΫΫϥ΢υ΁ͷҠߦ΋ਐΜͰ͍Δ ࠷ۙʹͳͬͯɺαʔϏεͷίϯςφԽ͕ਐΉΑ͏ʹͳ͖ͬͯͨ ෺ཧαʔό ίϯςφ Ծ૝αʔό
 (OpenStack) Ծ૝αʔό
 (಺੡Ϋϥ΢υ)
5. ͜Ε·Ͱͷ؂ࢹͱ͜Ε͔Βͷ؂ࢹ αʔϏεͷ؂ࢹ
 ݹ͘͸monɺNagiosɺMunin গ͠ਐΉͱZabbixɺSensu͕ൺֱతΑ͘࢖ΘΕ͍ͯͨ ͦͷޙɺࣗલͰ࣋ͭͷ͸ਏ͍ͱ͍͏ྲྀΕ͔Β MackerelɺDatadogʹ୅ද͞ΕΔSaaS΁ͷҠߦ͕࠷ۙͷྲྀΕ ෺ཧαʔό ίϯςφ Ծ૝αʔό
 (OpenStack) Ծ૝αʔό
 (಺੡Ϋϥ΢υ) mon
 Nagios
 Munin ??? Mackerel
 Datadog Zabbix
 Sensu
6. ͜Ε·Ͱͷ؂ࢹͱ͜Ε͔Βͷ؂ࢹ ৽͍͠σʔληϯλ
 2019೥3݄ʹΦʔϓϯ OpenStackΛ༻͍ͨԾ૝Խج൫ʹՃ͑ͯ
 ϚϧνςφϯτͷϚωʔδυKubernetesΛఏڙ Ծ૝Ϛγϯͷ؂ࢹαʔϏεͱͯ͠MackerelΛ࠾༻ ෺ཧαʔό ίϯςφ Ծ૝αʔό
 (OpenStack) Ծ૝αʔό
 (಺੡Ϋϥ΢υ) mon
 Nagios
 Munin ??? Mackerel
 Datadog Zabbix
 Sensu
7. ͜Ε·Ͱͷ؂ࢹͱ͜Ε͔Βͷ؂ࢹ ίϯςφ؀ڥͷ؂ࢹ
 ίϯςφ؀ڥͷ؂ࢹαʔϏεͱͯ͠MackerelΛ࠾༻ ࠾༻ཧ༝ 1. Ծ૝αʔόͱίϯςφΛಉ͡πʔϧͰ؂ࢹ͢Δ͜ͱ͕Մೳ 2. طଘͷϓϥάΠϯͷྲྀ༻͕Մೳ ෺ཧαʔό ίϯςφ Ծ૝αʔό
 (OpenStack) Ծ૝αʔό
 (಺੡Ϋϥ΢υ) mon
 Nagios
 Munin Mackerel Container Agent Mackerel
 Datadog Zabbix
 Sensu
8. Ϛϧνςφϯτkubernetesͷߏ੒ • Kubernetes (OpenShift) on OpenStack on Kubernetes Pod VM Pod Pod
9. Ϛϧνςφϯτkubernetesͷߏ੒ • Kubernetes (OpenShift) on OpenStack on Kubernetes Pod VM Pod Pod ຊ೔ͷൃදείʔϓ͸
 ίί
10. Ϛϧνςφϯτkubernetesͷߏ੒ → OKDͷུ֓ਤ • ུ֓ Master × 5 Etcd × 5 Route × 10 Node × ͍ͬͺ͍ Master Route(HAproxy) okd-node001(8core/48GB) Master Master Master okd-node002 Route(HAproxy) Etcd Etcd Etcd Etcd Etcd Prj-2
 pod1 Prj-1
 pod1 Prj-1
 pod2 Prj-3
 pod1 Prj-2
 pod2 Prj-4
 pod1
11. mackerelಋೖͷHOW TO okd-node001(8core/48GB) Prj-2
 pod1 Prj-1
 pod1 Prj-1
 pod2 ͜͜ʹͲ͏΍ͬͯmackrel-container-agentΛ
 ಋೖ͢Δʁʁ
12. mackerelಋೖͷHOW TO • manifestʹsidecarΛهࡌͯ͠ߦ͏σϓϩΠ • ७ਮͳํ๏ • MutatingAdmissionWebhookΛར༻ͨ͠σϓϩΠ • istioͰͷenvoyͷinjectionͱಉ͡Α͏ͳํ๏ • TemplateΛར༻ͨ͠σϓϩΠ • OpenShiftͷtemplateػೳΛར༻ͨ͠ํ๏
13. mackerelಋೖͷHOW TO manifestʹsidecarΛهࡌͯ͠ߦ͏σϓϩΠ
14. manifestʹsidecarΛهࡌͯ͠σϓϩΠ • ಉҰPod಺ʹίϯςφΛ૬৐Γͤ͞Δํ๏ • kubernetesͷsidecarΛར༻ͨ͠σϓϩΠͰ͕͢
 ΋ͪΖΜOpenShiftͰ΋࢖͑·͢
15. manifestʹsidecarΛهࡌͯ͠σϓϩΠ containers: - name: service-container-main ɹɾ ɹɾ ɹɾ - name: mackerel-container-agent spec.template.spec.containers
16. Secret apikey: "YOUR_MACKEREL_APIKEY" roles: - "mackerelmeetup:database" ignoreContainer: '\Amackerel-container-agent\z' mackerel.yaml Create Secret $ oc create secret generic mackerel-config --from-file=./mackerel.yaml secret/mackerel-config created apikeyΛsecretʹͯ͠ઃఆྨ͸configmapͰ͋ͯΔ͜ͱ΋Ͱ͖Δ͕ɺ secretͱconfigmapΛ؅ཧ͢Δͷ͸໘౗ͳͨΊɺઃఆϑΝΠϧΛؙʑSecretʹͨ͠
17. DeploymentConfig (spec.template.spec.containers[].env[]) - name: MACKEREL_KUBERNETES_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace MACKEREL_KUBERNETES_NAMESPACE MACKEREL_KUBERNETES_POD_NAME - name: MACKEREL_KUBERNETES_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name
18. - name: MACKEREL_KUBERNETES_KUBELET_HOST valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP MACKEREL_KUBERNETES_KUBELET_HOST MACKEREL_CONTAINER_PLATFORM - name: MACKEREL_CONTAINER_PLATFORM value: kubernetes MACKEREL_KUBERNETES_KUBELET_READ_ONLY_PORT - name: MACKEREL_KUBERNETES_KUBELET_READ_ONLY_PORT value: 10255 DeploymentConfig (spec.template.spec.containers[].env[])
19. - name: MACKEREL_AGENT_CONFIG value: /etc/mackerel/mackerel.yaml MACKEREL_AGENT_CONFIG DeploymentConfig (spec.template.spec.containers[].env[]) DeploymentConfig (spec.template.spec.containers[].env[])
20. volumeMounts: - mountPath: /etc/mackerel/ name: config DeploymentConfig (spec.template.spec.containers[])
21. - name: config secret: defaultMode: 420 secretName: mackerel-config DeploymentConfig (spec.template.spec.volumes[])
22. manifestʹsidecarΛهࡌͯ͠σϓϩΠ • ར఺ • ࣗ༝౓͕ߴ͍ → resource΍image౳ͷࢦఆ͕Ͱ͖Δ • ܽ఺ • okdͷmanifestͷॻ͖ํΛ஌͍ͬͯΔඞཁ͕͋Δ
 (kubernetesͱҰ෦ҟͳΔՕॴ͕ଘࡏ͢Δ)
23. mackerelಋೖͷHOW TO MutatingAdmissionWebhookΛར༻ͨ͠σϓϩΠ
24. MutatingAdmissionWebhookΛར༻ͨ͠σϓϩΠ • Πϝʔδతʹ͸istioʹ͓͚Δenvoyͷ஫ೖͱಉ͡ํ๏ • annotationʹkey/valueΛ౉ͯ͠ɺͦΕΛϑοΫͯ͠ɺ
 mackerel-container-agentΛೖΕΔsidecarͷઃఆΛࠩ͠ࠐΉ • mackrel-pluginɺapikey͸ϢʔβຖʹҟͳΔͷͰɺ
 annotationͰsecret໊Λࢦఆ
25. Secret apikey: "YOUR_MACKEREL_APIKEY" roles: - "mackerelmeetup:database" ignoreContainer: '\Amackerel-container-agent\z' mackerel.yaml Create Secret $ oc create secret generic mackerel-config --from-file=./mackerel.yaml secret/mackerel-config created
26. spec: replicas: 1 template: metadata: annotations: mackerel-sidecar-injector.cycloud.io/inject: "yes" mackerel-sidecar-injector.cycloud.io/secret: "mackerel-conf-secret" labels: app: sleep spec: containers: ɹɹɹ - name: service-container-main MutatingAdmissionWebhookΛར༻ͨ͠σϓϩΠ spec.template.metadata.annotationsʹ߲໨Λ௥Ճ
 mackerel-sidecar-injector.cycloud.io/inject → inject͢Δ৔߹͸yes mackerel-sidecar-injector.cycloud.io/secret → mackrel-container-agentʹ͋ͯΔsecretΛࢦఆ
27. MutatingAdmissionWebhookΛར༻ͨ͠σϓϩΠ • ར఺ • Ϣʔβ͸annotationͱsecretΛ௥Ճ͢Ε͹ྑ͍ • ܽ఺ • sidecarͷઃఆΛॻ͘৔߹ʹൺ΂ࣗ༝౓͸ˣ
 (resourceઃఆ΍Πϝʔδࢦఆ౳)
28. mackerelಋೖͷHOW TO TemplateΛར༻ͨ͠σϓϩΠ
29. Template ? • Pod Templateͱ͸શ͘ҧ͏΋ͷ • TemplateΛఆ͓ٛͯ͘͠ͱɺ
 ࣗ਎ͷϓϩδΣΫτͰར༻Ͱ͖Δݖݶ಺ͰσϓϩΠͳͲΛ
 ઃఆ͢Δ͜ͱ͕Մೳ • Global template libaraly഑Լɺ
 ·ͨ͸projectʹΞοϓϩʔυ͍ͯ͠Δͱը໘͔ΒϙνϙνͰ
 σϓϩΠͳͲΛߦ͑Δ(CLI΋ར༻Մೳ)
30. Template ? DBϢʔβͱ͔ύεϫʔυͱ͔ઃఆ஋ΛೖΕΔ ී௨ͷDBͱͯ͠ར༻Ͱ͖Δ
31. TemplateΛར༻ͨ͠σϓϩΠ ೚ҙͷςϯϓϨʔτΛΫϦοΫ
32. TemplateΛར༻ͨ͠σϓϩΠ ֤ύϥϝʔλʹରͯ͠೚ҙͷઃఆ஋ΛೖΕΔ Add to Project : mackerel-meetup13
 mackerel Api Token : XXXXXXXXXXXXXXXX
 
 mackerel service : mackerelmeetup
 mackerel role : database
33. TemplateΛར༻ͨ͠σϓϩΠ createΛԡ͢
34. TemplateΛར༻ͨ͠σϓϩΠ ग़དྷ্͕Γ·Ͱ଴ͭ
35. TemplateΛར༻ͨ͠σϓϩΠ
36. TemplateΛར༻ͨ͠σϓϩΠ
37. TemplateΛར༻ͨ͠σϓϩΠ
38. Templateઃఆํ๏ apiVersion: template.openshift.io/v1 kind: template labels: messages: metadata: objects: parameters: mackerel-middle-template.yaml
39. Templateઃఆํ๏ apiVersion: template.openshift.io/v1 kind: template labels: messages: metadata: objects: parameters: mackerel-middle-template.yaml
40. Templateઃఆํ๏ messages: ɹɹςϯϓϨʔτͰϢʔβ͕ࢦఆͨ͠஋ΛݟͤͨΓ͢Δ͜ͱ͕Ͱ͖Δ metadata: ɹɹmetadata.annotationͰtemplateͷσΟεϓϨΠ໊΍iconͷࢦఆ͕Ͱ͖Δ (༻ҙ͞Ε͍ͯͳ͍iconʹ͍ͭͯ͸ผ్༻ҙ͢Δඞཁ͋Γ)
41. Templateઃఆํ๏ apiVersion: template.openshift.io/v1 kind: template labels: messages: metadata: objects: parameters: mackerel-middle-template.yaml
42. Templateઃఆํ๏ parameters: ɹɹϢʔβ͕ೖΕΔ஋Λઃఆ͢Δ߲໨ parameters: - description: Mackerel Api Token displayName: mackerel Api Token name: MACKEREL_API_KEY required: true - description: Maximum amount of memory the container can use. displayName: Memory Limit name: MEMORY_LIMIT required: true value: 512Mi - description: Username for MySQL user that will be used for accessing the database. displayName: MySQL Connection Username from: user[A-Z0-9]{3} generate: expression name: MYSQL_USER required: true
43. Templateઃఆํ๏ apiVersion: template.openshift.io/v1 kind: template labels: messages: metadata: objects: parameters: mackerel-middle-template.yaml
44. Templateઃఆํ๏ objects: ɹɹ࣮ࡍʹσϓϩΠ͢ΔmanifestΛهࡌ͢Δ
 ɹɹ(mackerel-container-agent΋sidecarͱͯ͜͜͠ʹهࡌ͞Ε͍ͯΔ) objects: - apiVersion: v1 kind: Secret metadata: annotations: template.openshift.io/expose-apitoken: '{.data[''mackerel-api-key'']}' template.openshift.io/expose-database_name: '{.data[''database-name'']}' template.openshift.io/expose-password: '{.data[''database-password'']}' template.openshift.io/expose-username: '{.data[''database-user'']}' name: ${DATABASE_SERVICE_NAME} stringData: database-name: ${MYSQL_DATABASE} database-password: ${MYSQL_PASSWORD} database-user: ${MYSQL_USER} mackerel-api-key: ${MACKEREL_API_KEY} - apiVersion: v1 kind: DeploymentConfig metadata: annotations: template.alpha.openshift.io/wait-for-ready: "true" name: ${DATABASE_SERVICE_NAME} spec: replicas: 1 selector: name: ${DATABASE_SERVICE_NAME} strategy: type: Recreate template: metadata: labels: name: ${DATABASE_SERVICE_NAME} spec: containers: - name: mackerel-container-agent
45. Templateઃఆํ๏ objects: ɹɹ࣮ࡍʹσϓϩΠ͢ΔmanifestΛهࡌ͢Δ
 ɹɹ(mackerel-container-agent΋sidecarͱͯ͜͜͠ʹهࡌ͞Ε͍ͯΔ) objects: - apiVersion: v1 kind: Secret metadata: annotations: template.openshift.io/expose-apitoken: '{.data[''mackerel-api-key'']}' template.openshift.io/expose-database_name: '{.data[''database-name'']}' template.openshift.io/expose-password: '{.data[''database-password'']}' template.openshift.io/expose-username: '{.data[''database-user'']}' name: ${DATABASE_SERVICE_NAME} stringData: database-name: ${MYSQL_DATABASE} database-password: ${MYSQL_PASSWORD} database-user: ${MYSQL_USER} mackerel-api-key: ${MACKEREL_API_KEY} - apiVersion: v1 kind: DeploymentConfig metadata: annotations: template.alpha.openshift.io/wait-for-ready: "true" name: ${DATABASE_SERVICE_NAME} spec: replicas: 1 selector: name: ${DATABASE_SERVICE_NAME} strategy: type: Recreate template: metadata: labels: name: ${DATABASE_SERVICE_NAME} spec: containers: - name: mackerel-container-agent Secret DeploymentConfig
46. Templateઃఆํ๏ ઃఆ൓ө
 (൓ө·Ͱগ͕͔͔࣌ؒ͠Γ·͢) $ oc create -f ${template-manifest}.yaml -n openshift
47. TemplateΛར༻ͨ͠σϓϩΠ • ར఺ • Ϣʔβ͸σϓϩΠ࣌΋͸΍manifestΛશ͘ॻ͔ͳͯ͘΋ྑ͍
 ྫ) ը໘ϙνϙνͰDBͳͲΛ࡞੒
 ɹ mackerel dashboardʹࣗಈͰొ࿥ • ܽ఺ • ࣗ༝౓͸΄΅̌
48. OpenShiftΛར༻͍ͯ͠Δํ΁ͷ஫ҙࣄ߲ σϑΥͰΠϯετʔϧ͢Δ৔߹
 ReadOnlyPort͕ઃఆ͞Εͳ͍ͷͰɺར༻͢Δ৔߹͸ઃఆ͍ͯͩ͘͠͞
 (/etc/origin/node/node-config.yaml) kubeletArguments: read-only-port: - '10255' bootstrap-kubeconfig: - /etc/origin/node/bootstrap.kubeconfig cert-dir: - /etc/origin/node/certificates cloud-config: - /etc/origin/cloudprovider/openstack.conf
49. ·ͱΊ - mackerel-container-agentͷಋೖ͸γϯϓϧͰָ - ϢʔβϨϕϧ΍໨తʹ͋ͬͨΧελϚΠζ͕ߦ͑Δ - طଘͷplugin͕࢖͑ͯخ͍͠ - plugin͕ೖͬͨίϯςφΠϝʔδΛԿଔެࣜͰ༻ҙ͍͚ͯͨͩ͠Δͱm(_ _)m - MutatingAdmissionWebhookΛ༻ҙ͍͚ͯͨͩ͠Δͱm(_ _)m
50. Thank you for listening.