Pro Yearly is on sale from $80 to $50! »

mackerel-container-agentを用いたマルチテナントKubernetesのモニタリング

 mackerel-container-agentを用いたマルチテナントKubernetesのモニタリング

社内向けコンテナ基盤として構築中のマルチテナントKubernetesに対して、mackerel-container-agentを導入した際の知見の共有。

F4b273c6b954b244a55f3e9370bbb3a6?s=128

hidenobu goto

March 01, 2019
Tweet

Transcript

  1. mackerel-container-agentΛ༻͍ͨ ϚϧνςφϯτKubernetesͷ ϞχλϦϯά

  2. ൃද಺༰ • ࣗݾ঺հ • ͜Ε·Ͱͷ؂ࢹͱ͜Ε͔Βͷ؂ࢹ • Ϛϧνςφϯτkubernetesͷߏ੒ʹ͍ͭͯ • σϓϩΠํ๏ͷ঺հ SideCarͰར༻͢Δ৔߹

    MutatingAdmissionWebhookΛར༻ͨ͠৔߹ ServiceCatalog (TSB)Λར༻ͨ͠৔߹ • ·ͱΊ
  3. ࣗݾ঺հ • ໊લɿޙ౻ ल৴ • ॴଐɿٕज़ຊ෦ PrivateCloud ։ൃνʔϜ • ग़਎ɿେ෼

    • ܦྺɿ - ( 2014 / 04 ~ ) SIerʹ৽ଔೖࣾ - ( 2016 / 09 ~ ) ݱࡏͷձࣾʹத్ೖࣾ • ۀ຿ɿPrivateCloudʹ͍ͭͯॾʑ
  4. ͜Ε·Ͱͷ؂ࢹͱ͜Ε͔Βͷ؂ࢹ αʔϏεͷ଍ճΓ 
 ݹ͘͸શͯ෺ཧαʔόͷ্ʹ৐͍ͬͯͨ 2013೥ࠒ͔ΒϓϥΠϕʔτΫϥ΢υͷఏڙ͢ΔԾ૝αʔό΁Ҡߦ
 ਺೥લ͔ΒɺAWS΍GCPͷ༷ͳύϒϦοΫΫϥ΢υ΁ͷҠߦ΋ਐΜͰ͍Δ ࠷ۙʹͳͬͯɺαʔϏεͷίϯςφԽ͕ਐΉΑ͏ʹͳ͖ͬͯͨ ෺ཧαʔό ίϯςφ Ծ૝αʔό


    (OpenStack) Ծ૝αʔό
 (಺੡Ϋϥ΢υ)
  5. ͜Ε·Ͱͷ؂ࢹͱ͜Ε͔Βͷ؂ࢹ αʔϏεͷ؂ࢹ
 ݹ͘͸monɺNagiosɺMunin গ͠ਐΉͱZabbixɺSensu͕ൺֱతΑ͘࢖ΘΕ͍ͯͨ ͦͷޙɺࣗલͰ࣋ͭͷ͸ਏ͍ͱ͍͏ྲྀΕ͔Β MackerelɺDatadogʹ୅ද͞ΕΔSaaS΁ͷҠߦ͕࠷ۙͷྲྀΕ ෺ཧαʔό ίϯςφ Ծ૝αʔό
 (OpenStack)

    Ծ૝αʔό
 (಺੡Ϋϥ΢υ) mon
 Nagios
 Munin ??? Mackerel
 Datadog Zabbix
 Sensu
  6. ͜Ε·Ͱͷ؂ࢹͱ͜Ε͔Βͷ؂ࢹ ৽͍͠σʔληϯλ
 2019೥3݄ʹΦʔϓϯ OpenStackΛ༻͍ͨԾ૝Խج൫ʹՃ͑ͯ
 ϚϧνςφϯτͷϚωʔδυKubernetesΛఏڙ Ծ૝Ϛγϯͷ؂ࢹαʔϏεͱͯ͠MackerelΛ࠾༻ ෺ཧαʔό ίϯςφ Ծ૝αʔό
 (OpenStack)

    Ծ૝αʔό
 (಺੡Ϋϥ΢υ) mon
 Nagios
 Munin ??? Mackerel
 Datadog Zabbix
 Sensu
  7. ͜Ε·Ͱͷ؂ࢹͱ͜Ε͔Βͷ؂ࢹ ίϯςφ؀ڥͷ؂ࢹ
 ίϯςφ؀ڥͷ؂ࢹαʔϏεͱͯ͠MackerelΛ࠾༻ ࠾༻ཧ༝ 1. Ծ૝αʔόͱίϯςφΛಉ͡πʔϧͰ؂ࢹ͢Δ͜ͱ͕Մೳ 2. طଘͷϓϥάΠϯͷྲྀ༻͕Մೳ ෺ཧαʔό ίϯςφ

    Ծ૝αʔό
 (OpenStack) Ծ૝αʔό
 (಺੡Ϋϥ΢υ) mon
 Nagios
 Munin Mackerel Container Agent Mackerel
 Datadog Zabbix
 Sensu
  8. Ϛϧνςφϯτkubernetesͷߏ੒ • Kubernetes (OpenShift) on OpenStack on Kubernetes Pod VM

    Pod Pod
  9. Ϛϧνςφϯτkubernetesͷߏ੒ • Kubernetes (OpenShift) on OpenStack on Kubernetes Pod VM

    Pod Pod ຊ೔ͷൃදείʔϓ͸
 ίί
  10. Ϛϧνςφϯτkubernetesͷߏ੒ → OKDͷུ֓ਤ • ུ֓ Master × 5 Etcd ×

    5 Route × 10 Node × ͍ͬͺ͍ Master Route(HAproxy) okd-node001(8core/48GB) Master Master Master okd-node002 Route(HAproxy) Etcd Etcd Etcd Etcd Etcd Prj-2
 pod1 Prj-1
 pod1 Prj-1
 pod2 Prj-3
 pod1 Prj-2
 pod2 Prj-4
 pod1
  11. mackerelಋೖͷHOW TO okd-node001(8core/48GB) Prj-2
 pod1 Prj-1
 pod1 Prj-1
 pod2 ͜͜ʹͲ͏΍ͬͯmackrel-container-agentΛ


    ಋೖ͢Δʁʁ
  12. mackerelಋೖͷHOW TO • manifestʹsidecarΛهࡌͯ͠ߦ͏σϓϩΠ • ७ਮͳํ๏ • MutatingAdmissionWebhookΛར༻ͨ͠σϓϩΠ • istioͰͷenvoyͷinjectionͱಉ͡Α͏ͳํ๏

    • TemplateΛར༻ͨ͠σϓϩΠ • OpenShiftͷtemplateػೳΛར༻ͨ͠ํ๏
  13. mackerelಋೖͷHOW TO manifestʹsidecarΛهࡌͯ͠ߦ͏σϓϩΠ

  14. manifestʹsidecarΛهࡌͯ͠σϓϩΠ • ಉҰPod಺ʹίϯςφΛ૬৐Γͤ͞Δํ๏ • kubernetesͷsidecarΛར༻ͨ͠σϓϩΠͰ͕͢
 ΋ͪΖΜOpenShiftͰ΋࢖͑·͢

  15. manifestʹsidecarΛهࡌͯ͠σϓϩΠ containers: - name: service-container-main ɹɾ ɹɾ ɹɾ - name:

    mackerel-container-agent spec.template.spec.containers
  16. Secret apikey: "YOUR_MACKEREL_APIKEY" roles: - "mackerelmeetup:database" ignoreContainer: '\Amackerel-container-agent\z' mackerel.yaml Create

    Secret $ oc create secret generic mackerel-config --from-file=./mackerel.yaml secret/mackerel-config created apikeyΛsecretʹͯ͠ઃఆྨ͸configmapͰ͋ͯΔ͜ͱ΋Ͱ͖Δ͕ɺ secretͱconfigmapΛ؅ཧ͢Δͷ͸໘౗ͳͨΊɺઃఆϑΝΠϧΛؙʑSecretʹͨ͠
  17. DeploymentConfig (spec.template.spec.containers[].env[]) - name: MACKEREL_KUBERNETES_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath:

    metadata.namespace MACKEREL_KUBERNETES_NAMESPACE MACKEREL_KUBERNETES_POD_NAME - name: MACKEREL_KUBERNETES_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name
  18. - name: MACKEREL_KUBERNETES_KUBELET_HOST valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP MACKEREL_KUBERNETES_KUBELET_HOST

    MACKEREL_CONTAINER_PLATFORM - name: MACKEREL_CONTAINER_PLATFORM value: kubernetes MACKEREL_KUBERNETES_KUBELET_READ_ONLY_PORT - name: MACKEREL_KUBERNETES_KUBELET_READ_ONLY_PORT value: 10255 DeploymentConfig (spec.template.spec.containers[].env[])
  19. - name: MACKEREL_AGENT_CONFIG value: /etc/mackerel/mackerel.yaml MACKEREL_AGENT_CONFIG DeploymentConfig (spec.template.spec.containers[].env[]) DeploymentConfig (spec.template.spec.containers[].env[])

  20. volumeMounts: - mountPath: /etc/mackerel/ name: config DeploymentConfig (spec.template.spec.containers[])

  21. - name: config secret: defaultMode: 420 secretName: mackerel-config DeploymentConfig (spec.template.spec.volumes[])

  22. manifestʹsidecarΛهࡌͯ͠σϓϩΠ • ར఺ • ࣗ༝౓͕ߴ͍ → resource΍image౳ͷࢦఆ͕Ͱ͖Δ • ܽ఺ •

    okdͷmanifestͷॻ͖ํΛ஌͍ͬͯΔඞཁ͕͋Δ
 (kubernetesͱҰ෦ҟͳΔՕॴ͕ଘࡏ͢Δ)
  23. mackerelಋೖͷHOW TO MutatingAdmissionWebhookΛར༻ͨ͠σϓϩΠ

  24. MutatingAdmissionWebhookΛར༻ͨ͠σϓϩΠ • Πϝʔδతʹ͸istioʹ͓͚Δenvoyͷ஫ೖͱಉ͡ํ๏ • annotationʹkey/valueΛ౉ͯ͠ɺͦΕΛϑοΫͯ͠ɺ
 mackerel-container-agentΛೖΕΔsidecarͷઃఆΛࠩ͠ࠐΉ • mackrel-pluginɺapikey͸ϢʔβຖʹҟͳΔͷͰɺ
 annotationͰsecret໊Λࢦఆ

  25. Secret apikey: "YOUR_MACKEREL_APIKEY" roles: - "mackerelmeetup:database" ignoreContainer: '\Amackerel-container-agent\z' mackerel.yaml Create

    Secret $ oc create secret generic mackerel-config --from-file=./mackerel.yaml secret/mackerel-config created
  26. spec: replicas: 1 template: metadata: annotations: mackerel-sidecar-injector.cycloud.io/inject: "yes" mackerel-sidecar-injector.cycloud.io/secret: "mackerel-conf-secret"

    labels: app: sleep spec: containers: ɹɹɹ - name: service-container-main MutatingAdmissionWebhookΛར༻ͨ͠σϓϩΠ spec.template.metadata.annotationsʹ߲໨Λ௥Ճ
 mackerel-sidecar-injector.cycloud.io/inject → inject͢Δ৔߹͸yes mackerel-sidecar-injector.cycloud.io/secret → mackrel-container-agentʹ͋ͯΔsecretΛࢦఆ
  27. MutatingAdmissionWebhookΛར༻ͨ͠σϓϩΠ • ར఺ • Ϣʔβ͸annotationͱsecretΛ௥Ճ͢Ε͹ྑ͍ • ܽ఺ • sidecarͷઃఆΛॻ͘৔߹ʹൺ΂ࣗ༝౓͸ˣ
 (resourceઃఆ΍Πϝʔδࢦఆ౳)

  28. mackerelಋೖͷHOW TO TemplateΛར༻ͨ͠σϓϩΠ

  29. Template ? • Pod Templateͱ͸શ͘ҧ͏΋ͷ • TemplateΛఆ͓ٛͯ͘͠ͱɺ
 ࣗ਎ͷϓϩδΣΫτͰར༻Ͱ͖Δݖݶ಺ͰσϓϩΠͳͲΛ
 ઃఆ͢Δ͜ͱ͕Մೳ •

    Global template libaraly഑Լɺ
 ·ͨ͸projectʹΞοϓϩʔυ͍ͯ͠Δͱը໘͔ΒϙνϙνͰ
 σϓϩΠͳͲΛߦ͑Δ(CLI΋ར༻Մೳ)
  30. Template ? DBϢʔβͱ͔ύεϫʔυͱ͔ઃఆ஋ΛೖΕΔ ී௨ͷDBͱͯ͠ར༻Ͱ͖Δ

  31. TemplateΛར༻ͨ͠σϓϩΠ ೚ҙͷςϯϓϨʔτΛΫϦοΫ

  32. TemplateΛར༻ͨ͠σϓϩΠ ֤ύϥϝʔλʹରͯ͠೚ҙͷઃఆ஋ΛೖΕΔ Add to Project : mackerel-meetup13
 mackerel Api Token

    : XXXXXXXXXXXXXXXX
 
 mackerel service : mackerelmeetup
 mackerel role : database
  33. TemplateΛར༻ͨ͠σϓϩΠ createΛԡ͢

  34. TemplateΛར༻ͨ͠σϓϩΠ ग़དྷ্͕Γ·Ͱ଴ͭ

  35. TemplateΛར༻ͨ͠σϓϩΠ

  36. TemplateΛར༻ͨ͠σϓϩΠ

  37. TemplateΛར༻ͨ͠σϓϩΠ

  38. Templateઃఆํ๏ apiVersion: template.openshift.io/v1 kind: template labels: messages: metadata: objects: parameters:

    mackerel-middle-template.yaml
  39. Templateઃఆํ๏ apiVersion: template.openshift.io/v1 kind: template labels: messages: metadata: objects: parameters:

    mackerel-middle-template.yaml
  40. Templateઃఆํ๏ messages: ɹɹςϯϓϨʔτͰϢʔβ͕ࢦఆͨ͠஋ΛݟͤͨΓ͢Δ͜ͱ͕Ͱ͖Δ metadata: ɹɹmetadata.annotationͰtemplateͷσΟεϓϨΠ໊΍iconͷࢦఆ͕Ͱ͖Δ (༻ҙ͞Ε͍ͯͳ͍iconʹ͍ͭͯ͸ผ్༻ҙ͢Δඞཁ͋Γ)

  41. Templateઃఆํ๏ apiVersion: template.openshift.io/v1 kind: template labels: messages: metadata: objects: parameters:

    mackerel-middle-template.yaml
  42. Templateઃఆํ๏ parameters: ɹɹϢʔβ͕ೖΕΔ஋Λઃఆ͢Δ߲໨ parameters: - description: Mackerel Api Token displayName:

    mackerel Api Token name: MACKEREL_API_KEY required: true - description: Maximum amount of memory the container can use. displayName: Memory Limit name: MEMORY_LIMIT required: true value: 512Mi - description: Username for MySQL user that will be used for accessing the database. displayName: MySQL Connection Username from: user[A-Z0-9]{3} generate: expression name: MYSQL_USER required: true
  43. Templateઃఆํ๏ apiVersion: template.openshift.io/v1 kind: template labels: messages: metadata: objects: parameters:

    mackerel-middle-template.yaml
  44. Templateઃఆํ๏ objects: ɹɹ࣮ࡍʹσϓϩΠ͢ΔmanifestΛهࡌ͢Δ
 ɹɹ(mackerel-container-agent΋sidecarͱͯ͜͜͠ʹهࡌ͞Ε͍ͯΔ) objects: - apiVersion: v1 kind: Secret

    metadata: annotations: template.openshift.io/expose-apitoken: '{.data[''mackerel-api-key'']}' template.openshift.io/expose-database_name: '{.data[''database-name'']}' template.openshift.io/expose-password: '{.data[''database-password'']}' template.openshift.io/expose-username: '{.data[''database-user'']}' name: ${DATABASE_SERVICE_NAME} stringData: database-name: ${MYSQL_DATABASE} database-password: ${MYSQL_PASSWORD} database-user: ${MYSQL_USER} mackerel-api-key: ${MACKEREL_API_KEY} - apiVersion: v1 kind: DeploymentConfig metadata: annotations: template.alpha.openshift.io/wait-for-ready: "true" name: ${DATABASE_SERVICE_NAME} spec: replicas: 1 selector: name: ${DATABASE_SERVICE_NAME} strategy: type: Recreate template: metadata: labels: name: ${DATABASE_SERVICE_NAME} spec: containers: - name: mackerel-container-agent
  45. Templateઃఆํ๏ objects: ɹɹ࣮ࡍʹσϓϩΠ͢ΔmanifestΛهࡌ͢Δ
 ɹɹ(mackerel-container-agent΋sidecarͱͯ͜͜͠ʹهࡌ͞Ε͍ͯΔ) objects: - apiVersion: v1 kind: Secret

    metadata: annotations: template.openshift.io/expose-apitoken: '{.data[''mackerel-api-key'']}' template.openshift.io/expose-database_name: '{.data[''database-name'']}' template.openshift.io/expose-password: '{.data[''database-password'']}' template.openshift.io/expose-username: '{.data[''database-user'']}' name: ${DATABASE_SERVICE_NAME} stringData: database-name: ${MYSQL_DATABASE} database-password: ${MYSQL_PASSWORD} database-user: ${MYSQL_USER} mackerel-api-key: ${MACKEREL_API_KEY} - apiVersion: v1 kind: DeploymentConfig metadata: annotations: template.alpha.openshift.io/wait-for-ready: "true" name: ${DATABASE_SERVICE_NAME} spec: replicas: 1 selector: name: ${DATABASE_SERVICE_NAME} strategy: type: Recreate template: metadata: labels: name: ${DATABASE_SERVICE_NAME} spec: containers: - name: mackerel-container-agent Secret DeploymentConfig
  46. Templateઃఆํ๏ ઃఆ൓ө
 (൓ө·Ͱগ͕͔͔࣌ؒ͠Γ·͢) $ oc create -f ${template-manifest}.yaml -n openshift

  47. TemplateΛར༻ͨ͠σϓϩΠ • ར఺ • Ϣʔβ͸σϓϩΠ࣌΋͸΍manifestΛશ͘ॻ͔ͳͯ͘΋ྑ͍
 ྫ) ը໘ϙνϙνͰDBͳͲΛ࡞੒
 ɹ mackerel dashboardʹࣗಈͰొ࿥

    • ܽ఺ • ࣗ༝౓͸΄΅̌
  48. OpenShiftΛར༻͍ͯ͠Δํ΁ͷ஫ҙࣄ߲ σϑΥͰΠϯετʔϧ͢Δ৔߹
 ReadOnlyPort͕ઃఆ͞Εͳ͍ͷͰɺར༻͢Δ৔߹͸ઃఆ͍ͯͩ͘͠͞
 (/etc/origin/node/node-config.yaml) kubeletArguments: read-only-port: - '10255' bootstrap-kubeconfig: -

    /etc/origin/node/bootstrap.kubeconfig cert-dir: - /etc/origin/node/certificates cloud-config: - /etc/origin/cloudprovider/openstack.conf
  49. ·ͱΊ - mackerel-container-agentͷಋೖ͸γϯϓϧͰָ - ϢʔβϨϕϧ΍໨తʹ͋ͬͨΧελϚΠζ͕ߦ͑Δ - طଘͷplugin͕࢖͑ͯخ͍͠ - plugin͕ೖͬͨίϯςφΠϝʔδΛԿଔެࣜͰ༻ҙ͍͚ͯͨͩ͠Δͱm(_ _)m

    - MutatingAdmissionWebhookΛ༻ҙ͍͚ͯͨͩ͠Δͱm(_ _)m
  50. Thank you for listening.