Share
Embed
mackerel-container-agentを用いたマルチテナントKubernetesのモニタリング
社内向けコンテナ基盤として構築中のマルチテナントKubernetesに対して、mackerel-container-agentを導入した際の知見の共有。
hidenobu goto
March 01, 2019
Transcript
50 slides
-
ൃද༰ • ࣗݾհ • ͜Ε·Ͱͷࢹͱ͜Ε͔Βͷࢹ • Ϛϧνςφϯτkubernetesͷߏʹ͍ͭͯ • σϓϩΠํ๏ͷհ SideCarͰར༻͢Δ߹
MutatingAdmissionWebhookΛར༻ͨ͠߹ ServiceCatalog (TSB)Λར༻ͨ͠߹ • ·ͱΊ -
ࣗݾհ • ໊લɿޙ౻ ल৴ • ॴଐɿٕज़ຊ෦ PrivateCloud ։ൃνʔϜ • ग़ɿେ
• ܦྺɿ - ( 2014 / 04 ~ ) SIerʹ৽ଔೖࣾ - ( 2016 / 09 ~ ) ݱࡏͷձࣾʹத్ೖࣾ • ۀɿPrivateCloudʹ͍ͭͯॾʑ -
͜Ε·Ͱͷࢹͱ͜Ε͔Βͷࢹ αʔϏεͷࢹ ݹ͘monɺNagiosɺMunin গ͠ਐΉͱZabbixɺSensu͕ൺֱతΑ͘ΘΕ͍ͯͨ ͦͷޙɺࣗલͰ࣋ͭͷਏ͍ͱ͍͏ྲྀΕ͔Β MackerelɺDatadogʹද͞ΕΔSaaSͷҠߦ͕࠷ۙͷྲྀΕ ཧαʔό ίϯςφ Ծαʔό (OpenStack)
Ծαʔό (Ϋϥυ) mon Nagios Munin ??? Mackerel Datadog Zabbix Sensu -
͜Ε·Ͱͷࢹͱ͜Ε͔Βͷࢹ ৽͍͠σʔληϯλ 20193݄ʹΦʔϓϯ OpenStackΛ༻͍ͨԾԽج൫ʹՃ͑ͯ ϚϧνςφϯτͷϚωʔδυKubernetesΛఏڙ ԾϚγϯͷࢹαʔϏεͱͯ͠MackerelΛ࠾༻ ཧαʔό ίϯςφ Ծαʔό (OpenStack)
Ծαʔό (Ϋϥυ) mon Nagios Munin ??? Mackerel Datadog Zabbix Sensu -
͜Ε·Ͱͷࢹͱ͜Ε͔Βͷࢹ ίϯςφڥͷࢹ ίϯςφڥͷࢹαʔϏεͱͯ͠MackerelΛ࠾༻ ࠾༻ཧ༝ 1. ԾαʔόͱίϯςφΛಉ͡πʔϧͰࢹ͢Δ͜ͱ͕Մೳ 2. طଘͷϓϥάΠϯͷྲྀ༻͕Մೳ ཧαʔό ίϯςφ
Ծαʔό (OpenStack) Ծαʔό (Ϋϥυ) mon Nagios Munin Mackerel Container Agent Mackerel Datadog Zabbix Sensu -
Ϛϧνςφϯτkubernetesͷߏ • Kubernetes (OpenShift) on OpenStack on Kubernetes Pod VM
Pod Pod ຊͷൃදείʔϓ ίί -
Ϛϧνςφϯτkubernetesͷߏ → OKDͷུ֓ਤ • ུ֓ Master × 5 Etcd ×
5 Route × 10 Node × ͍ͬͺ͍ Master Route(HAproxy) okd-node001(8core/48GB) Master Master Master okd-node002 Route(HAproxy) Etcd Etcd Etcd Etcd Etcd Prj-2 pod1 Prj-1 pod1 Prj-1 pod2 Prj-3 pod1 Prj-2 pod2 Prj-4 pod1 -
mackerelಋೖͷHOW TO • manifestʹsidecarΛهࡌͯ͠ߦ͏σϓϩΠ • ७ਮͳํ๏ • MutatingAdmissionWebhookΛར༻ͨ͠σϓϩΠ • istioͰͷenvoyͷinjectionͱಉ͡Α͏ͳํ๏
• TemplateΛར༻ͨ͠σϓϩΠ • OpenShiftͷtemplateػೳΛར༻ͨ͠ํ๏ -
manifestʹsidecarΛهࡌͯ͠σϓϩΠ containers: - name: service-container-main ɹɾ ɹɾ ɹɾ - name:
mackerel-container-agent spec.template.spec.containers -
Secret apikey: "YOUR_MACKEREL_APIKEY" roles: - "mackerelmeetup:database" ignoreContainer: '\Amackerel-container-agent\z' mackerel.yaml Create
Secret $ oc create secret generic mackerel-config --from-file=./mackerel.yaml secret/mackerel-config created apikeyΛsecretʹͯ͠ઃఆྨconfigmapͰ͋ͯΔ͜ͱͰ͖Δ͕ɺ secretͱconfigmapΛཧ͢Δͷ໘ͳͨΊɺઃఆϑΝΠϧΛؙʑSecretʹͨ͠ -
DeploymentConfig (spec.template.spec.containers[].env[]) - name: MACKEREL_KUBERNETES_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath:
metadata.namespace MACKEREL_KUBERNETES_NAMESPACE MACKEREL_KUBERNETES_POD_NAME - name: MACKEREL_KUBERNETES_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name -
- name: MACKEREL_KUBERNETES_KUBELET_HOST valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP MACKEREL_KUBERNETES_KUBELET_HOST
MACKEREL_CONTAINER_PLATFORM - name: MACKEREL_CONTAINER_PLATFORM value: kubernetes MACKEREL_KUBERNETES_KUBELET_READ_ONLY_PORT - name: MACKEREL_KUBERNETES_KUBELET_READ_ONLY_PORT value: 10255 DeploymentConfig (spec.template.spec.containers[].env[]) -
manifestʹsidecarΛهࡌͯ͠σϓϩΠ • ར • ࣗ༝͕ߴ͍ → resourceimageͷࢦఆ͕Ͱ͖Δ • ܽ •
okdͷmanifestͷॻ͖ํΛ͍ͬͯΔඞཁ͕͋Δ (kubernetesͱҰ෦ҟͳΔՕॴ͕ଘࡏ͢Δ) -
Secret apikey: "YOUR_MACKEREL_APIKEY" roles: - "mackerelmeetup:database" ignoreContainer: '\Amackerel-container-agent\z' mackerel.yaml Create
Secret $ oc create secret generic mackerel-config --from-file=./mackerel.yaml secret/mackerel-config created -
spec: replicas: 1 template: metadata: annotations: mackerel-sidecar-injector.cycloud.io/inject: "yes" mackerel-sidecar-injector.cycloud.io/secret: "mackerel-conf-secret"
labels: app: sleep spec: containers: ɹɹɹ - name: service-container-main MutatingAdmissionWebhookΛར༻ͨ͠σϓϩΠ spec.template.metadata.annotationsʹ߲ΛՃ mackerel-sidecar-injector.cycloud.io/inject → inject͢Δ߹yes mackerel-sidecar-injector.cycloud.io/secret → mackrel-container-agentʹ͋ͯΔsecretΛࢦఆ -
Template ? • Pod Templateͱશ͘ҧ͏ͷ • TemplateΛఆ͓ٛͯ͘͠ͱɺ ࣗͷϓϩδΣΫτͰར༻Ͱ͖ΔݖݶͰσϓϩΠͳͲΛ ઃఆ͢Δ͜ͱ͕Մೳ •
Global template libaralyԼɺ ·ͨprojectʹΞοϓϩʔυ͍ͯ͠Δͱը໘͔ΒϙνϙνͰ σϓϩΠͳͲΛߦ͑Δ(CLIར༻Մೳ) -
TemplateΛར༻ͨ͠σϓϩΠ ֤ύϥϝʔλʹରͯ͠ҙͷઃఆΛೖΕΔ Add to Project : mackerel-meetup13 mackerel Api Token
: XXXXXXXXXXXXXXXX mackerel service : mackerelmeetup mackerel role : database -
Templateઃఆํ๏ apiVersion: template.openshift.io/v1 kind: template labels: messages: metadata: objects: parameters:
mackerel-middle-template.yaml -
Templateઃఆํ๏ apiVersion: template.openshift.io/v1 kind: template labels: messages: metadata: objects: parameters:
mackerel-middle-template.yaml -
Templateઃఆํ๏ apiVersion: template.openshift.io/v1 kind: template labels: messages: metadata: objects: parameters:
mackerel-middle-template.yaml -
Templateઃఆํ๏ parameters: ɹɹϢʔβ͕ೖΕΔΛઃఆ͢Δ߲ parameters: - description: Mackerel Api Token displayName:
mackerel Api Token name: MACKEREL_API_KEY required: true - description: Maximum amount of memory the container can use. displayName: Memory Limit name: MEMORY_LIMIT required: true value: 512Mi - description: Username for MySQL user that will be used for accessing the database. displayName: MySQL Connection Username from: user[A-Z0-9]{3} generate: expression name: MYSQL_USER required: true -
Templateઃఆํ๏ apiVersion: template.openshift.io/v1 kind: template labels: messages: metadata: objects: parameters:
mackerel-middle-template.yaml -
Templateઃఆํ๏ objects: ɹɹ࣮ࡍʹσϓϩΠ͢ΔmanifestΛهࡌ͢Δ ɹɹ(mackerel-container-agentsidecarͱͯ͜͜͠ʹهࡌ͞Ε͍ͯΔ) objects: - apiVersion: v1 kind: Secret
metadata: annotations: template.openshift.io/expose-apitoken: '{.data[''mackerel-api-key'']}' template.openshift.io/expose-database_name: '{.data[''database-name'']}' template.openshift.io/expose-password: '{.data[''database-password'']}' template.openshift.io/expose-username: '{.data[''database-user'']}' name: ${DATABASE_SERVICE_NAME} stringData: database-name: ${MYSQL_DATABASE} database-password: ${MYSQL_PASSWORD} database-user: ${MYSQL_USER} mackerel-api-key: ${MACKEREL_API_KEY} - apiVersion: v1 kind: DeploymentConfig metadata: annotations: template.alpha.openshift.io/wait-for-ready: "true" name: ${DATABASE_SERVICE_NAME} spec: replicas: 1 selector: name: ${DATABASE_SERVICE_NAME} strategy: type: Recreate template: metadata: labels: name: ${DATABASE_SERVICE_NAME} spec: containers: - name: mackerel-container-agent -
Templateઃఆํ๏ objects: ɹɹ࣮ࡍʹσϓϩΠ͢ΔmanifestΛهࡌ͢Δ ɹɹ(mackerel-container-agentsidecarͱͯ͜͜͠ʹهࡌ͞Ε͍ͯΔ) objects: - apiVersion: v1 kind: Secret
metadata: annotations: template.openshift.io/expose-apitoken: '{.data[''mackerel-api-key'']}' template.openshift.io/expose-database_name: '{.data[''database-name'']}' template.openshift.io/expose-password: '{.data[''database-password'']}' template.openshift.io/expose-username: '{.data[''database-user'']}' name: ${DATABASE_SERVICE_NAME} stringData: database-name: ${MYSQL_DATABASE} database-password: ${MYSQL_PASSWORD} database-user: ${MYSQL_USER} mackerel-api-key: ${MACKEREL_API_KEY} - apiVersion: v1 kind: DeploymentConfig metadata: annotations: template.alpha.openshift.io/wait-for-ready: "true" name: ${DATABASE_SERVICE_NAME} spec: replicas: 1 selector: name: ${DATABASE_SERVICE_NAME} strategy: type: Recreate template: metadata: labels: name: ${DATABASE_SERVICE_NAME} spec: containers: - name: mackerel-container-agent Secret DeploymentConfig -
OpenShiftΛར༻͍ͯ͠Δํͷҙࣄ߲ σϑΥͰΠϯετʔϧ͢Δ߹ ReadOnlyPort͕ઃఆ͞Εͳ͍ͷͰɺར༻͢Δ߹ઃఆ͍ͯͩ͘͠͞ (/etc/origin/node/node-config.yaml) kubeletArguments: read-only-port: - '10255' bootstrap-kubeconfig: -
/etc/origin/node/bootstrap.kubeconfig cert-dir: - /etc/origin/node/certificates cloud-config: - /etc/origin/cloudprovider/openstack.conf -
·ͱΊ - mackerel-container-agentͷಋೖγϯϓϧͰָ - ϢʔβϨϕϧతʹ͋ͬͨΧελϚΠζ͕ߦ͑Δ - طଘͷplugin͕͑ͯخ͍͠ - plugin͕ೖͬͨίϯςφΠϝʔδΛԿଔެࣜͰ༻ҙ͍͚ͯͨͩ͠Δͱm(_ _)m
- MutatingAdmissionWebhookΛ༻ҙ͍͚ͯͨͩ͠Δͱm(_ _)m