Upgrade to Pro — share decks privately, control downloads, hide ads and more …

mackerel-container-agentを用いたマルチテナントKubernetesのモニタリング

 mackerel-container-agentを用いたマルチテナントKubernetesのモニタリング

社内向けコンテナ基盤として構築中のマルチテナントKubernetesに対して、mackerel-container-agentを導入した際の知見の共有。

hidenobu goto

March 01, 2019
Tweet

More Decks by hidenobu goto

Other Decks in Technology

Transcript

  1. mackerel-container-agentΛ༻͍ͨ
    ϚϧνςφϯτKubernetesͷ
    ϞχλϦϯά

    View full-size slide

  2. ൃද಺༰
    • ࣗݾ঺հ
    • ͜Ε·Ͱͷ؂ࢹͱ͜Ε͔Βͷ؂ࢹ
    • Ϛϧνςφϯτkubernetesͷߏ੒ʹ͍ͭͯ
    • σϓϩΠํ๏ͷ঺հ
    SideCarͰར༻͢Δ৔߹
    MutatingAdmissionWebhookΛར༻ͨ͠৔߹
    ServiceCatalog (TSB)Λར༻ͨ͠৔߹
    • ·ͱΊ

    View full-size slide

  3. ࣗݾ঺հ
    • ໊લɿޙ౻ ल৴
    • ॴଐɿٕज़ຊ෦ PrivateCloud ։ൃνʔϜ
    • ग़਎ɿେ෼
    • ܦྺɿ
    - ( 2014 / 04 ~ ) SIerʹ৽ଔೖࣾ
    - ( 2016 / 09 ~ ) ݱࡏͷձࣾʹத్ೖࣾ
    • ۀ຿ɿPrivateCloudʹ͍ͭͯॾʑ

    View full-size slide

  4. ͜Ε·Ͱͷ؂ࢹͱ͜Ε͔Βͷ؂ࢹ
    αʔϏεͷ଍ճΓ

    ݹ͘͸શͯ෺ཧαʔόͷ্ʹ৐͍ͬͯͨ
    2013೥ࠒ͔ΒϓϥΠϕʔτΫϥ΢υͷఏڙ͢ΔԾ૝αʔό΁Ҡߦ

    ਺೥લ͔ΒɺAWS΍GCPͷ༷ͳύϒϦοΫΫϥ΢υ΁ͷҠߦ΋ਐΜͰ͍Δ
    ࠷ۙʹͳͬͯɺαʔϏεͷίϯςφԽ͕ਐΉΑ͏ʹͳ͖ͬͯͨ
    ෺ཧαʔό ίϯςφ
    Ծ૝αʔό

    (OpenStack)
    Ծ૝αʔό

    (಺੡Ϋϥ΢υ)

    View full-size slide

  5. ͜Ε·Ͱͷ؂ࢹͱ͜Ε͔Βͷ؂ࢹ
    αʔϏεͷ؂ࢹ

    ݹ͘͸monɺNagiosɺMunin
    গ͠ਐΉͱZabbixɺSensu͕ൺֱతΑ͘࢖ΘΕ͍ͯͨ
    ͦͷޙɺࣗલͰ࣋ͭͷ͸ਏ͍ͱ͍͏ྲྀΕ͔Β
    MackerelɺDatadogʹ୅ද͞ΕΔSaaS΁ͷҠߦ͕࠷ۙͷྲྀΕ
    ෺ཧαʔό ίϯςφ
    Ծ૝αʔό

    (OpenStack)
    Ծ૝αʔό

    (಺੡Ϋϥ΢υ)
    mon

    Nagios

    Munin
    ???
    Mackerel

    Datadog
    Zabbix

    Sensu

    View full-size slide

  6. ͜Ε·Ͱͷ؂ࢹͱ͜Ε͔Βͷ؂ࢹ
    ৽͍͠σʔληϯλ

    2019೥3݄ʹΦʔϓϯ
    OpenStackΛ༻͍ͨԾ૝Խج൫ʹՃ͑ͯ

    ϚϧνςφϯτͷϚωʔδυKubernetesΛఏڙ
    Ծ૝Ϛγϯͷ؂ࢹαʔϏεͱͯ͠MackerelΛ࠾༻
    ෺ཧαʔό ίϯςφ
    Ծ૝αʔό

    (OpenStack)
    Ծ૝αʔό

    (಺੡Ϋϥ΢υ)
    mon

    Nagios

    Munin
    ???
    Mackerel

    Datadog
    Zabbix

    Sensu

    View full-size slide

  7. ͜Ε·Ͱͷ؂ࢹͱ͜Ε͔Βͷ؂ࢹ
    ίϯςφ؀ڥͷ؂ࢹ

    ίϯςφ؀ڥͷ؂ࢹαʔϏεͱͯ͠MackerelΛ࠾༻
    ࠾༻ཧ༝
    1. Ծ૝αʔόͱίϯςφΛಉ͡πʔϧͰ؂ࢹ͢Δ͜ͱ͕Մೳ
    2. طଘͷϓϥάΠϯͷྲྀ༻͕Մೳ
    ෺ཧαʔό ίϯςφ
    Ծ૝αʔό

    (OpenStack)
    Ծ૝αʔό

    (಺੡Ϋϥ΢υ)
    mon

    Nagios

    Munin
    Mackerel
    Container Agent
    Mackerel

    Datadog
    Zabbix

    Sensu

    View full-size slide

  8. Ϛϧνςφϯτkubernetesͷߏ੒
    • Kubernetes (OpenShift) on OpenStack on Kubernetes
    Pod
    VM
    Pod
    Pod

    View full-size slide

  9. Ϛϧνςφϯτkubernetesͷߏ੒
    • Kubernetes (OpenShift) on OpenStack on Kubernetes
    Pod
    VM
    Pod
    Pod
    ຊ೔ͷൃදείʔϓ͸

    ίί

    View full-size slide

  10. Ϛϧνςφϯτkubernetesͷߏ੒ → OKDͷུ֓ਤ
    • ུ֓
    Master × 5
    Etcd × 5
    Route × 10
    Node × ͍ͬͺ͍
    Master
    Route(HAproxy)
    okd-node001(8core/48GB)
    Master
    Master
    Master
    okd-node002
    Route(HAproxy)
    Etcd
    Etcd
    Etcd
    Etcd
    Etcd
    Prj-2

    pod1
    Prj-1

    pod1
    Prj-1

    pod2
    Prj-3

    pod1
    Prj-2

    pod2
    Prj-4

    pod1

    View full-size slide

  11. mackerelಋೖͷHOW TO
    okd-node001(8core/48GB)
    Prj-2

    pod1
    Prj-1

    pod1
    Prj-1

    pod2
    ͜͜ʹͲ͏΍ͬͯmackrel-container-agentΛ

    ಋೖ͢Δʁʁ

    View full-size slide

  12. mackerelಋೖͷHOW TO
    • manifestʹsidecarΛهࡌͯ͠ߦ͏σϓϩΠ
    • ७ਮͳํ๏
    • MutatingAdmissionWebhookΛར༻ͨ͠σϓϩΠ
    • istioͰͷenvoyͷinjectionͱಉ͡Α͏ͳํ๏
    • TemplateΛར༻ͨ͠σϓϩΠ
    • OpenShiftͷtemplateػೳΛར༻ͨ͠ํ๏

    View full-size slide

  13. mackerelಋೖͷHOW TO
    manifestʹsidecarΛهࡌͯ͠ߦ͏σϓϩΠ

    View full-size slide

  14. manifestʹsidecarΛهࡌͯ͠σϓϩΠ
    • ಉҰPod಺ʹίϯςφΛ૬৐Γͤ͞Δํ๏
    • kubernetesͷsidecarΛར༻ͨ͠σϓϩΠͰ͕͢

    ΋ͪΖΜOpenShiftͰ΋࢖͑·͢

    View full-size slide

  15. manifestʹsidecarΛهࡌͯ͠σϓϩΠ
    containers:
    - name: service-container-main
    ɹɾ
    ɹɾ
    ɹɾ
    - name: mackerel-container-agent
    spec.template.spec.containers

    View full-size slide

  16. Secret
    apikey: "YOUR_MACKEREL_APIKEY"
    roles:
    - "mackerelmeetup:database"
    ignoreContainer: '\Amackerel-container-agent\z'
    mackerel.yaml
    Create Secret
    $ oc create secret generic mackerel-config --from-file=./mackerel.yaml
    secret/mackerel-config created
    apikeyΛsecretʹͯ͠ઃఆྨ͸configmapͰ͋ͯΔ͜ͱ΋Ͱ͖Δ͕ɺ
    secretͱconfigmapΛ؅ཧ͢Δͷ͸໘౗ͳͨΊɺઃఆϑΝΠϧΛؙʑSecretʹͨ͠

    View full-size slide

  17. DeploymentConfig (spec.template.spec.containers[].env[])
    - name: MACKEREL_KUBERNETES_NAMESPACE
    valueFrom:
    fieldRef:
    apiVersion: v1
    fieldPath: metadata.namespace
    MACKEREL_KUBERNETES_NAMESPACE
    MACKEREL_KUBERNETES_POD_NAME
    - name: MACKEREL_KUBERNETES_POD_NAME
    valueFrom:
    fieldRef:
    apiVersion: v1
    fieldPath: metadata.name

    View full-size slide

  18. - name: MACKEREL_KUBERNETES_KUBELET_HOST
    valueFrom:
    fieldRef:
    apiVersion: v1
    fieldPath: status.hostIP
    MACKEREL_KUBERNETES_KUBELET_HOST
    MACKEREL_CONTAINER_PLATFORM
    - name: MACKEREL_CONTAINER_PLATFORM
    value: kubernetes
    MACKEREL_KUBERNETES_KUBELET_READ_ONLY_PORT
    - name: MACKEREL_KUBERNETES_KUBELET_READ_ONLY_PORT
    value: 10255
    DeploymentConfig (spec.template.spec.containers[].env[])

    View full-size slide

  19. - name: MACKEREL_AGENT_CONFIG
    value: /etc/mackerel/mackerel.yaml
    MACKEREL_AGENT_CONFIG
    DeploymentConfig (spec.template.spec.containers[].env[])
    DeploymentConfig (spec.template.spec.containers[].env[])

    View full-size slide

  20. volumeMounts:
    - mountPath: /etc/mackerel/
    name: config
    DeploymentConfig (spec.template.spec.containers[])

    View full-size slide

  21. - name: config
    secret:
    defaultMode: 420
    secretName: mackerel-config
    DeploymentConfig (spec.template.spec.volumes[])

    View full-size slide

  22. manifestʹsidecarΛهࡌͯ͠σϓϩΠ
    • ར఺
    • ࣗ༝౓͕ߴ͍ → resource΍image౳ͷࢦఆ͕Ͱ͖Δ
    • ܽ఺
    • okdͷmanifestͷॻ͖ํΛ஌͍ͬͯΔඞཁ͕͋Δ

    (kubernetesͱҰ෦ҟͳΔՕॴ͕ଘࡏ͢Δ)

    View full-size slide

  23. mackerelಋೖͷHOW TO
    MutatingAdmissionWebhookΛར༻ͨ͠σϓϩΠ

    View full-size slide

  24. MutatingAdmissionWebhookΛར༻ͨ͠σϓϩΠ
    • Πϝʔδతʹ͸istioʹ͓͚Δenvoyͷ஫ೖͱಉ͡ํ๏
    • annotationʹkey/valueΛ౉ͯ͠ɺͦΕΛϑοΫͯ͠ɺ

    mackerel-container-agentΛೖΕΔsidecarͷઃఆΛࠩ͠ࠐΉ
    • mackrel-pluginɺapikey͸ϢʔβຖʹҟͳΔͷͰɺ

    annotationͰsecret໊Λࢦఆ

    View full-size slide

  25. Secret
    apikey: "YOUR_MACKEREL_APIKEY"
    roles:
    - "mackerelmeetup:database"
    ignoreContainer: '\Amackerel-container-agent\z'
    mackerel.yaml
    Create Secret
    $ oc create secret generic mackerel-config --from-file=./mackerel.yaml
    secret/mackerel-config created

    View full-size slide

  26. spec:
    replicas: 1
    template:
    metadata:
    annotations:
    mackerel-sidecar-injector.cycloud.io/inject: "yes"
    mackerel-sidecar-injector.cycloud.io/secret: "mackerel-conf-secret"
    labels:
    app: sleep
    spec:
    containers:
    ɹɹɹ - name: service-container-main
    MutatingAdmissionWebhookΛར༻ͨ͠σϓϩΠ
    spec.template.metadata.annotationsʹ߲໨Λ௥Ճ

    mackerel-sidecar-injector.cycloud.io/inject → inject͢Δ৔߹͸yes
    mackerel-sidecar-injector.cycloud.io/secret → mackrel-container-agentʹ͋ͯΔsecretΛࢦఆ

    View full-size slide

  27. MutatingAdmissionWebhookΛར༻ͨ͠σϓϩΠ
    • ར఺
    • Ϣʔβ͸annotationͱsecretΛ௥Ճ͢Ε͹ྑ͍
    • ܽ఺
    • sidecarͷઃఆΛॻ͘৔߹ʹൺ΂ࣗ༝౓͸ˣ

    (resourceઃఆ΍Πϝʔδࢦఆ౳)

    View full-size slide

  28. mackerelಋೖͷHOW TO
    TemplateΛར༻ͨ͠σϓϩΠ

    View full-size slide

  29. Template ?
    • Pod Templateͱ͸શ͘ҧ͏΋ͷ
    • TemplateΛఆ͓ٛͯ͘͠ͱɺ

    ࣗ਎ͷϓϩδΣΫτͰར༻Ͱ͖Δݖݶ಺ͰσϓϩΠͳͲΛ

    ઃఆ͢Δ͜ͱ͕Մೳ
    • Global template libaraly഑Լɺ

    ·ͨ͸projectʹΞοϓϩʔυ͍ͯ͠Δͱը໘͔ΒϙνϙνͰ

    σϓϩΠͳͲΛߦ͑Δ(CLI΋ར༻Մೳ)

    View full-size slide

  30. Template ?
    DBϢʔβͱ͔ύεϫʔυͱ͔ઃఆ஋ΛೖΕΔ
    ී௨ͷDBͱͯ͠ར༻Ͱ͖Δ

    View full-size slide

  31. TemplateΛར༻ͨ͠σϓϩΠ
    ೚ҙͷςϯϓϨʔτΛΫϦοΫ

    View full-size slide

  32. TemplateΛར༻ͨ͠σϓϩΠ
    ֤ύϥϝʔλʹରͯ͠೚ҙͷઃఆ஋ΛೖΕΔ
    Add to Project : mackerel-meetup13

    mackerel Api Token : XXXXXXXXXXXXXXXX


    mackerel service : mackerelmeetup

    mackerel role : database

    View full-size slide

  33. TemplateΛར༻ͨ͠σϓϩΠ
    createΛԡ͢

    View full-size slide

  34. TemplateΛར༻ͨ͠σϓϩΠ
    ग़དྷ্͕Γ·Ͱ଴ͭ

    View full-size slide

  35. TemplateΛར༻ͨ͠σϓϩΠ

    View full-size slide

  36. TemplateΛར༻ͨ͠σϓϩΠ

    View full-size slide

  37. TemplateΛར༻ͨ͠σϓϩΠ

    View full-size slide

  38. Templateઃఆํ๏
    apiVersion: template.openshift.io/v1
    kind: template
    labels:
    messages:
    metadata:
    objects:
    parameters:
    mackerel-middle-template.yaml

    View full-size slide

  39. Templateઃఆํ๏
    apiVersion: template.openshift.io/v1
    kind: template
    labels:
    messages:
    metadata:
    objects:
    parameters:
    mackerel-middle-template.yaml

    View full-size slide

  40. Templateઃఆํ๏
    messages:
    ɹɹςϯϓϨʔτͰϢʔβ͕ࢦఆͨ͠஋ΛݟͤͨΓ͢Δ͜ͱ͕Ͱ͖Δ
    metadata:
    ɹɹmetadata.annotationͰtemplateͷσΟεϓϨΠ໊΍iconͷࢦఆ͕Ͱ͖Δ
    (༻ҙ͞Ε͍ͯͳ͍iconʹ͍ͭͯ͸ผ్༻ҙ͢Δඞཁ͋Γ)

    View full-size slide

  41. Templateઃఆํ๏
    apiVersion: template.openshift.io/v1
    kind: template
    labels:
    messages:
    metadata:
    objects:
    parameters:
    mackerel-middle-template.yaml

    View full-size slide

  42. Templateઃఆํ๏
    parameters:
    ɹɹϢʔβ͕ೖΕΔ஋Λઃఆ͢Δ߲໨
    parameters:
    - description: Mackerel Api Token
    displayName: mackerel Api Token
    name: MACKEREL_API_KEY
    required: true
    - description: Maximum amount of memory the container can use.
    displayName: Memory Limit
    name: MEMORY_LIMIT
    required: true
    value: 512Mi
    - description: Username for MySQL user that will be used for accessing the database.
    displayName: MySQL Connection Username
    from: user[A-Z0-9]{3}
    generate: expression
    name: MYSQL_USER
    required: true

    View full-size slide

  43. Templateઃఆํ๏
    apiVersion: template.openshift.io/v1
    kind: template
    labels:
    messages:
    metadata:
    objects:
    parameters:
    mackerel-middle-template.yaml

    View full-size slide

  44. Templateઃఆํ๏
    objects:
    ɹɹ࣮ࡍʹσϓϩΠ͢ΔmanifestΛهࡌ͢Δ

    ɹɹ(mackerel-container-agent΋sidecarͱͯ͜͜͠ʹهࡌ͞Ε͍ͯΔ)
    objects:
    - apiVersion: v1
    kind: Secret
    metadata:
    annotations:
    template.openshift.io/expose-apitoken: '{.data[''mackerel-api-key'']}'
    template.openshift.io/expose-database_name: '{.data[''database-name'']}'
    template.openshift.io/expose-password: '{.data[''database-password'']}'
    template.openshift.io/expose-username: '{.data[''database-user'']}'
    name: ${DATABASE_SERVICE_NAME}
    stringData:
    database-name: ${MYSQL_DATABASE}
    database-password: ${MYSQL_PASSWORD}
    database-user: ${MYSQL_USER}
    mackerel-api-key: ${MACKEREL_API_KEY}
    - apiVersion: v1
    kind: DeploymentConfig
    metadata:
    annotations:
    template.alpha.openshift.io/wait-for-ready: "true"
    name: ${DATABASE_SERVICE_NAME}
    spec:
    replicas: 1
    selector:
    name: ${DATABASE_SERVICE_NAME}
    strategy:
    type: Recreate
    template:
    metadata:
    labels:
    name: ${DATABASE_SERVICE_NAME}
    spec:
    containers:
    - name: mackerel-container-agent

    View full-size slide

  45. Templateઃఆํ๏
    objects:
    ɹɹ࣮ࡍʹσϓϩΠ͢ΔmanifestΛهࡌ͢Δ

    ɹɹ(mackerel-container-agent΋sidecarͱͯ͜͜͠ʹهࡌ͞Ε͍ͯΔ)
    objects:
    - apiVersion: v1
    kind: Secret
    metadata:
    annotations:
    template.openshift.io/expose-apitoken: '{.data[''mackerel-api-key'']}'
    template.openshift.io/expose-database_name: '{.data[''database-name'']}'
    template.openshift.io/expose-password: '{.data[''database-password'']}'
    template.openshift.io/expose-username: '{.data[''database-user'']}'
    name: ${DATABASE_SERVICE_NAME}
    stringData:
    database-name: ${MYSQL_DATABASE}
    database-password: ${MYSQL_PASSWORD}
    database-user: ${MYSQL_USER}
    mackerel-api-key: ${MACKEREL_API_KEY}
    - apiVersion: v1
    kind: DeploymentConfig
    metadata:
    annotations:
    template.alpha.openshift.io/wait-for-ready: "true"
    name: ${DATABASE_SERVICE_NAME}
    spec:
    replicas: 1
    selector:
    name: ${DATABASE_SERVICE_NAME}
    strategy:
    type: Recreate
    template:
    metadata:
    labels:
    name: ${DATABASE_SERVICE_NAME}
    spec:
    containers:
    - name: mackerel-container-agent
    Secret
    DeploymentConfig

    View full-size slide

  46. Templateઃఆํ๏
    ઃఆ൓ө

    (൓ө·Ͱগ͕͔͔࣌ؒ͠Γ·͢)
    $ oc create -f ${template-manifest}.yaml -n openshift

    View full-size slide

  47. TemplateΛར༻ͨ͠σϓϩΠ
    • ར఺
    • Ϣʔβ͸σϓϩΠ࣌΋͸΍manifestΛશ͘ॻ͔ͳͯ͘΋ྑ͍

    ྫ) ը໘ϙνϙνͰDBͳͲΛ࡞੒

    ɹ mackerel dashboardʹࣗಈͰొ࿥
    • ܽ఺
    • ࣗ༝౓͸΄΅̌

    View full-size slide

  48. OpenShiftΛར༻͍ͯ͠Δํ΁ͷ஫ҙࣄ߲
    σϑΥͰΠϯετʔϧ͢Δ৔߹

    ReadOnlyPort͕ઃఆ͞Εͳ͍ͷͰɺར༻͢Δ৔߹͸ઃఆ͍ͯͩ͘͠͞

    (/etc/origin/node/node-config.yaml)
    kubeletArguments:
    read-only-port:
    - '10255'
    bootstrap-kubeconfig:
    - /etc/origin/node/bootstrap.kubeconfig
    cert-dir:
    - /etc/origin/node/certificates
    cloud-config:
    - /etc/origin/cloudprovider/openstack.conf

    View full-size slide

  49. ·ͱΊ
    - mackerel-container-agentͷಋೖ͸γϯϓϧͰָ
    - ϢʔβϨϕϧ΍໨తʹ͋ͬͨΧελϚΠζ͕ߦ͑Δ
    - طଘͷplugin͕࢖͑ͯخ͍͠
    - plugin͕ೖͬͨίϯςφΠϝʔδΛԿଔެࣜͰ༻ҙ͍͚ͯͨͩ͠Δͱm(_ _)m
    - MutatingAdmissionWebhookΛ༻ҙ͍͚ͯͨͩ͠Δͱm(_ _)m

    View full-size slide

  50. Thank you for listening.

    View full-size slide