$30 off During Our Annual Pro Sale. View Details »

Kubernetes on AWS at Chatwork

Ryo Sakamoto
May 28, 2018
1.4k

Kubernetes on AWS at Chatwork

AWS Summit 2018 スタートアップ特設エリアにおける発表内容で、ChatWorkにおけるKubernetesの運用話です。

Ryo Sakamoto

May 28, 2018
Tweet

Transcript

  1. νϟοτϫʔΫʹ͓͚Δ

    Kubernetes on AWS /
    Kubernetes on AWS at ChatWork
    SRE
    Ryo Sakamoto

    View Slide

  2. © ChatWork
    ▸ ೔ຊൃϏδωενϟοτ
    ▸ λεΫ؅ཧ΍ϏσΦ௨࿩͕Մೳ
    ▸ ಋೖاۀ174,000ࣾҎ্ʢ※2018೥4݄຤೔࣌఺ʣ

    View Slide

  3. © ChatWork
    ΞδΣϯμ
    ▸ Kubernetesͷ؀ڥ΍ಈ͍͍ͯΔΞϓϦ
    ▸ KubernetesͰར༻͍ͯ͠Δπʔϧ
    ▸ Kubernetesͷ؂ࢹɺϩΪϯά
    ▸ Kubernetesͷversion up
    ▸ ·ͱΊ

    View Slide

  4. © ChatWork
    Kubernetesͷར༻
    ▸ ϝοηʔδॲཧ෦෼ͷϦϓϨΠε(2016೥12݄)
    ▸ backen appΛkubernetes Ͱಈ͔͢
    ▸ ࡉ͔͍࿩͸ࡢ೥ͷAWS summitͰ…

    View Slide

  5. © ChatWork
    Kubernetesͷ؀ڥ
    ▸ ؀ڥ AWS
    ▸ AWSͳͲطଘͷࢿݯΛར༻͔ͨͬͨͨ͠Ί
    ▸ ߏஙπʔϧ kube-aws
    ▸ https://github.com/kubernetes-incubator/kube-aws
    ▸ ϝΠϯϝϯςφ͸mumoshu (chatwork kubernetes ސ໰)
    ▸ cloudformation(ͱcloud-init)Ͱ·ΔͬͱߏஙͰ͖Δ

    View Slide

  6. © ChatWork
    KubernetesͰಈ͍͍ͯΔ΋ͷ
    ▸ backend
    ▸ ࡢ೥ͷAWS summitͰmessage backendΛϦϓϨΠεͨ͠࿩
    ▸ ͜ͷϓϩδΣΫτҎ߱(webhook, oauthͳͲ)͸͢΂ͯkubernetes
    ▸ ϊʔυ਺͸m4.2xlarge * 10୆ఔ౓
    ▸ CD؀ڥ(concourse)
    ▸ spot instance ͳnodepoolΛར༻

    View Slide

  7. © ChatWork
    KubernetesͰར༻͍ͯ͠Δπʔϧ(1)
    ▸ cluster-autoscaler
    ▸ podͷauto scaleͰ͸ͳ͘ɺnodeͷauto scaleͯ͘͠ΕΔ
    ▸ schedulerΛ؂ࢹͯ͠ɺϦιʔε͕଍Γͳ͍pod͕͍ΔͱASGΛૢ࡞
    ▸ nodeͷ୆਺࡟ݮʹେ͖͘ߩݙ
    ▸ σϓϩΠ࣌ͷpodͷೖସ͑ͳͲͰҰ࣌తʹϊʔυ͕଍Γͳ͘ͳΔͱ͖
    ΋͞ΒͬͱରԠͯ͘͠ΕΔ

    View Slide

  8. © ChatWork
    cluster-autoscalerͷಈ͖(scale out)
    controller nodepool
    api-server
    scheduler
    cluster-
    autoscaler
    pod
    (1) watch
    (2) 

    “fails to be scheduled
    due to insufficient”
    (4)scale out
    (3) 

    set-desired-capacity
    ൑ྫ

    View Slide

  9. © ChatWork
    cluster-autoscalerͷಈ͖(scale in)
    controller nodepool
    api-server
    scheduler
    cluster-
    autoscaler
    pod
    (1) watch

    (apiܦ༝)
    nodeͷ࢖༻཰௿
    (3) 

    set-desired-capacity
    ൑ྫ
    a
    b
    a
    b
    nodeͷ࢖༻཰௿
    a
    b
    a
    b
    (4) scale in
    (2) evict

    View Slide

  10. © ChatWork
    KubernetesͰར༻͍ͯ͠Δπʔϧ(2)
    ▸ kube2iam
    ▸ podຖʹroleͷ෇༩
    ▸ ௨ৗ͸ΠϯελϯεͷϩʔϧΛར༻
    ▸ ෆཁͳpolicy͕෇͘ & ଍Γͳ͍΋ͷ͸APIKEYΛ΋ͨͤΔ͜ͱʹͳΔ
    ▸ secret͸base64ͳ͚ͩ
    ▸ एׯෆ҆ఆͰɺkiamʹஔ͖׵͑༧ఆ

    View Slide

  11. © ChatWork
    kube2iam
    ▸ annotationʹroleΛهࡌ
    ▸ role͸workerͷroleΛ৴པ͓ͯ͘͠
    ▸ worker͕asuumeͰ͖ΔΑ͏ʹ͓ͯ͘͠
    ▸ pod͕ɺAWSͷAPIΛར༻͠Α͏ͱ͢ΔͱɺmetadataʹΞΫηε͢Δ
    ▸ metadata΁ͷΞΫηεΛiptablesͰkube2iamʹసૹ
    ▸ kube2iam͕annotationͷroleͷΫϨσϯγϟϧΛൃߦ

    View Slide

  12. © ChatWork
    kube2iam
    app
    kube2iam 1. credentialͷൃߦ(ec2-metadata)
    2. iptablesͰkube2iamͷpodʹϦΫΤετ͕సૹ
    3. credentialͷൃߦ
    pod
    ൑ྫ

    View Slide

  13. © ChatWork
    Kubernetesͷ؂ࢹ
    ▸ datadog only
    ▸ daemonsetͰ഑ஔ
    ▸ not k8sͳ؀ڥͷ؂ࢹͱ౷Ұ͍ͨ͠ & prometheusͷ؅ཧΛͨ͘͠ͳ͍
    ▸ prometheusͷΑ͏ʹΤϯυϙΠϯτΛੜ΍͢ͷͰ͸ͳ͘ɺ֤ϗετͷ
    statsdʹૹ৴
    ▸ version 6Λར༻
    ▸ v5ͰϝτϦΫε͕͚͍ܽͯͨ(ϝτϦΫεᷓΕ)͕ɺv6Ͱ͚ܽͳ͘ͳͬͨ

    View Slide

  14. © ChatWork
    datadogͷlive container monitoring

    View Slide

  15. © ChatWork
    KubernetesͷϩΪϯά
    ▸ fluentd + stackdriver
    ▸ fluentdΛdaemonsetͰ഑ஔ
    ▸ ֤ίϯςφ͸ϗετͷಛఆͷ৔ॴʹstdoutΛు͖ग़͍ͯ͠Δ
    ▸ audit-log΋fluentdͰstackdriverʹૹ৴
    ▸ S3Ͱ΋Α͔͕ͬͨɺKubernetesΛಋೖͨ࣌͠ʹ͸Athena͸ग़ͨ͹͔Γ
    ▸ stackdriver + bigquery΋Ұ෦ͷϩάͰಋೖ

    View Slide

  16. © ChatWork
    Kubernetesͷversion up
    ▸ kube-awsͰ؅ཧ͍ͯ͠ΔҎ্ɺϚωʔδυͳversion up͸Ͱ͖ͳ͍
    ▸ version up΍kubernetes ౷߹Λߦͬͨ
    ▸ version up 1.7 -> 1.8, 1.5 -> 1.8 && 1.8ԽͷλΠϛϯάͰΫϥελ౷߹
    ▸ νϟοτϫʔΫͰ͸·ͩingress͸ར༻Ͱ͖͍ͯͳͯ͘ɺELB + NodePort
    ▸ ͳͷͰɺversion up͸ELBʹ৽چ྆ํΛͿΒԼ͛ͯɺݹ͍ํΛޙୀ
    ▸ ࠓͷΞϓϦέʔγϣϯͱͯ͠͸onlineͰversion up׬ྃ

    View Slide

  17. © ChatWork
    version up
    old k8s
    pod
    ൑ྫ
    app
    chatwork web
    NodePort

    View Slide

  18. © ChatWork
    version up
    old k8s
    pod
    ൑ྫ
    app
    app
    chatwork web
    NodePort
    NodePort
    new k8s

    View Slide

  19. © ChatWork
    version up
    pod
    ൑ྫ
    app
    chatwork web
    NodePort
    new k8s

    View Slide

  20. © ChatWork
    KubernetesͰࠓޙ΍Γ͍ͨ(1)
    ▸ EKS
    ▸ kube-awsʹ૊Έࠐ·ΕΔ༧ఆ
    ▸ ΍ͬͺΓϩʔϦϯάΞοϓσʔτ͍ͨ͠
    ▸ service mesh
    ▸ envoyͷಋೖ
    ▸ istio, linkerd
    ▸ grpc loadbalancer -> envoy, nginx-ingress-controller

    View Slide

  21. © ChatWork
    KubernetesͰࠓޙ΍Γ͍ͨ(2)
    ▸ prometheusͷಋೖ
    ▸ hpa͸σϑΥϧτͰ͸cpu͔͠ͳ͍ͷͰ͔ͭʹ͍͘
    ▸ cpuͰ͸ͳ͘kafkaͷeventͷ٧·Γ۩߹Ͱscale in/out͍ͨ͠
    ▸ datadogͰ΋apiΛ࢖ͬͯͰ͖Δ͚Ͳɺdatadogͱͷ઀ଓෆ҆
    ▸ ϓϥοτϑΥʔϜԽ
    ▸ openFaaSͳͲ

    View Slide

  22. © ChatWork
    EKS΁ͷظ଴
    ▸ preview൛Λར༻͍͍ͤͯͨͩͨ͞
    ▸ workerͷ௥Ճ͕͕͕͕͕….configmapܦ༝Ͱొ࿥͢Δɺͱ͍͏ํ๏ͩͬͨ
    ▸ ͜Εͩͱkubernetesͷ֎ͰɺΫϥελߏங͕ด͡ͳ͍
    ▸ AWSͷϦιʔε׆༻(IAMɺVPC)ͳͲظ଴
    ▸ fargateͰnodeͦͷ΋ͷΛҙࣝ͠ͳ͍ͷ͸͍͍͕ɺlogging΍؂ࢹ͸…

    View Slide

  23. © ChatWork
    ·ͱΊ
    ▸ νϟοτϫʔΫͷKubernetes؀ڥʹ͍ͭͯͷ࿩
    ▸ ͍Ζ͍Ζ΍Γ͍ͨ͜ͱ͸͋Δ
    ▸ EKSʹ΋ظ଴
    ▸ controll plane͕Ϛωʔδυ͞ΕΔ҆৺ײ

    View Slide

  24. © ChatWork
    ΤϯδχΞืूத
    http://corp.chatwork.com/ja/recruit/
    ▸ ओମੑΛ࣋ͪɺࣗΒߦಈͰ͖Δ
    ▸ ଞऀΛೝΊɺଚॏͰ͖Δ
    ▸ ৘ใΛूΊɺڞ༗Ͱ͖Δ
    ͱ͍͏ํΛ׻ܴ͠·͢ʂ

    View Slide