Kubernetes on AWS at Chatwork

AWS Summit 2018 スタートアップ特設エリアにおける発表内容で、ChatWorkにおけるKubernetesの運用話です。

Transcript

1. νϟοτϫʔΫʹ͓͚Δ

   Kubernetes on AWS /
   Kubernetes on AWS at ChatWork
   SRE
   Ryo Sakamoto
   

   View Slide

2. © ChatWork
   ▸ ೔ຊൃϏδωενϟοτ
   ▸ λεΫ؅ཧ΍ϏσΦ௨࿩͕Մೳ
   ▸ ಋೖاۀ174,000ࣾҎ্ʢ※2018೥4݄຤೔࣌఺ʣ
   

   View Slide

3. © ChatWork
   ΞδΣϯμ
   ▸ Kubernetesͷ؀ڥ΍ಈ͍͍ͯΔΞϓϦ
   ▸ KubernetesͰར༻͍ͯ͠Δπʔϧ
   ▸ Kubernetesͷ؂ࢹɺϩΪϯά
   ▸ Kubernetesͷversion up
   ▸ ·ͱΊ
   

   View Slide

4. © ChatWork
   Kubernetesͷར༻
   ▸ ϝοηʔδॲཧ෦෼ͷϦϓϨΠε(2016೥12݄)
   ▸ backen appΛkubernetes Ͱಈ͔͢
   ▸ ࡉ͔͍࿩͸ࡢ೥ͷAWS summitͰ…
   

   View Slide

5. © ChatWork
   Kubernetesͷ؀ڥ
   ▸ ؀ڥ AWS
   ▸ AWSͳͲطଘͷࢿݯΛར༻͔ͨͬͨͨ͠Ί
   ▸ ߏஙπʔϧ kube-aws
   ▸ https://github.com/kubernetes-incubator/kube-aws
   ▸ ϝΠϯϝϯςφ͸mumoshu (chatwork kubernetes ސ໰)
   ▸ cloudformation(ͱcloud-init)Ͱ·ΔͬͱߏஙͰ͖Δ
   

   View Slide

6. © ChatWork
   KubernetesͰಈ͍͍ͯΔ΋ͷ
   ▸ backend
   ▸ ࡢ೥ͷAWS summitͰmessage backendΛϦϓϨΠεͨ͠࿩
   ▸ ͜ͷϓϩδΣΫτҎ߱(webhook, oauthͳͲ)͸͢΂ͯkubernetes
   ▸ ϊʔυ਺͸m4.2xlarge * 10୆ఔ౓
   ▸ CD؀ڥ(concourse)
   ▸ spot instance ͳnodepoolΛར༻
   

   View Slide

7. © ChatWork
   KubernetesͰར༻͍ͯ͠Δπʔϧ(1)
   ▸ cluster-autoscaler
   ▸ podͷauto scaleͰ͸ͳ͘ɺnodeͷauto scaleͯ͘͠ΕΔ
   ▸ schedulerΛ؂ࢹͯ͠ɺϦιʔε͕଍Γͳ͍pod͕͍ΔͱASGΛૢ࡞
   ▸ nodeͷ୆਺࡟ݮʹେ͖͘ߩݙ
   ▸ σϓϩΠ࣌ͷpodͷೖସ͑ͳͲͰҰ࣌తʹϊʔυ͕଍Γͳ͘ͳΔͱ͖
   ΋͞ΒͬͱରԠͯ͘͠ΕΔ
   

   View Slide

8. © ChatWork
   cluster-autoscalerͷಈ͖(scale out)
   controller nodepool
   api-server
   scheduler
   cluster-
   autoscaler
   pod
   (1) watch
   (2) 

   “fails to be scheduled
   due to insufficient”
   (4)scale out
   (3) 

   set-desired-capacity
   ൑ྫ
   

   View Slide

9. © ChatWork
   cluster-autoscalerͷಈ͖(scale in)
   controller nodepool
   api-server
   scheduler
   cluster-
   autoscaler
   pod
   (1) watch

   (apiܦ༝)
   nodeͷ࢖༻཰௿
   (3) 

   set-desired-capacity
   ൑ྫ
   a
   b
   a
   b
   nodeͷ࢖༻཰௿
   a
   b
   a
   b
   (4) scale in
   (2) evict
   

   View Slide

10. © ChatWork
    KubernetesͰར༻͍ͯ͠Δπʔϧ(2)
    ▸ kube2iam
    ▸ podຖʹroleͷ෇༩
    ▸ ௨ৗ͸ΠϯελϯεͷϩʔϧΛར༻
    ▸ ෆཁͳpolicy͕෇͘ & ଍Γͳ͍΋ͷ͸APIKEYΛ΋ͨͤΔ͜ͱʹͳΔ
    ▸ secret͸base64ͳ͚ͩ
    ▸ एׯෆ҆ఆͰɺkiamʹஔ͖׵͑༧ఆ
    

    View Slide

11. © ChatWork
    kube2iam
    ▸ annotationʹroleΛهࡌ
    ▸ role͸workerͷroleΛ৴པ͓ͯ͘͠
    ▸ worker͕asuumeͰ͖ΔΑ͏ʹ͓ͯ͘͠
    ▸ pod͕ɺAWSͷAPIΛར༻͠Α͏ͱ͢ΔͱɺmetadataʹΞΫηε͢Δ
    ▸ metadata΁ͷΞΫηεΛiptablesͰkube2iamʹసૹ
    ▸ kube2iam͕annotationͷroleͷΫϨσϯγϟϧΛൃߦ
    

    View Slide

12. © ChatWork
    kube2iam
    app
    kube2iam 1. credentialͷൃߦ(ec2-metadata)
    2. iptablesͰkube2iamͷpodʹϦΫΤετ͕సૹ
    3. credentialͷൃߦ
    pod
    ൑ྫ
    

    View Slide

13. © ChatWork
    Kubernetesͷ؂ࢹ
    ▸ datadog only
    ▸ daemonsetͰ഑ஔ
    ▸ not k8sͳ؀ڥͷ؂ࢹͱ౷Ұ͍ͨ͠ & prometheusͷ؅ཧΛͨ͘͠ͳ͍
    ▸ prometheusͷΑ͏ʹΤϯυϙΠϯτΛੜ΍͢ͷͰ͸ͳ͘ɺ֤ϗετͷ
    statsdʹૹ৴
    ▸ version 6Λར༻
    ▸ v5ͰϝτϦΫε͕͚͍ܽͯͨ(ϝτϦΫεᷓΕ)͕ɺv6Ͱ͚ܽͳ͘ͳͬͨ
    

    View Slide

14. © ChatWork
    datadogͷlive container monitoring
    

    View Slide

15. © ChatWork
    KubernetesͷϩΪϯά
    ▸ fluentd + stackdriver
    ▸ fluentdΛdaemonsetͰ഑ஔ
    ▸ ֤ίϯςφ͸ϗετͷಛఆͷ৔ॴʹstdoutΛు͖ग़͍ͯ͠Δ
    ▸ audit-log΋fluentdͰstackdriverʹૹ৴
    ▸ S3Ͱ΋Α͔͕ͬͨɺKubernetesΛಋೖͨ࣌͠ʹ͸Athena͸ग़ͨ͹͔Γ
    ▸ stackdriver + bigquery΋Ұ෦ͷϩάͰಋೖ
    

    View Slide

16. © ChatWork
    Kubernetesͷversion up
    ▸ kube-awsͰ؅ཧ͍ͯ͠ΔҎ্ɺϚωʔδυͳversion up͸Ͱ͖ͳ͍
    ▸ version up΍kubernetes ౷߹Λߦͬͨ
    ▸ version up 1.7 -> 1.8, 1.5 -> 1.8 && 1.8ԽͷλΠϛϯάͰΫϥελ౷߹
    ▸ νϟοτϫʔΫͰ͸·ͩingress͸ར༻Ͱ͖͍ͯͳͯ͘ɺELB + NodePort
    ▸ ͳͷͰɺversion up͸ELBʹ৽چ྆ํΛͿΒԼ͛ͯɺݹ͍ํΛޙୀ
    ▸ ࠓͷΞϓϦέʔγϣϯͱͯ͠͸onlineͰversion up׬ྃ
    

    View Slide

17. © ChatWork
    version up
    old k8s
    pod
    ൑ྫ
    app
    chatwork web
    NodePort
    

    View Slide

18. © ChatWork
    version up
    old k8s
    pod
    ൑ྫ
    app
    app
    chatwork web
    NodePort
    NodePort
    new k8s
    

    View Slide

19. © ChatWork
    version up
    pod
    ൑ྫ
    app
    chatwork web
    NodePort
    new k8s
    

    View Slide

20. © ChatWork
    KubernetesͰࠓޙ΍Γ͍ͨ(1)
    ▸ EKS
    ▸ kube-awsʹ૊Έࠐ·ΕΔ༧ఆ
    ▸ ΍ͬͺΓϩʔϦϯάΞοϓσʔτ͍ͨ͠
    ▸ service mesh
    ▸ envoyͷಋೖ
    ▸ istio, linkerd
    ▸ grpc loadbalancer -> envoy, nginx-ingress-controller
    

    View Slide

21. © ChatWork
    KubernetesͰࠓޙ΍Γ͍ͨ(2)
    ▸ prometheusͷಋೖ
    ▸ hpa͸σϑΥϧτͰ͸cpu͔͠ͳ͍ͷͰ͔ͭʹ͍͘
    ▸ cpuͰ͸ͳ͘kafkaͷeventͷ٧·Γ۩߹Ͱscale in/out͍ͨ͠
    ▸ datadogͰ΋apiΛ࢖ͬͯͰ͖Δ͚Ͳɺdatadogͱͷ઀ଓෆ҆
    ▸ ϓϥοτϑΥʔϜԽ
    ▸ openFaaSͳͲ
    

    View Slide

22. © ChatWork
    EKS΁ͷظ଴
    ▸ preview൛Λར༻͍͍ͤͯͨͩͨ͞
    ▸ workerͷ௥Ճ͕͕͕͕͕….configmapܦ༝Ͱొ࿥͢Δɺͱ͍͏ํ๏ͩͬͨ
    ▸ ͜Εͩͱkubernetesͷ֎ͰɺΫϥελߏங͕ด͡ͳ͍
    ▸ AWSͷϦιʔε׆༻(IAMɺVPC)ͳͲظ଴
    ▸ fargateͰnodeͦͷ΋ͷΛҙࣝ͠ͳ͍ͷ͸͍͍͕ɺlogging΍؂ࢹ͸…
    

    View Slide

23. © ChatWork
    ·ͱΊ
    ▸ νϟοτϫʔΫͷKubernetes؀ڥʹ͍ͭͯͷ࿩
    ▸ ͍Ζ͍Ζ΍Γ͍ͨ͜ͱ͸͋Δ
    ▸ EKSʹ΋ظ଴
    ▸ controll plane͕Ϛωʔδυ͞ΕΔ҆৺ײ
    

    View Slide

24. © ChatWork
    ΤϯδχΞืूத
    http://corp.chatwork.com/ja/recruit/
    ▸ ओମੑΛ࣋ͪɺࣗΒߦಈͰ͖Δ
    ▸ ଞऀΛೝΊɺଚॏͰ͖Δ
    ▸ ৘ใΛूΊɺڞ༗Ͱ͖Δ
    ͱ͍͏ํΛ׻ܴ͠·͢ʂ
    

    View Slide