Kubernetes on AWS at Chatwork

AWS Summit 2018 スタートアップ特設エリアにおける発表内容で、ChatWorkにおけるKubernetesの運用話です。

1. νϟοτϫʔΫʹ͓͚Δ
 Kubernetes on AWS / Kubernetes on AWS at ChatWork SRE Ryo Sakamoto
2. © ChatWork ▸ ೔ຊൃϏδωενϟοτ ▸ λεΫ؅ཧ΍ϏσΦ௨࿩͕Մೳ ▸ ಋೖاۀ174,000ࣾҎ্ʢ※2018೥4݄຤೔࣌఺ʣ
3. © ChatWork ΞδΣϯμ ▸ Kubernetesͷ؀ڥ΍ಈ͍͍ͯΔΞϓϦ ▸ KubernetesͰར༻͍ͯ͠Δπʔϧ ▸ Kubernetesͷ؂ࢹɺϩΪϯά ▸ Kubernetesͷversion up ▸ ·ͱΊ
4. © ChatWork Kubernetesͷར༻ ▸ ϝοηʔδॲཧ෦෼ͷϦϓϨΠε(2016೥12݄) ▸ backen appΛkubernetes Ͱಈ͔͢ ▸ ࡉ͔͍࿩͸ࡢ೥ͷAWS summitͰ…
5. © ChatWork Kubernetesͷ؀ڥ ▸ ؀ڥ AWS ▸ AWSͳͲطଘͷࢿݯΛར༻͔ͨͬͨͨ͠Ί ▸ ߏஙπʔϧ kube-aws ▸ https://github.com/kubernetes-incubator/kube-aws ▸ ϝΠϯϝϯςφ͸mumoshu (chatwork kubernetes ސ໰) ▸ cloudformation(ͱcloud-init)Ͱ·ΔͬͱߏஙͰ͖Δ
6. © ChatWork KubernetesͰಈ͍͍ͯΔ΋ͷ ▸ backend ▸ ࡢ೥ͷAWS summitͰmessage backendΛϦϓϨΠεͨ͠࿩ ▸ ͜ͷϓϩδΣΫτҎ߱(webhook, oauthͳͲ)͸͢΂ͯkubernetes ▸ ϊʔυ਺͸m4.2xlarge * 10୆ఔ౓ ▸ CD؀ڥ(concourse) ▸ spot instance ͳnodepoolΛར༻
7. © ChatWork KubernetesͰར༻͍ͯ͠Δπʔϧ(1) ▸ cluster-autoscaler ▸ podͷauto scaleͰ͸ͳ͘ɺnodeͷauto scaleͯ͘͠ΕΔ ▸ schedulerΛ؂ࢹͯ͠ɺϦιʔε͕଍Γͳ͍pod͕͍ΔͱASGΛૢ࡞ ▸ nodeͷ୆਺࡟ݮʹେ͖͘ߩݙ ▸ σϓϩΠ࣌ͷpodͷೖସ͑ͳͲͰҰ࣌తʹϊʔυ͕଍Γͳ͘ͳΔͱ͖ ΋͞ΒͬͱରԠͯ͘͠ΕΔ
8. © ChatWork cluster-autoscalerͷಈ͖(scale out) controller nodepool api-server scheduler cluster- autoscaler pod (1) watch (2) 
 “fails to be scheduled due to insufficient” (4)scale out (3) 
 set-desired-capacity ൑ྫ
9. © ChatWork cluster-autoscalerͷಈ͖(scale in) controller nodepool api-server scheduler cluster- autoscaler pod (1) watch
 (apiܦ༝) nodeͷ࢖༻཰௿ (3) 
 set-desired-capacity ൑ྫ a b a b nodeͷ࢖༻཰௿ a b a b (4) scale in (2) evict
10. © ChatWork KubernetesͰར༻͍ͯ͠Δπʔϧ(2) ▸ kube2iam ▸ podຖʹroleͷ෇༩ ▸ ௨ৗ͸ΠϯελϯεͷϩʔϧΛར༻ ▸ ෆཁͳpolicy͕෇͘ & ଍Γͳ͍΋ͷ͸APIKEYΛ΋ͨͤΔ͜ͱʹͳΔ ▸ secret͸base64ͳ͚ͩ ▸ एׯෆ҆ఆͰɺkiamʹஔ͖׵͑༧ఆ
11. © ChatWork kube2iam ▸ annotationʹroleΛهࡌ ▸ role͸workerͷroleΛ৴པ͓ͯ͘͠ ▸ worker͕asuumeͰ͖ΔΑ͏ʹ͓ͯ͘͠ ▸ pod͕ɺAWSͷAPIΛར༻͠Α͏ͱ͢ΔͱɺmetadataʹΞΫηε͢Δ ▸ metadata΁ͷΞΫηεΛiptablesͰkube2iamʹసૹ ▸ kube2iam͕annotationͷroleͷΫϨσϯγϟϧΛൃߦ
12. © ChatWork kube2iam app kube2iam 1. credentialͷൃߦ(ec2-metadata) 2. iptablesͰkube2iamͷpodʹϦΫΤετ͕సૹ 3. credentialͷൃߦ pod ൑ྫ
13. © ChatWork Kubernetesͷ؂ࢹ ▸ datadog only ▸ daemonsetͰ഑ஔ ▸ not k8sͳ؀ڥͷ؂ࢹͱ౷Ұ͍ͨ͠ & prometheusͷ؅ཧΛͨ͘͠ͳ͍ ▸ prometheusͷΑ͏ʹΤϯυϙΠϯτΛੜ΍͢ͷͰ͸ͳ͘ɺ֤ϗετͷ statsdʹૹ৴ ▸ version 6Λར༻ ▸ v5ͰϝτϦΫε͕͚͍ܽͯͨ(ϝτϦΫεᷓΕ)͕ɺv6Ͱ͚ܽͳ͘ͳͬͨ
14. © ChatWork datadogͷlive container monitoring
15. © ChatWork KubernetesͷϩΪϯά ▸ fluentd + stackdriver ▸ fluentdΛdaemonsetͰ഑ஔ ▸ ֤ίϯςφ͸ϗετͷಛఆͷ৔ॴʹstdoutΛు͖ग़͍ͯ͠Δ ▸ audit-log΋fluentdͰstackdriverʹૹ৴ ▸ S3Ͱ΋Α͔͕ͬͨɺKubernetesΛಋೖͨ࣌͠ʹ͸Athena͸ग़ͨ͹͔Γ ▸ stackdriver + bigquery΋Ұ෦ͷϩάͰಋೖ
16. © ChatWork Kubernetesͷversion up ▸ kube-awsͰ؅ཧ͍ͯ͠ΔҎ্ɺϚωʔδυͳversion up͸Ͱ͖ͳ͍ ▸ version up΍kubernetes ౷߹Λߦͬͨ ▸ version up 1.7 -> 1.8, 1.5 -> 1.8 && 1.8ԽͷλΠϛϯάͰΫϥελ౷߹ ▸ νϟοτϫʔΫͰ͸·ͩingress͸ར༻Ͱ͖͍ͯͳͯ͘ɺELB + NodePort ▸ ͳͷͰɺversion up͸ELBʹ৽چ྆ํΛͿΒԼ͛ͯɺݹ͍ํΛޙୀ ▸ ࠓͷΞϓϦέʔγϣϯͱͯ͠͸onlineͰversion up׬ྃ
17. © ChatWork version up old k8s pod ൑ྫ app chatwork web NodePort
18. © ChatWork version up old k8s pod ൑ྫ app app chatwork web NodePort NodePort new k8s
19. © ChatWork version up pod ൑ྫ app chatwork web NodePort new k8s
20. © ChatWork KubernetesͰࠓޙ΍Γ͍ͨ(1) ▸ EKS ▸ kube-awsʹ૊Έࠐ·ΕΔ༧ఆ ▸ ΍ͬͺΓϩʔϦϯάΞοϓσʔτ͍ͨ͠ ▸ service mesh ▸ envoyͷಋೖ ▸ istio, linkerd ▸ grpc loadbalancer -> envoy, nginx-ingress-controller
21. © ChatWork KubernetesͰࠓޙ΍Γ͍ͨ(2) ▸ prometheusͷಋೖ ▸ hpa͸σϑΥϧτͰ͸cpu͔͠ͳ͍ͷͰ͔ͭʹ͍͘ ▸ cpuͰ͸ͳ͘kafkaͷeventͷ٧·Γ۩߹Ͱscale in/out͍ͨ͠ ▸ datadogͰ΋apiΛ࢖ͬͯͰ͖Δ͚Ͳɺdatadogͱͷ઀ଓෆ҆ ▸ ϓϥοτϑΥʔϜԽ ▸ openFaaSͳͲ
22. © ChatWork EKS΁ͷظ଴ ▸ preview൛Λར༻͍͍ͤͯͨͩͨ͞ ▸ workerͷ௥Ճ͕͕͕͕͕….configmapܦ༝Ͱొ࿥͢Δɺͱ͍͏ํ๏ͩͬͨ ▸ ͜Εͩͱkubernetesͷ֎ͰɺΫϥελߏங͕ด͡ͳ͍ ▸ AWSͷϦιʔε׆༻(IAMɺVPC)ͳͲظ଴ ▸ fargateͰnodeͦͷ΋ͷΛҙࣝ͠ͳ͍ͷ͸͍͍͕ɺlogging΍؂ࢹ͸…
23. © ChatWork ·ͱΊ ▸ νϟοτϫʔΫͷKubernetes؀ڥʹ͍ͭͯͷ࿩ ▸ ͍Ζ͍Ζ΍Γ͍ͨ͜ͱ͸͋Δ ▸ EKSʹ΋ظ଴ ▸ controll plane͕Ϛωʔδυ͞ΕΔ҆৺ײ
24. © ChatWork ΤϯδχΞืूத http://corp.chatwork.com/ja/recruit/ ▸ ओମੑΛ࣋ͪɺࣗΒߦಈͰ͖Δ ▸ ଞऀΛೝΊɺଚॏͰ͖Δ ▸ ৘ใΛूΊɺڞ༗Ͱ͖Δ ͱ͍͏ํΛ׻ܴ͠·͢ʂ