Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
チャットワークにおけるKubernetesOnAWS.pdf
Search
Ryo Sakamoto
May 28, 2018
0
80
チャットワークにおけるKubernetesOnAWS.pdf
Ryo Sakamoto
May 28, 2018
Tweet
Share
More Decks by Ryo Sakamoto
See All by Ryo Sakamoto
いろいろなAWSアカウントのArgo CDを統合した話
cwsakamoto
1
590
ArgoCDとGitHub Self Hosted Runnerを使って リリース時間を1/4にした話
cwsakamoto
0
1.8k
Adventure around Kubernetes at Chatwork
cwsakamoto
6
7.5k
チャットワークにおけるKubernetesOnAWS.pdf
cwsakamoto
0
66
Kubernetes on AWS at Chatwork
cwsakamoto
0
1.5k
Featured
See All Featured
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
275
13k
Thoughts on Productivity
jonyablonski
62
4k
Producing Creativity
orderedlist
PRO
339
39k
How STYLIGHT went responsive
nonsquared
92
5k
ParisWeb 2013: Learning to Love: Crash Course in Emotional UX Design
dotmariusz
104
6.7k
What’s in a name? Adding method to the madness
productmarketing
PRO
18
2.8k
A Philosophy of Restraint
colly
199
16k
Music & Morning Musume
bryan
43
5.8k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
24
1.7k
Faster Mobile Websites
deanohume
301
30k
Optimising Largest Contentful Paint
csswizardry
14
2.6k
Building Adaptive Systems
keathley
33
2k
Transcript
νϟοτϫʔΫʹ͓͚Δ Kubernetes on AWS SRE Ryo Sakamoto
© ChatWork ▸ ຊൃϏδωενϟοτ ▸ λεΫཧϏσΦ௨͕Մೳ ▸ ಋೖاۀ174,000ࣾҎ্ʢ※20184݄࣌ʣ
© ChatWork ΞδΣϯμ ▸ Kubernetesͷڥಈ͍͍ͯΔΞϓϦ ▸ KubernetesͰར༻͍ͯ͠Δπʔϧ ▸ KubernetesͷࢹɺϩΪϯά ▸
Kubernetesͷversion up ▸ ·ͱΊ
© ChatWork Kubernetesͷར༻ ▸ ϝοηʔδॲཧ෦ͷϦϓϨΠε(201612݄) ▸ backen appΛkubernetes Ͱಈ͔͢ ▸
ࡉ͔͍ࡢͷAWS summitͰ…
© ChatWork Kubernetesͷڥ ▸ ڥ AWS ▸ AWSͳͲطଘͷࢿݯΛར༻͔ͨͬͨͨ͠Ί ▸ ߏஙπʔϧ
kube-aws ▸ https://github.com/kubernetes-incubator/kube-aws ▸ ϝΠϯϝϯςφmumoshu (chatwork kubernetes ސ) ▸ cloudformation(ͱcloud-init)Ͱ·ΔͬͱߏஙͰ͖Δ
© ChatWork KubernetesͰಈ͍͍ͯΔͷ ▸ backend ▸ ࡢͷAWS summitͰmessage backendΛϦϓϨΠεͨ͠ ▸
͜ͷϓϩδΣΫτҎ߱(webhook, oauthͳͲ)ͯ͢kubernetes ▸ ϊʔυm4.2xlarge * 10ఔ ▸ CDڥ(concourse) ▸ spot instance ͳnodepoolΛར༻
© ChatWork KubernetesͰར༻͍ͯ͠Δπʔϧ(1) ▸ cluster-autoscaler ▸ podͷauto scaleͰͳ͘ɺnodeͷauto scaleͯ͘͠ΕΔ ▸
schedulerΛࢹͯ͠ɺϦιʔε͕Γͳ͍pod͕͍ΔͱASGΛૢ࡞ ▸ nodeͷݮʹେ͖͘ߩݙ ▸ σϓϩΠ࣌ͷpodͷೖସ͑ͳͲͰҰ࣌తʹϊʔυ͕Γͳ͘ͳΔͱ͖ ͞ΒͬͱରԠͯ͘͠ΕΔ
© ChatWork cluster-autoscalerͷಈ͖(scale out) controller nodepool api-server scheduler cluster- autoscaler
pod (1) watch (2) “fails to be scheduled due to insufficient” (4)scale out (3) set-desired-capacity ྫ
© ChatWork cluster-autoscalerͷಈ͖(scale in) controller nodepool api-server scheduler cluster- autoscaler
pod (1) watch (apiܦ༝) nodeͷ༻ (3) set-desired-capacity ྫ a b a b nodeͷ༻ a b a b (4) scale in (2) evict
© ChatWork KubernetesͰར༻͍ͯ͠Δπʔϧ(2) ▸ kube2iam ▸ podຖʹroleͷ༩ ▸ ௨ৗΠϯελϯεͷϩʔϧΛར༻ ▸
ෆཁͳpolicy͕͘ & Γͳ͍ͷAPIKEYΛͨͤΔ͜ͱʹͳΔ ▸ secretbase64ͳ͚ͩ ▸ एׯෆ҆ఆͰɺkiamʹஔ͖͑༧ఆ
© ChatWork kube2iam ▸ annotationʹroleΛهࡌ ▸ roleworkerͷroleΛ৴པ͓ͯ͘͠ ▸ worker͕asuumeͰ͖ΔΑ͏ʹ͓ͯ͘͠ ▸
pod͕ɺAWSͷAPIΛར༻͠Α͏ͱ͢ΔͱɺmetadataʹΞΫηε͢Δ ▸ metadataͷΞΫηεΛiptablesͰkube2iamʹసૹ ▸ kube2iam͕annotationͷroleͷΫϨσϯγϟϧΛൃߦ
© ChatWork kube2iam app kube2iam 1. credentialͷൃߦ(ec2-metadata) 2. iptablesͰkube2iamͷpodʹϦΫΤετ͕సૹ 3.
credentialͷൃߦ pod ྫ
© ChatWork Kubernetesͷࢹ ▸ datadog only ▸ daemonsetͰஔ ▸ not
k8sͳڥͷࢹͱ౷Ұ͍ͨ͠ & prometheusͷཧΛͨ͘͠ͳ͍ ▸ prometheusͷΑ͏ʹΤϯυϙΠϯτΛੜ͢ͷͰͳ͘ɺ֤ϗετͷ statsdʹૹ৴ ▸ version 6Λར༻ ▸ v5ͰϝτϦΫε͕͚͍ܽͯͨ(ϝτϦΫεᷓΕ)͕ɺv6Ͱ͚ܽͳ͘ͳͬͨ
© ChatWork datadogͷlive container monitoring
© ChatWork KubernetesͷϩΪϯά ▸ fluentd + stackdriver ▸ fluentdΛdaemonsetͰஔ ▸
֤ίϯςφϗετͷಛఆͷॴʹstdoutΛు͖ग़͍ͯ͠Δ ▸ audit-logfluentdͰstackdriverʹૹ৴ ▸ S3ͰΑ͔͕ͬͨɺKubernetesΛಋೖͨ࣌͠ʹAthenaग़͔ͨΓ ▸ stackdriver + bigqueryҰ෦ͷϩάͰಋೖ
© ChatWork Kubernetesͷversion up ▸ kube-awsͰཧ͍ͯ͠ΔҎ্ɺϚωʔδυͳversion upͰ͖ͳ͍ ▸ version upkubernetes
౷߹Λߦͬͨ ▸ version up 1.7 -> 1.8, 1.5 -> 1.8 && 1.8ԽͷλΠϛϯάͰΫϥελ౷߹ ▸ νϟοτϫʔΫͰ·ͩingressར༻Ͱ͖͍ͯͳͯ͘ɺELB + NodePort ▸ ͳͷͰɺversion upELBʹ৽چ྆ํΛͿΒԼ͛ͯɺݹ͍ํΛޙୀ ▸ ࠓͷΞϓϦέʔγϣϯͱͯ͠onlineͰversion upྃ
© ChatWork version up old k8s pod ྫ app chatwork
web NodePort
© ChatWork version up old k8s pod ྫ app app
chatwork web NodePort NodePort new k8s
© ChatWork version up pod ྫ app chatwork web NodePort
new k8s
© ChatWork KubernetesͰࠓޙΓ͍ͨ(1) ▸ EKS ▸ kube-awsʹΈࠐ·ΕΔ༧ఆ ▸ ͬͺΓϩʔϦϯάΞοϓσʔτ͍ͨ͠ ▸
service mesh ▸ envoyͷಋೖ ▸ istio, linkerd ▸ grpc loadbalancer -> envoy, nginx-ingress-controller
© ChatWork KubernetesͰࠓޙΓ͍ͨ(2) ▸ prometheusͷಋೖ ▸ hpaσϑΥϧτͰcpu͔͠ͳ͍ͷͰ͔ͭʹ͍͘ ▸ cpuͰͳ͘kafkaͷeventͷ٧·Γ۩߹Ͱscale in/out͍ͨ͠
▸ datadogͰapiΛͬͯͰ͖Δ͚Ͳɺdatadogͱͷଓෆ҆ ▸ ϓϥοτϑΥʔϜԽ ▸ openFaaSͳͲ
© ChatWork EKSͷظ ▸ preview൛Λར༻͍͍ͤͯͨͩͨ͞ ▸ workerͷՃ͕͕͕͕͕….configmapܦ༝Ͱొ͢Δɺͱ͍͏ํ๏ͩͬͨ ▸ ͜Εͩͱkubernetesͷ֎ͰɺΫϥελߏங͕ด͡ͳ͍ ▸
AWSͷϦιʔε׆༻(IAMɺVPC)ͳͲظ ▸ fargateͰnodeͦͷͷΛҙࣝ͠ͳ͍ͷ͍͍͕ɺloggingࢹ…
© ChatWork ·ͱΊ ▸ νϟοτϫʔΫͷKubernetesڥʹ͍ͭͯͷ ▸ ͍Ζ͍ΖΓ͍ͨ͜ͱ͋Δ ▸ EKSʹظ ▸
controll plane͕Ϛωʔδυ͞ΕΔ҆৺ײ
© ChatWork ΤϯδχΞืूத http://corp.chatwork.com/ja/recruit/ ▸ ओମੑΛ࣋ͪɺࣗΒߦಈͰ͖Δ ▸ ଞऀΛೝΊɺଚॏͰ͖Δ ▸ ใΛूΊɺڞ༗Ͱ͖Δ
ͱ͍͏ํΛܴ͠·͢ʂ