Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Kubernetes on AWS at Chatwork
Ryo Sakamoto
May 28, 2018
0
1.3k
Kubernetes on AWS at Chatwork
AWS Summit 2018 スタートアップ特設エリアにおける発表内容で、ChatWorkにおけるKubernetesの運用話です。
Ryo Sakamoto
May 28, 2018
Tweet
Share
More Decks by Ryo Sakamoto
See All by Ryo Sakamoto
ArgoCDとGitHub Self Hosted Runnerを使って リリース時間を1/4にした話
cwsakamoto
0
200
Adventure around Kubernetes at Chatwork
cwsakamoto
6
6.8k
チャットワークにおけるKubernetesOnAWS.pdf
cwsakamoto
0
72
チャットワークにおけるKubernetesOnAWS.pdf
cwsakamoto
0
35
Featured
See All Featured
The MySQL Ecosystem @ GitHub 2015
samlambert
240
11k
Typedesign – Prime Four
hannesfritz
34
1.5k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
15
1.2k
Why Our Code Smells
bkeepers
PRO
326
55k
Designing with Data
zakiwarfel
91
4.2k
Debugging Ruby Performance
tmm1
67
11k
Building a Modern Day E-commerce SEO Strategy
aleyda
6
4.5k
Learning to Love Humans: Emotional Interface Design
aarron
263
38k
Art Directing for the Web. Five minutes with CSS Template Areas
malarkey
196
9.8k
Making Projects Easy
brettharned
102
4.8k
Become a Pro
speakerdeck
PRO
6
3.2k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
236
1.1M
Transcript
νϟοτϫʔΫʹ͓͚Δ Kubernetes on AWS / Kubernetes on AWS at ChatWork
SRE Ryo Sakamoto
© ChatWork ▸ ຊൃϏδωενϟοτ ▸ λεΫཧϏσΦ௨͕Մೳ ▸ ಋೖاۀ174,000ࣾҎ্ʢ※20184݄࣌ʣ
© ChatWork ΞδΣϯμ ▸ Kubernetesͷڥಈ͍͍ͯΔΞϓϦ ▸ KubernetesͰར༻͍ͯ͠Δπʔϧ ▸ KubernetesͷࢹɺϩΪϯά ▸
Kubernetesͷversion up ▸ ·ͱΊ
© ChatWork Kubernetesͷར༻ ▸ ϝοηʔδॲཧ෦ͷϦϓϨΠε(201612݄) ▸ backen appΛkubernetes Ͱಈ͔͢ ▸
ࡉ͔͍ࡢͷAWS summitͰ…
© ChatWork Kubernetesͷڥ ▸ ڥ AWS ▸ AWSͳͲطଘͷࢿݯΛར༻͔ͨͬͨͨ͠Ί ▸ ߏஙπʔϧ
kube-aws ▸ https://github.com/kubernetes-incubator/kube-aws ▸ ϝΠϯϝϯςφmumoshu (chatwork kubernetes ސ) ▸ cloudformation(ͱcloud-init)Ͱ·ΔͬͱߏஙͰ͖Δ
© ChatWork KubernetesͰಈ͍͍ͯΔͷ ▸ backend ▸ ࡢͷAWS summitͰmessage backendΛϦϓϨΠεͨ͠ ▸
͜ͷϓϩδΣΫτҎ߱(webhook, oauthͳͲ)ͯ͢kubernetes ▸ ϊʔυm4.2xlarge * 10ఔ ▸ CDڥ(concourse) ▸ spot instance ͳnodepoolΛར༻
© ChatWork KubernetesͰར༻͍ͯ͠Δπʔϧ(1) ▸ cluster-autoscaler ▸ podͷauto scaleͰͳ͘ɺnodeͷauto scaleͯ͘͠ΕΔ ▸
schedulerΛࢹͯ͠ɺϦιʔε͕Γͳ͍pod͕͍ΔͱASGΛૢ࡞ ▸ nodeͷݮʹେ͖͘ߩݙ ▸ σϓϩΠ࣌ͷpodͷೖସ͑ͳͲͰҰ࣌తʹϊʔυ͕Γͳ͘ͳΔͱ͖ ͞ΒͬͱରԠͯ͘͠ΕΔ
© ChatWork cluster-autoscalerͷಈ͖(scale out) controller nodepool api-server scheduler cluster- autoscaler
pod (1) watch (2) “fails to be scheduled due to insufficient” (4)scale out (3) set-desired-capacity ྫ
© ChatWork cluster-autoscalerͷಈ͖(scale in) controller nodepool api-server scheduler cluster- autoscaler
pod (1) watch (apiܦ༝) nodeͷ༻ (3) set-desired-capacity ྫ a b a b nodeͷ༻ a b a b (4) scale in (2) evict
© ChatWork KubernetesͰར༻͍ͯ͠Δπʔϧ(2) ▸ kube2iam ▸ podຖʹroleͷ༩ ▸ ௨ৗΠϯελϯεͷϩʔϧΛར༻ ▸
ෆཁͳpolicy͕͘ & Γͳ͍ͷAPIKEYΛͨͤΔ͜ͱʹͳΔ ▸ secretbase64ͳ͚ͩ ▸ एׯෆ҆ఆͰɺkiamʹஔ͖͑༧ఆ
© ChatWork kube2iam ▸ annotationʹroleΛهࡌ ▸ roleworkerͷroleΛ৴པ͓ͯ͘͠ ▸ worker͕asuumeͰ͖ΔΑ͏ʹ͓ͯ͘͠ ▸
pod͕ɺAWSͷAPIΛར༻͠Α͏ͱ͢ΔͱɺmetadataʹΞΫηε͢Δ ▸ metadataͷΞΫηεΛiptablesͰkube2iamʹసૹ ▸ kube2iam͕annotationͷroleͷΫϨσϯγϟϧΛൃߦ
© ChatWork kube2iam app kube2iam 1. credentialͷൃߦ(ec2-metadata) 2. iptablesͰkube2iamͷpodʹϦΫΤετ͕సૹ 3.
credentialͷൃߦ pod ྫ
© ChatWork Kubernetesͷࢹ ▸ datadog only ▸ daemonsetͰஔ ▸ not
k8sͳڥͷࢹͱ౷Ұ͍ͨ͠ & prometheusͷཧΛͨ͘͠ͳ͍ ▸ prometheusͷΑ͏ʹΤϯυϙΠϯτΛੜ͢ͷͰͳ͘ɺ֤ϗετͷ statsdʹૹ৴ ▸ version 6Λར༻ ▸ v5ͰϝτϦΫε͕͚͍ܽͯͨ(ϝτϦΫεᷓΕ)͕ɺv6Ͱ͚ܽͳ͘ͳͬͨ
© ChatWork datadogͷlive container monitoring
© ChatWork KubernetesͷϩΪϯά ▸ fluentd + stackdriver ▸ fluentdΛdaemonsetͰஔ ▸
֤ίϯςφϗετͷಛఆͷॴʹstdoutΛు͖ग़͍ͯ͠Δ ▸ audit-logfluentdͰstackdriverʹૹ৴ ▸ S3ͰΑ͔͕ͬͨɺKubernetesΛಋೖͨ࣌͠ʹAthenaग़͔ͨΓ ▸ stackdriver + bigqueryҰ෦ͷϩάͰಋೖ
© ChatWork Kubernetesͷversion up ▸ kube-awsͰཧ͍ͯ͠ΔҎ্ɺϚωʔδυͳversion upͰ͖ͳ͍ ▸ version upkubernetes
౷߹Λߦͬͨ ▸ version up 1.7 -> 1.8, 1.5 -> 1.8 && 1.8ԽͷλΠϛϯάͰΫϥελ౷߹ ▸ νϟοτϫʔΫͰ·ͩingressར༻Ͱ͖͍ͯͳͯ͘ɺELB + NodePort ▸ ͳͷͰɺversion upELBʹ৽چ྆ํΛͿΒԼ͛ͯɺݹ͍ํΛޙୀ ▸ ࠓͷΞϓϦέʔγϣϯͱͯ͠onlineͰversion upྃ
© ChatWork version up old k8s pod ྫ app chatwork
web NodePort
© ChatWork version up old k8s pod ྫ app app
chatwork web NodePort NodePort new k8s
© ChatWork version up pod ྫ app chatwork web NodePort
new k8s
© ChatWork KubernetesͰࠓޙΓ͍ͨ(1) ▸ EKS ▸ kube-awsʹΈࠐ·ΕΔ༧ఆ ▸ ͬͺΓϩʔϦϯάΞοϓσʔτ͍ͨ͠ ▸
service mesh ▸ envoyͷಋೖ ▸ istio, linkerd ▸ grpc loadbalancer -> envoy, nginx-ingress-controller
© ChatWork KubernetesͰࠓޙΓ͍ͨ(2) ▸ prometheusͷಋೖ ▸ hpaσϑΥϧτͰcpu͔͠ͳ͍ͷͰ͔ͭʹ͍͘ ▸ cpuͰͳ͘kafkaͷeventͷ٧·Γ۩߹Ͱscale in/out͍ͨ͠
▸ datadogͰapiΛͬͯͰ͖Δ͚Ͳɺdatadogͱͷଓෆ҆ ▸ ϓϥοτϑΥʔϜԽ ▸ openFaaSͳͲ
© ChatWork EKSͷظ ▸ preview൛Λར༻͍͍ͤͯͨͩͨ͞ ▸ workerͷՃ͕͕͕͕͕….configmapܦ༝Ͱొ͢Δɺͱ͍͏ํ๏ͩͬͨ ▸ ͜Εͩͱkubernetesͷ֎ͰɺΫϥελߏங͕ด͡ͳ͍ ▸
AWSͷϦιʔε׆༻(IAMɺVPC)ͳͲظ ▸ fargateͰnodeͦͷͷΛҙࣝ͠ͳ͍ͷ͍͍͕ɺloggingࢹ…
© ChatWork ·ͱΊ ▸ νϟοτϫʔΫͷKubernetesڥʹ͍ͭͯͷ ▸ ͍Ζ͍ΖΓ͍ͨ͜ͱ͋Δ ▸ EKSʹظ ▸
controll plane͕Ϛωʔδυ͞ΕΔ҆৺ײ
© ChatWork ΤϯδχΞืूத http://corp.chatwork.com/ja/recruit/ ▸ ओମੑΛ࣋ͪɺࣗΒߦಈͰ͖Δ ▸ ଞऀΛೝΊɺଚॏͰ͖Δ ▸ ใΛूΊɺڞ༗Ͱ͖Δ
ͱ͍͏ํΛܴ͠·͢ʂ