Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
チャットワークにおけるKubernetesOnAWS.pdf
Search
Ryo Sakamoto
May 28, 2018
0
86
チャットワークにおけるKubernetesOnAWS.pdf
Ryo Sakamoto
May 28, 2018
Tweet
Share
More Decks by Ryo Sakamoto
See All by Ryo Sakamoto
いろいろなAWSアカウントのArgo CDを統合した話
cwsakamoto
1
880
ArgoCDとGitHub Self Hosted Runnerを使って リリース時間を1/4にした話
cwsakamoto
0
2.1k
Adventure around Kubernetes at Chatwork
cwsakamoto
6
7.7k
チャットワークにおけるKubernetesOnAWS.pdf
cwsakamoto
0
77
Kubernetes on AWS at Chatwork
cwsakamoto
0
1.7k
Featured
See All Featured
Building Flexible Design Systems
yeseniaperezcruz
328
38k
The World Runs on Bad Software
bkeepers
PRO
67
11k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
4
430
Building Applications with DynamoDB
mza
93
6.2k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
30
2.2k
YesSQL, Process and Tooling at Scale
rocio
172
14k
Practical Orchestrator
shlominoach
186
10k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
233
17k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.7k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
44
7k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
29
1k
Why Our Code Smells
bkeepers
PRO
336
57k
Transcript
νϟοτϫʔΫʹ͓͚Δ Kubernetes on AWS SRE Ryo Sakamoto
© ChatWork ▸ ຊൃϏδωενϟοτ ▸ λεΫཧϏσΦ௨͕Մೳ ▸ ಋೖاۀ174,000ࣾҎ্ʢ※20184݄࣌ʣ
© ChatWork ΞδΣϯμ ▸ Kubernetesͷڥಈ͍͍ͯΔΞϓϦ ▸ KubernetesͰར༻͍ͯ͠Δπʔϧ ▸ KubernetesͷࢹɺϩΪϯά ▸
Kubernetesͷversion up ▸ ·ͱΊ
© ChatWork Kubernetesͷར༻ ▸ ϝοηʔδॲཧ෦ͷϦϓϨΠε(201612݄) ▸ backen appΛkubernetes Ͱಈ͔͢ ▸
ࡉ͔͍ࡢͷAWS summitͰ…
© ChatWork Kubernetesͷڥ ▸ ڥ AWS ▸ AWSͳͲطଘͷࢿݯΛར༻͔ͨͬͨͨ͠Ί ▸ ߏஙπʔϧ
kube-aws ▸ https://github.com/kubernetes-incubator/kube-aws ▸ ϝΠϯϝϯςφmumoshu (chatwork kubernetes ސ) ▸ cloudformation(ͱcloud-init)Ͱ·ΔͬͱߏஙͰ͖Δ
© ChatWork KubernetesͰಈ͍͍ͯΔͷ ▸ backend ▸ ࡢͷAWS summitͰmessage backendΛϦϓϨΠεͨ͠ ▸
͜ͷϓϩδΣΫτҎ߱(webhook, oauthͳͲ)ͯ͢kubernetes ▸ ϊʔυm4.2xlarge * 10ఔ ▸ CDڥ(concourse) ▸ spot instance ͳnodepoolΛར༻
© ChatWork KubernetesͰར༻͍ͯ͠Δπʔϧ(1) ▸ cluster-autoscaler ▸ podͷauto scaleͰͳ͘ɺnodeͷauto scaleͯ͘͠ΕΔ ▸
schedulerΛࢹͯ͠ɺϦιʔε͕Γͳ͍pod͕͍ΔͱASGΛૢ࡞ ▸ nodeͷݮʹେ͖͘ߩݙ ▸ σϓϩΠ࣌ͷpodͷೖସ͑ͳͲͰҰ࣌తʹϊʔυ͕Γͳ͘ͳΔͱ͖ ͞ΒͬͱରԠͯ͘͠ΕΔ
© ChatWork cluster-autoscalerͷಈ͖(scale out) controller nodepool api-server scheduler cluster- autoscaler
pod (1) watch (2) “fails to be scheduled due to insufficient” (4)scale out (3) set-desired-capacity ྫ
© ChatWork cluster-autoscalerͷಈ͖(scale in) controller nodepool api-server scheduler cluster- autoscaler
pod (1) watch (apiܦ༝) nodeͷ༻ (3) set-desired-capacity ྫ a b a b nodeͷ༻ a b a b (4) scale in (2) evict
© ChatWork KubernetesͰར༻͍ͯ͠Δπʔϧ(2) ▸ kube2iam ▸ podຖʹroleͷ༩ ▸ ௨ৗΠϯελϯεͷϩʔϧΛར༻ ▸
ෆཁͳpolicy͕͘ & Γͳ͍ͷAPIKEYΛͨͤΔ͜ͱʹͳΔ ▸ secretbase64ͳ͚ͩ ▸ एׯෆ҆ఆͰɺkiamʹஔ͖͑༧ఆ
© ChatWork kube2iam ▸ annotationʹroleΛهࡌ ▸ roleworkerͷroleΛ৴པ͓ͯ͘͠ ▸ worker͕asuumeͰ͖ΔΑ͏ʹ͓ͯ͘͠ ▸
pod͕ɺAWSͷAPIΛར༻͠Α͏ͱ͢ΔͱɺmetadataʹΞΫηε͢Δ ▸ metadataͷΞΫηεΛiptablesͰkube2iamʹసૹ ▸ kube2iam͕annotationͷroleͷΫϨσϯγϟϧΛൃߦ
© ChatWork kube2iam app kube2iam 1. credentialͷൃߦ(ec2-metadata) 2. iptablesͰkube2iamͷpodʹϦΫΤετ͕సૹ 3.
credentialͷൃߦ pod ྫ
© ChatWork Kubernetesͷࢹ ▸ datadog only ▸ daemonsetͰஔ ▸ not
k8sͳڥͷࢹͱ౷Ұ͍ͨ͠ & prometheusͷཧΛͨ͘͠ͳ͍ ▸ prometheusͷΑ͏ʹΤϯυϙΠϯτΛੜ͢ͷͰͳ͘ɺ֤ϗετͷ statsdʹૹ৴ ▸ version 6Λར༻ ▸ v5ͰϝτϦΫε͕͚͍ܽͯͨ(ϝτϦΫεᷓΕ)͕ɺv6Ͱ͚ܽͳ͘ͳͬͨ
© ChatWork datadogͷlive container monitoring
© ChatWork KubernetesͷϩΪϯά ▸ fluentd + stackdriver ▸ fluentdΛdaemonsetͰஔ ▸
֤ίϯςφϗετͷಛఆͷॴʹstdoutΛు͖ग़͍ͯ͠Δ ▸ audit-logfluentdͰstackdriverʹૹ৴ ▸ S3ͰΑ͔͕ͬͨɺKubernetesΛಋೖͨ࣌͠ʹAthenaग़͔ͨΓ ▸ stackdriver + bigqueryҰ෦ͷϩάͰಋೖ
© ChatWork Kubernetesͷversion up ▸ kube-awsͰཧ͍ͯ͠ΔҎ্ɺϚωʔδυͳversion upͰ͖ͳ͍ ▸ version upkubernetes
౷߹Λߦͬͨ ▸ version up 1.7 -> 1.8, 1.5 -> 1.8 && 1.8ԽͷλΠϛϯάͰΫϥελ౷߹ ▸ νϟοτϫʔΫͰ·ͩingressར༻Ͱ͖͍ͯͳͯ͘ɺELB + NodePort ▸ ͳͷͰɺversion upELBʹ৽چ྆ํΛͿΒԼ͛ͯɺݹ͍ํΛޙୀ ▸ ࠓͷΞϓϦέʔγϣϯͱͯ͠onlineͰversion upྃ
© ChatWork version up old k8s pod ྫ app chatwork
web NodePort
© ChatWork version up old k8s pod ྫ app app
chatwork web NodePort NodePort new k8s
© ChatWork version up pod ྫ app chatwork web NodePort
new k8s
© ChatWork KubernetesͰࠓޙΓ͍ͨ(1) ▸ EKS ▸ kube-awsʹΈࠐ·ΕΔ༧ఆ ▸ ͬͺΓϩʔϦϯάΞοϓσʔτ͍ͨ͠ ▸
service mesh ▸ envoyͷಋೖ ▸ istio, linkerd ▸ grpc loadbalancer -> envoy, nginx-ingress-controller
© ChatWork KubernetesͰࠓޙΓ͍ͨ(2) ▸ prometheusͷಋೖ ▸ hpaσϑΥϧτͰcpu͔͠ͳ͍ͷͰ͔ͭʹ͍͘ ▸ cpuͰͳ͘kafkaͷeventͷ٧·Γ۩߹Ͱscale in/out͍ͨ͠
▸ datadogͰapiΛͬͯͰ͖Δ͚Ͳɺdatadogͱͷଓෆ҆ ▸ ϓϥοτϑΥʔϜԽ ▸ openFaaSͳͲ
© ChatWork EKSͷظ ▸ preview൛Λར༻͍͍ͤͯͨͩͨ͞ ▸ workerͷՃ͕͕͕͕͕….configmapܦ༝Ͱొ͢Δɺͱ͍͏ํ๏ͩͬͨ ▸ ͜Εͩͱkubernetesͷ֎ͰɺΫϥελߏங͕ด͡ͳ͍ ▸
AWSͷϦιʔε׆༻(IAMɺVPC)ͳͲظ ▸ fargateͰnodeͦͷͷΛҙࣝ͠ͳ͍ͷ͍͍͕ɺloggingࢹ…
© ChatWork ·ͱΊ ▸ νϟοτϫʔΫͷKubernetesڥʹ͍ͭͯͷ ▸ ͍Ζ͍ΖΓ͍ͨ͜ͱ͋Δ ▸ EKSʹظ ▸
controll plane͕Ϛωʔδυ͞ΕΔ҆৺ײ
© ChatWork ΤϯδχΞืूத http://corp.chatwork.com/ja/recruit/ ▸ ओମੑΛ࣋ͪɺࣗΒߦಈͰ͖Δ ▸ ଞऀΛೝΊɺଚॏͰ͖Δ ▸ ใΛूΊɺڞ༗Ͱ͖Δ
ͱ͍͏ํΛܴ͠·͢ʂ