Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
チャットワークにおけるKubernetesOnAWS.pdf
Search
Ryo Sakamoto
May 28, 2018
0
75
チャットワークにおけるKubernetesOnAWS.pdf
Ryo Sakamoto
May 28, 2018
Tweet
Share
More Decks by Ryo Sakamoto
See All by Ryo Sakamoto
いろいろなAWSアカウントのArgo CDを統合した話
cwsakamoto
1
760
ArgoCDとGitHub Self Hosted Runnerを使って リリース時間を1/4にした話
cwsakamoto
0
2k
Adventure around Kubernetes at Chatwork
cwsakamoto
6
7.6k
チャットワークにおけるKubernetesOnAWS.pdf
cwsakamoto
0
83
Kubernetes on AWS at Chatwork
cwsakamoto
0
1.6k
Featured
See All Featured
StorybookのUI Testing Handbookを読んだ
zakiyama
26
5.2k
5 minutes of I Can Smell Your CMS
philhawksworth
202
19k
Building an army of robots
kneath
302
42k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
32
1.8k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
3
370
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
27
790
Why Our Code Smells
bkeepers
PRO
334
57k
Making Projects Easy
brettharned
115
5.9k
GraphQLとの向き合い方2022年版
quramy
43
13k
Why You Should Never Use an ORM
jnunemaker
PRO
53
9k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
131
33k
Stop Working from a Prison Cell
hatefulcrawdad
267
20k
Transcript
νϟοτϫʔΫʹ͓͚Δ Kubernetes on AWS / Kubernetes on AWS at ChatWork
SRE Ryo Sakamoto
© ChatWork ▸ ຊൃϏδωενϟοτ ▸ λεΫཧϏσΦ௨͕Մೳ ▸ ಋೖاۀ174,000ࣾҎ্ʢ※20184݄࣌ʣ
© ChatWork ΞδΣϯμ ▸ Kubernetesͷڥಈ͍͍ͯΔΞϓϦ ▸ KubernetesͰར༻͍ͯ͠Δπʔϧ ▸ KubernetesͷࢹɺϩΪϯά ▸
Kubernetesͷversion up ▸ ·ͱΊ
© ChatWork Kubernetesͷར༻ ▸ ϝοηʔδॲཧ෦ͷϦϓϨΠε(201612݄) ▸ backen appΛkubernetes Ͱಈ͔͢ ▸
ࡉ͔͍ࡢͷAWS summitͰ…
© ChatWork Kubernetesͷڥ ▸ ڥ AWS ▸ AWSͳͲطଘͷࢿݯΛར༻͔ͨͬͨͨ͠Ί ▸ ߏஙπʔϧ
kube-aws ▸ https://github.com/kubernetes-incubator/kube-aws ▸ ϝΠϯϝϯςφmumoshu (chatwork kubernetes ސ) ▸ cloudformation(ͱcloud-init)Ͱ·ΔͬͱߏஙͰ͖Δ
© ChatWork KubernetesͰಈ͍͍ͯΔͷ ▸ backend ▸ ࡢͷAWS summitͰmessage backendΛϦϓϨΠεͨ͠ ▸
͜ͷϓϩδΣΫτҎ߱(webhook, oauthͳͲ)ͯ͢kubernetes ▸ ϊʔυm4.2xlarge * 10ఔ ▸ CDڥ(concourse) ▸ spot instance ͳnodepoolΛར༻
© ChatWork KubernetesͰར༻͍ͯ͠Δπʔϧ(1) ▸ cluster-autoscaler ▸ podͷauto scaleͰͳ͘ɺnodeͷauto scaleͯ͘͠ΕΔ ▸
schedulerΛࢹͯ͠ɺϦιʔε͕Γͳ͍pod͕͍ΔͱASGΛૢ࡞ ▸ nodeͷݮʹେ͖͘ߩݙ ▸ σϓϩΠ࣌ͷpodͷೖସ͑ͳͲͰҰ࣌తʹϊʔυ͕Γͳ͘ͳΔͱ͖ ͞ΒͬͱରԠͯ͘͠ΕΔ
© ChatWork cluster-autoscalerͷಈ͖(scale out) controller nodepool api-server scheduler cluster- autoscaler
pod (1) watch (2) “fails to be scheduled due to insufficient” (4)scale out (3) set-desired-capacity ྫ
© ChatWork cluster-autoscalerͷಈ͖(scale in) controller nodepool api-server scheduler cluster- autoscaler
pod (1) watch (apiܦ༝) nodeͷ༻ (3) set-desired-capacity ྫ a b a b nodeͷ༻ a b a b (4) scale in (2) evict
© ChatWork KubernetesͰར༻͍ͯ͠Δπʔϧ(2) ▸ kube2iam ▸ podຖʹroleͷ༩ ▸ ௨ৗΠϯελϯεͷϩʔϧΛར༻ ▸
ෆཁͳpolicy͕͘ & Γͳ͍ͷAPIKEYΛͨͤΔ͜ͱʹͳΔ ▸ secretbase64ͳ͚ͩ ▸ एׯෆ҆ఆͰɺkiamʹஔ͖͑༧ఆ
© ChatWork kube2iam ▸ annotationʹroleΛهࡌ ▸ roleworkerͷroleΛ৴པ͓ͯ͘͠ ▸ worker͕asuumeͰ͖ΔΑ͏ʹ͓ͯ͘͠ ▸
pod͕ɺAWSͷAPIΛར༻͠Α͏ͱ͢ΔͱɺmetadataʹΞΫηε͢Δ ▸ metadataͷΞΫηεΛiptablesͰkube2iamʹసૹ ▸ kube2iam͕annotationͷroleͷΫϨσϯγϟϧΛൃߦ
© ChatWork kube2iam app kube2iam 1. credentialͷൃߦ(ec2-metadata) 2. iptablesͰkube2iamͷpodʹϦΫΤετ͕సૹ 3.
credentialͷൃߦ pod ྫ
© ChatWork Kubernetesͷࢹ ▸ datadog only ▸ daemonsetͰஔ ▸ not
k8sͳڥͷࢹͱ౷Ұ͍ͨ͠ & prometheusͷཧΛͨ͘͠ͳ͍ ▸ prometheusͷΑ͏ʹΤϯυϙΠϯτΛੜ͢ͷͰͳ͘ɺ֤ϗετͷ statsdʹૹ৴ ▸ version 6Λར༻ ▸ v5ͰϝτϦΫε͕͚͍ܽͯͨ(ϝτϦΫεᷓΕ)͕ɺv6Ͱ͚ܽͳ͘ͳͬͨ
© ChatWork datadogͷlive container monitoring
© ChatWork KubernetesͷϩΪϯά ▸ fluentd + stackdriver ▸ fluentdΛdaemonsetͰஔ ▸
֤ίϯςφϗετͷಛఆͷॴʹstdoutΛు͖ग़͍ͯ͠Δ ▸ audit-logfluentdͰstackdriverʹૹ৴ ▸ S3ͰΑ͔͕ͬͨɺKubernetesΛಋೖͨ࣌͠ʹAthenaग़͔ͨΓ ▸ stackdriver + bigqueryҰ෦ͷϩάͰಋೖ
© ChatWork Kubernetesͷversion up ▸ kube-awsͰཧ͍ͯ͠ΔҎ্ɺϚωʔδυͳversion upͰ͖ͳ͍ ▸ version upkubernetes
౷߹Λߦͬͨ ▸ version up 1.7 -> 1.8, 1.5 -> 1.8 && 1.8ԽͷλΠϛϯάͰΫϥελ౷߹ ▸ νϟοτϫʔΫͰ·ͩingressར༻Ͱ͖͍ͯͳͯ͘ɺELB + NodePort ▸ ͳͷͰɺversion upELBʹ৽چ྆ํΛͿΒԼ͛ͯɺݹ͍ํΛޙୀ ▸ ࠓͷΞϓϦέʔγϣϯͱͯ͠onlineͰversion upྃ
© ChatWork version up old k8s pod ྫ app chatwork
web NodePort
© ChatWork version up old k8s pod ྫ app app
chatwork web NodePort NodePort new k8s
© ChatWork version up pod ྫ app chatwork web NodePort
new k8s
© ChatWork KubernetesͰࠓޙΓ͍ͨ(1) ▸ EKS ▸ kube-awsʹΈࠐ·ΕΔ༧ఆ ▸ ͬͺΓϩʔϦϯάΞοϓσʔτ͍ͨ͠ ▸
service mesh ▸ envoyͷಋೖ ▸ istio, linkerd ▸ grpc loadbalancer -> envoy, nginx-ingress-controller
© ChatWork KubernetesͰࠓޙΓ͍ͨ(2) ▸ prometheusͷಋೖ ▸ hpaσϑΥϧτͰcpu͔͠ͳ͍ͷͰ͔ͭʹ͍͘ ▸ cpuͰͳ͘kafkaͷeventͷ٧·Γ۩߹Ͱscale in/out͍ͨ͠
▸ datadogͰapiΛͬͯͰ͖Δ͚Ͳɺdatadogͱͷଓෆ҆ ▸ ϓϥοτϑΥʔϜԽ ▸ openFaaSͳͲ
© ChatWork EKSͷظ ▸ preview൛Λར༻͍͍ͤͯͨͩͨ͞ ▸ workerͷՃ͕͕͕͕͕….configmapܦ༝Ͱొ͢Δɺͱ͍͏ํ๏ͩͬͨ ▸ ͜Εͩͱkubernetesͷ֎ͰɺΫϥελߏங͕ด͡ͳ͍ ▸
AWSͷϦιʔε׆༻(IAMɺVPC)ͳͲظ ▸ fargateͰnodeͦͷͷΛҙࣝ͠ͳ͍ͷ͍͍͕ɺloggingࢹ…
© ChatWork ·ͱΊ ▸ νϟοτϫʔΫͷKubernetesڥʹ͍ͭͯͷ ▸ ͍Ζ͍ΖΓ͍ͨ͜ͱ͋Δ ▸ EKSʹظ ▸
controll plane͕Ϛωʔδυ͞ΕΔ҆৺ײ
© ChatWork ΤϯδχΞืूத http://corp.chatwork.com/ja/recruit/ ▸ ओମੑΛ࣋ͪɺࣗΒߦಈͰ͖Δ ▸ ଞऀΛೝΊɺଚॏͰ͖Δ ▸ ใΛूΊɺڞ༗Ͱ͖Δ
ͱ͍͏ํΛܴ͠·͢ʂ