,VCFSOFUFTΛΊ͙Δݥ+"846(ίϯςφࢧ෦$IBUXPSL43&νʔϜࡔຊ
View Slide
© Chatworkࣗݾհ2▸ ࡔຊ ྒ (͔͞ͱɹΓΐ͏)▸ Chatwork SREΤϯδχΞ▸ ϥϯχϯάΤϯδχΞ▸ ϑϧϚϥιϯ͕Α͏͘3࣌ؒ17͙Β͍(20193݄)▸ 2019ͷߦڑ2700km͙Β͍▸ NY- LA4000kmͳͷͰ, ೦ͳ͕ΒΞϝϦΧԣஅͰ͖ͣ
▸ ϏδωενϟοταʔϏε▸ άϧʔϓνϟοτɺλεΫཧɺ ϑΝΠϧڞ༗ɺϏσΦɾԻ௨▸ ಋೖاۀ 246,000 ࣾ (201912݄ݱࡏ)
© ChatworkΞδΣϯμ4▸ ChatworkͷKubernetesͷྺ࢙ΛৼΓฦΓͳ͕Βݱࡏͷӡ༻ʹ͍ͭͯ▸ ྺ࢙▸ ӡ༻πʔϧ▸ ChatworkͷKubernetes CI/CD
© Chatworkҙ5Fargateͷ͋Γ·ͤΜ
© Chatworkલఏ$IBUXPSLͷ,VCFSOFUFTར༻ܗଶ6▸ Ϛϧνςφϯτ▸ ΫϥελࣗମSRE͕࡞Δ▸ ཧܥΞϓϦέʔγϣϯ(Datadog, Fluentd ͳͲ) -> SRE▸ αʔϏεΞϓϦέʔγϣϯ -> DevνʔϜ
© Chatwork$IBUXPSLͷ,VCFSOFUFTͷྺ࢙7▸ ಋೖ2016▸ ࠷ॳͷProductionͷόʔδϣϯ1.5▸ ϝοηʔδϯάγεςϜͷ৽ϓϩδΣΫτ▸ AWS Summit Tokyo 2017 Ͱൃද▸ ͦͷޙԿ͔ΫϥελҠߦ(όʔδϣϯΞοϓ)Λͭͭ͠ɺݱࡏʹࢸΔ▸ ࣗલKubernetes, EKSͷࠞ߹ঢ়ଶ
© Chatwork&,4ͷಋೖ8▸ EKSར༻લkube-awsͱ͍͏ͷΛར༻ͯ͠, ࣗલͰϗεςΟϯά▸ https://github.com/kubernetes-incubator/kube-aws▸ EKSͷొʹΑΓkube-awsͷϢʔβݮ…▸ ࡉ͔͍ػೳʹόά͕ଟ͘, దٓPRରԠͭͭࣗ͠લύονӡ༻▸ EKSࣗମίϯτϩʔϥʔ෦͚ͩͰ, ͍ʹ͍͘ͳ͊…ͱϓϨϏϡʔͰࢥ͍͕ͬͯͨeksctlʹΑΓͦ͏͍͏ͱ͜Ζ͕·Δͬͱָʹ
© ChatworkFLTDUMͱLVCFBXTͷൺֱ9▸ eksctl▸ ։ൃ͕ૣ͍▸ جຊతʹΫϥελͷߏupdateͰ͖ͳ͍▸ ϊʔυάϧʔϓͷઃఆมߋΛ͍ͨ͠߹ɺ৽ن࡞ɾچϊʔυάϧʔϓআ▸ ϊʔυάϧʔϓͷߏΛࢼ͍ͯ͠Δͱ͖ͪΐͬͱखؒ
ΦϑΟγϟϧͳͷͰɺEKS৽͍͠ػೳେମAWSͷൃදͱಉ࣌ʹରԠ͞Ε͍ͯΔ
© ChatworkFLTDUMͱLVCFBXTͷൺֱ11▸ kube-aws▸ ΫϥελͷߏupdateՄೳ▸ ϊʔυάϧʔϓͷઃఆมߋΛ͍ͨ͠߹ɺچϊʔυάϧʔϓ͔ΒϩʔϦϯάΞοϓσʔτ(CloudFormationͷupdateͳͷͰ...)▸ EKSΛར༻͢ΔͷͰͳ͘, controller, etcdؚΊͯ࡞▸ ੜKubernetesͳͷͰɺΧελϚΠζੑ͔ͳΓߴ͍▸ KubernetesͷઃఆϑΝΠϧ͕େͳyamlʹͳΔ(Productionͷ686ߦ)
© ChatworkFLTDUMͱLVCFBXTͷൺֱ12▸ ظతʹݟΔͱ, eksctlͷ΄͏͕ϝϦοτେ͖͍▸ Ϣʔβೝূ͕iamͰͰ͖ͨΓ, podʹiam roleΛ͚ΒΕͨΓ, eksͦͷͷͷϝϦοτେ͖͍▸ iam role for podͱͯخ͍͠▸ kube2iam, kiam, kube-aws-iam-controllerͯ͢ӡ༻͖ͯͨ͠ܦݧ▸ ͜Ε͜ΕͰखܰ͋͞Δ͚ΕͲ
ΫϥελΛ࡞Δ͚ͩͳΒ͜ΕͰOK ͔͠͠ɾɾ
© Chatwork,VCFSOFUFTͷӡ༻Ͱେมͳ͜ͱ14▸ Kubernetes version up 3ϲ݄ʹ1ճ▸ ͯ͢ʹै͠ͳ͍ͱͯ͠, ʹ1ճ͙Β͍version up▸ test, stg, productionڥͷ࡞Γม͑ɺσϓϩΠ͕සൃʂ▸ EKSʹϩʔϦϯάΞοϓάϨʔυ͋Δ͚ΕͲ....▸ खಈ(υΩϡϝϯτରԠ)Ͱӡ༻ෛՙ͕ߴ͗͢Δ
͜ΕΛΓӽ͑Δӡ༻ɾΈ͕ඞཁ
© Chatwork,VCFSOFUFTͷӡ༻Λࢧ͑Δπʔϧͨͪ16▸ eksctl (kube-aws)▸ Ϋϥελࣗମ͜ΕͰ࡞▸ Variant▸ https://github.com/mumoshu/variant▸ helm, helmfile▸ https://github.com/roboll/helmfile
© Chatwork7BSJBOU17▸ task runner▸ eksctlͳͲͷίϚϯυΛϥοϓ͢Δ▸ wrapper shellͰ͍͍͕ɺͪ͜ΒyamlͰॻ͚ͯɺύϥϝʔλͳͲͷࢦఆΓ͍͢▸ dockerϕʔεͷtask runner͕ఆٛͰ͖ɺڥґଘΛݮΒͤΔ▸ ڥͱίϚϯυΛηοτʹͰ͖Δ
© ChatworkFLTDUM͚ͩͰͳͥͩΊͳͷ͔18▸ eksctl͚ͩͰेͳ߹ଟ͍▸ ͱΓ͋͑ͣࢼݧڥΛىಈͤ͞ΔʹίϚϯυ͚ͩͰे▸ ͨͩ͠ࡉ͔͍ઃఆΛ͠Α͏ͱ͢Δͱ, ϑΝΠϧͰͷઃఆͷ΄͏ָ͕▸ sampleҎ֎ͷυΩϡϝϯτͳ͍ͷͰ, ҙ▸ https://github.com/weaveworks/eksctl/blob/master/pkg/apis/eksctl.io/v1alpha5/types.go ͔Β୳͔͢͠ͳ͍ͱ͖͋Δ▸ ઃఆΛͤΔVariantͰϥοϓͯ͠, eksctl༻ઃఆϑΝΠϧͷΛஔ
# Ϋϥελ໊# αϒωοτ# cluster-autoscalerͷiam roleͭ͘Δ# eksຊମ(ϊʔυάϧʔϓআ͘)ͷeksctlͷઃఆϑΝΠϧ
© ChatworkFLTDUM͚ͩͰͳͥͩΊͳͷ͔20▸ Ϋϥελ࡞Δ͚ͩͳΒ, ୯ମར༻Ͱ͍͚ͳ͘ͳ͍▸ FluxͩΕ͕ೖΕΔͷɺͷରԠ▸ Flux ... GitOps tool ޙड़͠·͢▸ GitOpsͰઃఆͷөΞϓϦέʔγϣϯͷσϓϩΠΛ͢Δ▸ Ͱ, GitOpsΛ࣮ݱ͢ΔͨΊͷπʔϧҰମ୭͕ೖΕΔͷ͔...!▸ GitOps Kubernetes෦ʹ࣮ݱ͢ΔͨΊͷΞϓϦέʔγϣϯ͕͍Δ
© Chatwork 21▸ eksctlίϚϯυͰFluxΛ͍ΕΔ͜ͱՄೳ▸ υΩϡϝϯτϕʔεͰରԠͰ͖Δ͕ɺԼهͷϑϩʔΛ·ͱΊ͍ͨ▸ eksctlͷઃఆϑΝΠϧ࡞▸ eksctl create cluster▸ eksctl enable repo ... (Fluxͷinstall)▸ ͜ΕΒΛvariantͰλεΫ੍ޚ(Infrastructure as codeͷҰ)FLTDUM͚ͩͰͳͥͩΊͳͷ͔
© Chatwork 22FLTDUMΛ7BSJBOUͰXSBQͨ݁͠Ռ͜Ε͚ͩͰEKSڥOKhttps://en.wikipedia.org/wiki/AchlisEKS࡞ίϚϯυΛachlisͱ໋໊
© Chatwork,VCFSOFUFTͷӡ༻Ͱେมͳ͜ͱ23▸ Kubernetes version up 3ϲ݄ʹ1ճ▸ ͯ͢ʹै͠ͳ͍ͱͯ͠, ʹ1ճ͙Β͍version up▸ test, stg, productionڥͷ࡞Γม͑ɺσϓϩΠ͕සൃʂ▸ EKSʹϩʔϦϯάΞοϓάϨʔυ͋Δ͚ΕͲ....▸ खಈ(υΩϡϝϯτରԠ)Ͱӡ༻ෛՙ͕ߴ͗͢Δ
͜͜·ͰKubernetesࣗମͷߏங͔͜͜ΒKubernetes෦ͷߏங
© Chatwork 25▸ https://toris.io/2019/12/what-i-think-about-when-i-think-about-kubernetes-and-ecs/▸ (ͳΜ͔ଜ্य़थͷλΠτϧͬΆ͍...)▸ EKSϚωʔδυͰ͕͢ɺϚωʔδϝϯτίϯιʔϧͰ࡞͚ͨͩ͠Ͱ΄΅ԿͰ͖·ͤΜ▸ ϊʔυάϧʔϓ(࣮ࡍʹίϯςφ͕ಈ͘ڥ)Λ࡞ͬͯɺ·ͩϩάૹΕͣɺϝτϦΫεݟΕͣɺͰ͢▸ ͳͷͰɺEKSͰཧܥͷΞϓϦέʔγϣϯ(ex. Datadog)ͷӡ༻͕ඞਢ&$4ͱͷҧ͍
© ChatworkIFMN IFMNpMF26▸ helm▸ The package manager for Kubernetes▸ KubernetesʹΞϓϦέʔγϣϯΛೖΕΔࡍʹɺඞཁͳͷΛύοέʔδԽͯ͠ɺೖΕͯ͘ΕΔπʔϧ▸ ChatworkͰ΄ͱΜͲΞϓϦέʔγϣϯΛHelmΛར༻ͯ͠σϓϩΠ▸ Chartࣗମڥʹґଘ͠ͳ͍ʂ▸ σϓϩΠͷ࠶ݱੑ(ࣗલKubernetes(kube-aws) -> EKSͰͦͷ··)
© ChatworkIFMN IFMNpMF27▸ helmfile▸ helm chartΛͲ͏ͬͯద༻͢Δͷ͔Λએݴతʹॻ͚Δ▸ helmͰσϓϩΠ͢ΔࡍͷΞϓϦέʔγϣϯͷઃఆɺhelmίϚϯυͷΦϓγϣϯͳͲΛҰׅͰهࡌ͠ɺhelmͰσϓϩΠͰ͖Δπʔϧ▸ https://github.com/helm/charts/tree/master/stable▸ ͍ΖΜͳΞϓϦέʔγϣϯͷhelm chart▸ https://developers.freee.co.jp/entry/2019/12/03/122657
© ChatworkͳͥIFMNpMF͕ඞཁͳͷ͔28▸ ΞϓϦέʔγϣϯ͝ͱʹ͍͍ͨΦϓγϣϯ(લॲཧ)͕มΘΔ▸ ྫ͑)ΞϓϦέʔγϣϯಉ࢜Ͱґଘؔͷ͋Δͷ, ͳͲ▸ helmͷઃఆϑΝΠϧ͕ڥ͝ͱ(test, stg, prod)ͰมΘΔ▸ ྫ͑)AWSͷΞΧϯτ͝ͱʹҧ͏, ϩάͷు͖ग़͠ઌ͕ҧ͏▸ go template ͑ΔͷͰ, helmͷઃఆΛಈతʹੜ͢Δ͜ͱՄೳ▸ ChatworkͰɺ͞ΒʹVariantͰhelmfileΛแΈࠐΈ
© Chatwork 29IFMNpMF IFMNΛ7BSJBOUͰXSBQͨ݁͠Ռ͜Ε͚ͩͰKubernetes෦OK
© Chatwork 30͍Ζ͍Ζ7BSJBOUͰXSBQͨ݁͠Ռ
© Chatwork,VCFSOFUFTͷӡ༻Ͱେมͳ͜ͱ31▸ Kubernetes version up 3ϲ݄ʹ1ճ▸ ͯ͢ʹै͠ͳ͍ͱͯ͠, ʹ1ճ͙Β͍version up▸ test, stg, productionڥͷ࡞Γม͑ɺσϓϩΠ͕සൃʂ▸ EKSʹϩʔϦϯάΞοϓάϨʔυ͋Δ͚ΕͲ....▸ खಈ(υΩϡϝϯτରԠ)Ͱӡ༻ෛՙ͕ߴ͗͢Δ
͜͜·ͰKubernetesӡ༻͔͜͜ΒCI/CD
© Chatwork,VCFSOFUFTͷӡ༻ͰҰ൪େมͳ͜ͱ33▸ Kubernetes version up 3ϲ݄ʹ1ճ▸ ͯ͢ʹै͠ͳ͍ͱͯ͠, ʹ1ճ͙Β͍version up▸ test, stg, productionڥͷ࡞Γม͑ɺσϓϩΠ͕සൃʂ▸ EKSʹϩʔϦϯάΞοϓάϨʔυ͋Δ͚ΕͲ....▸ खಈ(υΩϡϝϯτରԠ)Ͱӡ༻ෛՙ͕ߴ͗͢Δ
© Chatwork,VCFSOFUFTڥͷ$*$%34▸ KubernetesͰͷCI/CD▸ αʔϏεͷΞϓϦέʔγϣϯ▸ Concourse CIΛར༻ͯ͠CI/CD▸ ͯ͢ΛyamlܗࣜͷpipelineͰఆٛͰ͖Δπʔϧ
© Chatwork,VCFSOFUFTڥͷ$*$%35▸ ࠔͬͨ▸ CIͳΓϩʔΧϧ͔ΒσϓϩΠ͢Δ߹, ςϯϓϨʔτԽ͍ͯ͠Δͱͯ͠ɺΫϥελ͕Ͱ͖Δͨͼʹ͖ઌΛม͑Δඞཁ͕͋Δ▸ ͦͦαʔϏεΞϓϦͷσϓϩΠΛߦ͏Concourse CIࣗΛͲ͜Ͱӡ༻͢Δͷ͔
© Chatwork(JU0QTʂ36https://www.weave.works/technologies/gitops/
© Chatwork(JU0QTʂ!$IBUXPSL37Argoͷ෦·ͩରԠͰ͖͍ͯͳ͍
© Chatwork$IBUXPSLͷ(JU0QTߏ38▸ Flux + Argo CDͷϋΠϒϦουߏ▸ Flux▸ ѻ͑ΔͷݪଇmanifestͷΈͰ, 1flux - 1repo - 1branch▸ Argo CDࣗͷΠϯετʔϧnamespace, aws-auth(eksಠࣗ)ͳͲmanifestద༻Ͱ͋·Γมߋ͕ͳ͍ͷ▸ Argo CD▸ GUI͕͋ͬͨΓ, helmʹରԠ͍ͯͨ͠Γɺͱػೳ͕ଟ͍͍͘͢
© Chatwork$IBUXPSLͷ(JU0QTߏ39▸ αʔϏεͷΞϓϦέʔγϣϯ͚ͩͰͳ͘ɺཧܥͷΞϓϦͷΞϓϦέʔγϣϯGitOpsͰద༻͢Δํࣜʹมߋ༧ఆ▸ Kubernetes͕Ͱ͖ΔͱɺඞཁͳΞϓϦέʔγϣϯͯ͢উखʹೖ͍ͬͯΔঢ়ଶ▸ αʔϏεͷΞϓϦέʔγϣϯʹؔͯͪ͠ΖΜΓସ͑ͳͲ͕ඞཁ
© Chatwork(JU0QTΛΊ͙Δڥ40
© Chatwork,VCFSOFUFTͷӡ༻Ͱେมͳ͜ͱ41▸ Kubernetes version up 3ϲ݄ʹ1ճ▸ ͯ͢ʹै͠ͳ͍ͱͯ͠, ʹ1ճ͙Β͍version up▸ test, stg, productionڥͷ࡞Γม͑ɺσϓϩΠ͕සൃʂ▸ EKSʹϩʔϦϯάΞοϓάϨʔυ͋Δ͚ΕͲ....▸ खಈ(υΩϡϝϯτରԠ)Ͱӡ༻ෛՙ͕ߴ͗͢Δ
© Chatwork·ͱΊ42▸ Chatworkʹ͓͚ΔKubernetesͷӡ༻(ߏங)▸ eksctl, variant, helm, helmfile▸ GitOpsߏ▸ EKSɺֹʂ
cwsakamoto
soracom
salaboy
gonkun
lemon
smt7174
kohbis
.jpg){kind=avatar}
izzii
whywaita
edwardkenfox
yoshiiryo1
ryomaru0825
yrhorita
smashingmag
addyosmani
trallard
jcasabona
jrom
garrettdimon
eileencodes
schacon
danielanewman
malarkey