Adventure around Kubernetes at Chatwork

Transcript

1. ,VCFSOFUFTΛΊ͙Δ๯ݥ 
   +"846(ίϯςφࢧ෦
   
   $IBUXPSL
   43&νʔϜ
   ࡔຊ

2. © Chatwork ࣗݾ঺հ 2 ▸ ࡔຊ ྒ (͔͞΋ͱɹΓΐ͏) ▸ Chatwork

   SREΤϯδχΞ ▸ ϥϯχϯάΤϯδχΞ ▸ ϑϧϚϥιϯ͕Α͏΍͘3࣌ؒ17෼͙Β͍(2019೥3݄) ▸ 2019೥ͷ૸ߦڑ཭͸2700km͙Β͍ ▸ NY- LA͸4000kmͳͷͰ, ࢒೦ͳ͕ΒΞϝϦΧԣஅ͸Ͱ͖ͣ

3. ▸ ϏδωενϟοταʔϏε ▸ άϧʔϓνϟοτɺλεΫ؅ཧɺ
 ϑΝΠϧڞ༗ɺϏσΦɾԻ੠௨࿩ ▸ ಋೖاۀ 246,000 ࣾ
 (2019೥12݄ݱࡏ)

4. © Chatwork ΞδΣϯμ 4 ▸ ChatworkͷKubernetesͷྺ࢙ΛৼΓฦΓͳ͕Βݱࡏͷӡ༻ʹ͍ͭͯ ▸ ྺ࢙ ▸ ӡ༻πʔϧ

   ▸ ChatworkͷKubernetes CI/CD

5. © Chatwork ஫ҙ 5 Fargateͷ࿩͸͋Γ·ͤΜ

6. © Chatwork લఏ
   $IBUXPSLͷ,VCFSOFUFTར༻ܗଶ 6 ▸ Ϛϧνςφϯτ ▸ Ϋϥελࣗମ͸SRE͕࡞Δ ▸ ؅ཧܥΞϓϦέʔγϣϯ(Datadog,

   Fluentd ͳͲ) -> SRE ▸ αʔϏεΞϓϦέʔγϣϯ -> DevνʔϜ

7. © Chatwork $IBUXPSLͷ,VCFSOFUFTͷྺ࢙ 7 ▸ ಋೖ͸2016೥ ▸ ࠷ॳͷProductionͷόʔδϣϯ͸1.5 ▸ ϝοηʔδϯάγεςϜͷ࡮৽ϓϩδΣΫτ

   ▸ AWS Summit Tokyo 2017 Ͱൃද ▸ ͦͷޙԿ౓͔ΫϥελҠߦ(όʔδϣϯΞοϓ)Λͭͭ͠ɺݱࡏʹࢸΔ ▸ ࣗલKubernetes, EKSͷࠞ߹ঢ়ଶ

8. © Chatwork &,4ͷಋೖ 8 ▸ EKSར༻લ͸kube-awsͱ͍͏ͷΛར༻ͯ͠, ࣗલͰϗεςΟϯά ▸ https://github.com/kubernetes-incubator/kube-aws ▸

   EKSͷొ৔ʹΑΓkube-awsͷϢʔβݮ… ▸ ࡉ͔͍ػೳʹόά͕ଟ͘, దٓPRରԠͭͭ͠΋ࣗલύονӡ༻ ▸ EKSࣗମ͸ίϯτϩʔϥʔ෦෼͚ͩͰ, ࢖͍ʹ͍͘ͳ͊…ͱϓϨϏϡʔͰ ࢥ͍͕ͬͯͨeksctlʹΑΓͦ͏͍͏ͱ͜Ζ͕·Δͬͱָʹ

9. © Chatwork FLTDUMͱLVCFBXTͷൺֱ 9 ▸ eksctl ▸ ։ൃ͕ૣ͍ ▸ جຊతʹ͸Ϋϥελͷߏ੒͸updateͰ͖ͳ͍

   ▸ ϊʔυάϧʔϓͷઃఆมߋΛ͍ͨ͠৔߹͸ɺ৽ن࡞੒ɾچϊʔυά ϧʔϓ࡟আ ▸ ϊʔυάϧʔϓͷߏ੒Λࢼ͍ͯ͠Δͱ͖͸ͪΐͬͱखؒ

10. ΦϑΟγϟϧͳͷͰɺEKS৽͍͠ػೳ͸େମAWSͷൃදͱಉ࣌ʹରԠ͞Ε͍ͯΔ

11. © Chatwork FLTDUMͱLVCFBXTͷൺֱ 11 ▸ kube-aws ▸ Ϋϥελͷߏ੒͸updateՄೳ ▸ ϊʔυάϧʔϓͷઃఆมߋΛ͍ͨ͠৔߹͸ɺچϊʔυάϧʔϓ͔Β

    ϩʔϦϯάΞοϓσʔτ(CloudFormationͷupdateͳͷͰ...) ▸ EKSΛར༻͢Δ΋ͷͰ͸ͳ͘, controller, etcdؚΊͯ࡞੒ ▸ ੜKubernetesͳͷͰɺΧελϚΠζੑ͸͔ͳΓߴ͍ ▸ KubernetesͷઃఆϑΝΠϧ͕௕େͳyamlʹͳΔ(Productionͷ686ߦ)

12. © Chatwork FLTDUMͱLVCFBXTͷൺֱ 12 ▸ ௕ظతʹݟΔͱ, eksctlͷ΄͏͕ϝϦοτ͸େ͖͍ ▸ Ϣʔβೝূ͕iamͰͰ͖ͨΓ, podʹiam

    roleΛ෇͚ΒΕͨΓ, eksͦͷ΋ͷ ͷϝϦοτ΋େ͖͍ ▸ iam role for pod͸ͱͯ΋خ͍͠ ▸ kube2iam, kiam, kube-aws-iam-controller͢΂ͯӡ༻͖ͯͨ͠ܦݧ ▸ ͜Ε͸͜ΕͰखܰ͞΋͋Δ͚ΕͲ

13. ΫϥελΛ࡞Δ͚ͩͳΒ͜ΕͰOK
 ͔͠͠ɾɾ

14. © Chatwork ,VCFSOFUFTͷӡ༻Ͱେมͳ͜ͱ 14 ▸ Kubernetes version up ͸ 3ϲ݄ʹ1ճ

    ▸ ͢΂ͯʹ௥ै͠ͳ͍ͱͯ͠΋, ൒೥ʹ1ճ͙Β͍͸version up ▸ test, stg, production؀ڥͷ࡞Γม͑ɺσϓϩΠ͕සൃʂ ▸ EKSʹ͸ϩʔϦϯάΞοϓάϨʔυ͋Δ͚ΕͲ.... ▸ खಈ(υΩϡϝϯτରԠ)Ͱ͸ӡ༻ෛՙ͕ߴ͗͢Δ

15. ͜ΕΛ৐Γӽ͑Δӡ༻ɾ࢓૊Έ͕ඞཁ

16. © Chatwork ,VCFSOFUFTͷӡ༻Λࢧ͑Δπʔϧͨͪ 16 ▸ eksctl (kube-aws) ▸ Ϋϥελࣗମ͸͜ΕͰ࡞੒ ▸

    Variant ▸ https://github.com/mumoshu/variant ▸ helm, helmfile ▸ https://github.com/roboll/helmfile

17. © Chatwork 7BSJBOU 17 ▸ task runner ▸ eksctlͳͲͷίϚϯυΛϥοϓ͢Δ ▸

    wrapper shellͰ΋͍͍͕ɺͪ͜Β͸yamlͰॻ͚ͯɺύϥϝʔλͳͲͷࢦ ఆ΋΍Γ΍͍͢ ▸ dockerϕʔεͷtask runner͕ఆٛͰ͖ɺ؀ڥґଘΛݮΒͤΔ ▸ ؀ڥͱίϚϯυΛηοτʹͰ͖Δ

18. © Chatwork FLTDUM͚ͩͰ͸ͳͥͩΊͳͷ͔ 18 ▸ eksctl͚ͩͰ΋े෼ͳ৔߹΋ଟ͍ ▸ ͱΓ͋͑ͣࢼݧ؀ڥΛىಈͤ͞Δʹ͸ίϚϯυ͚ͩͰे෼ ▸ ͨͩ͠ࡉ͔͍ઃఆΛ͠Α͏ͱ͢Δͱ,

    ϑΝΠϧͰͷઃఆͷ΄͏ָ͕ ▸ sampleҎ֎ͷυΩϡϝϯτ͸ͳ͍ͷͰ, ஫ҙ ▸ https://github.com/weaveworks/eksctl/blob/master/pkg/apis/eksctl.io/ v1alpha5/types.go ͔Β୳͔͢͠ͳ͍ͱ͖΋͋Δ ▸ ઃఆΛ౉ͤΔVariantͰϥοϓͯ͠, eksctl༻ઃఆϑΝΠϧͷ஋Λஔ׵

19. # Ϋϥελ໊ # αϒωοτ # cluster-autoscalerͷiam roleͭ͘Δ # eksຊମ(ϊʔυάϧʔϓআ͘)ͷeksctlͷઃఆϑΝΠϧ

20. © Chatwork FLTDUM͚ͩͰ͸ͳͥͩΊͳͷ͔ 20 ▸ Ϋϥελ࡞Δ͚ͩͳΒ, ୯ମར༻Ͱ΋͍͚ͳ͘͸ͳ͍ ▸ FluxͩΕ͕ೖΕΔͷ໰୊ɺ΁ͷରԠ ▸

    Flux ... GitOps tool ޙड़͠·͢ ▸ GitOpsͰઃఆͷ൓ө΍ΞϓϦέʔγϣϯͷσϓϩΠΛ͢Δ ▸ Ͱ͸, GitOpsΛ࣮ݱ͢ΔͨΊͷπʔϧ͸Ұମ୭͕ೖΕΔͷ͔...! ▸ GitOps ͸Kubernetes಺෦ʹ࣮ݱ͢ΔͨΊͷΞϓϦέʔγϣϯ͕͍Δ

21. © Chatwork 21 ▸ eksctlίϚϯυͰFluxΛ͍ΕΔ͜ͱ͸Մೳ ▸ υΩϡϝϯτϕʔεͰ΋ରԠͰ͖Δ͕ɺԼهͷϑϩʔΛ·ͱΊ͍ͨ ▸ eksctlͷઃఆϑΝΠϧ࡞੒ ▸

    eksctl create cluster ▸ eksctl enable repo ... (Fluxͷinstall) ▸ ͜ΕΒΛvariantͰλεΫ੍ޚ(Infrastructure as codeͷҰ؀) FLTDUM͚ͩͰ͸ͳͥͩΊͳͷ͔

22. © Chatwork 22 FLTDUMΛ7BSJBOUͰXSBQͨ݁͠Ռ ͜Ε͚ͩͰEKS؀ڥOK https://en.wikipedia.org/wiki/Achlis EKS࡞੒ίϚϯυΛachlisͱ໋໊

23. © Chatwork ,VCFSOFUFTͷӡ༻Ͱେมͳ͜ͱ 23 ▸ Kubernetes version up ͸ 3ϲ݄ʹ1ճ

    ▸ ͢΂ͯʹ௥ै͠ͳ͍ͱͯ͠΋, ൒೥ʹ1ճ͙Β͍͸version up ▸ test, stg, production؀ڥͷ࡞Γม͑ɺσϓϩΠ͕සൃʂ ▸ EKSʹ͸ϩʔϦϯάΞοϓάϨʔυ͋Δ͚ΕͲ.... ▸ खಈ(υΩϡϝϯτରԠ)Ͱ͸ӡ༻ෛՙ͕ߴ͗͢Δ

24. ͜͜·Ͱ͸Kubernetesࣗମͷߏங ͔͜͜Β͸Kubernetes಺෦ͷߏங

25. © Chatwork 25 ▸ https://toris.io/2019/12/what-i-think-about-when-i-think-about-kubernetes- and-ecs/ ▸ (ͳΜ͔ଜ্य़थͷλΠτϧͬΆ͍...) ▸ EKS͸ϚωʔδυͰ͕͢ɺϚωʔδϝϯτίϯιʔϧͰ࡞੒͚ͨͩ͠Ͱ͸

    ΄΅Կ΋Ͱ͖·ͤΜ ▸ ϊʔυάϧʔϓ(࣮ࡍʹίϯςφ͕ಈ͘؀ڥ)Λ࡞ͬͯ΋ɺ·ͩϩά΋ૹΕ ͣɺϝτϦΫε΋ݟΕͣɺͰ͢ ▸ ͳͷͰɺEKSͰ΋؅ཧܥͷΞϓϦέʔγϣϯ(ex. Datadog)ͷӡ༻͕ඞਢ &$4ͱͷҧ͍

26. © Chatwork IFMN
    IFMNpMF 26 ▸ helm ▸ The package

    manager for Kubernetes ▸ KubernetesʹΞϓϦέʔγϣϯΛೖΕΔࡍʹɺඞཁͳ΋ͷΛύοέʔδ Խͯ͠ɺೖΕͯ͘ΕΔπʔϧ ▸ ChatworkͰ͸΄ͱΜͲΞϓϦέʔγϣϯΛHelmΛར༻ͯ͠σϓϩΠ ▸ Chartࣗମ͸؀ڥʹґଘ͠ͳ͍ʂ ▸ σϓϩΠͷ࠶ݱੑ(ࣗલKubernetes(kube-aws) -> EKSͰ΋ͦͷ··)

27. © Chatwork IFMN
    IFMNpMF 27 ▸ helmfile ▸ helm chartΛͲ͏΍ͬͯద༻͢Δͷ͔Λએݴతʹॻ͚Δ

    ▸ helmͰσϓϩΠ͢ΔࡍͷΞϓϦέʔγϣϯͷઃఆ஋΍ɺhelmίϚϯυ ͷΦϓγϣϯͳͲΛҰׅͰهࡌ͠ɺhelmͰσϓϩΠͰ͖Δπʔϧ ▸ https://github.com/helm/charts/tree/master/stable ▸ ͍ΖΜͳΞϓϦέʔγϣϯͷhelm chart ▸ https://developers.freee.co.jp/entry/2019/12/03/122657

28. © Chatwork ͳͥIFMNpMF͕ඞཁͳͷ͔ 28 ▸ ΞϓϦέʔγϣϯ͝ͱʹ࢖͍͍ͨΦϓγϣϯ(΍લॲཧ)͕มΘΔ ▸ ྫ͑͹)ΞϓϦέʔγϣϯಉ࢜Ͱґଘؔ܎ͷ͋Δ΋ͷ, ͳͲ ▸

    helmͷઃఆϑΝΠϧ͕؀ڥ͝ͱ(test, stg, prod)ͰมΘΔ ▸ ྫ͑͹)AWSͷΞΧ΢ϯτ͝ͱʹҧ͏, ϩάͷు͖ग़͠ઌ͕ҧ͏ ▸ go template ΋࢖͑ΔͷͰ, helmͷઃఆ஋Λಈతʹੜ੒͢Δ͜ͱ΋Մೳ ▸ ChatworkͰ͸ɺ͞ΒʹVariantͰhelmfileΛแΈࠐΈ

29. © Chatwork 29 IFMNpMF IFMN Λ7BSJBOUͰXSBQͨ݁͠Ռ ͜Ε͚ͩͰKubernetes಺෦΋OK

30. © Chatwork 30 ͍Ζ͍Ζ7BSJBOUͰXSBQͨ݁͠Ռ

31. © Chatwork ,VCFSOFUFTͷӡ༻Ͱେมͳ͜ͱ 31 ▸ Kubernetes version up ͸ 3ϲ݄ʹ1ճ

    ▸ ͢΂ͯʹ௥ै͠ͳ͍ͱͯ͠΋, ൒೥ʹ1ճ͙Β͍͸version up ▸ test, stg, production؀ڥͷ࡞Γม͑ɺσϓϩΠ͕සൃʂ ▸ EKSʹ͸ϩʔϦϯάΞοϓάϨʔυ͋Δ͚ΕͲ.... ▸ खಈ(υΩϡϝϯτରԠ)Ͱ͸ӡ༻ෛՙ͕ߴ͗͢Δ

32. ͜͜·Ͱ͸Kubernetesӡ༻ ͔͜͜Β͸CI/CD

33. © Chatwork ,VCFSOFUFTͷӡ༻ͰҰ൪େมͳ͜ͱ 33 ▸ Kubernetes version up ͸ 3ϲ݄ʹ1ճ

    ▸ ͢΂ͯʹ௥ै͠ͳ͍ͱͯ͠΋, ൒೥ʹ1ճ͙Β͍͸version up ▸ test, stg, production؀ڥͷ࡞Γม͑ɺσϓϩΠ͕සൃʂ ▸ EKSʹ͸ϩʔϦϯάΞοϓάϨʔυ͋Δ͚ΕͲ.... ▸ खಈ(υΩϡϝϯτରԠ)Ͱ͸ӡ༻ෛՙ͕ߴ͗͢Δ

34. © Chatwork ,VCFSOFUFT؀ڥͷ$*$% 34 ▸ KubernetesͰͷCI/CD ▸ αʔϏεͷΞϓϦέʔγϣϯ ▸ Concourse

    CIΛར༻ͯ͠CI/CD ▸ ͢΂ͯΛyamlܗࣜͷpipelineͰఆٛͰ͖Δπʔϧ

35. © Chatwork ,VCFSOFUFT؀ڥͷ$*$% 35 ▸ ࠔͬͨ ▸ CIͳΓϩʔΧϧ͔ΒσϓϩΠ͢Δ৔߹, ςϯϓϨʔτԽ͍ͯ͠Δͱͯ͠ ΋ɺΫϥελ͕Ͱ͖Δͨͼʹ޲͖ઌΛม͑Δඞཁ͕͋Δ

    ▸ ͦ΋ͦ΋αʔϏεΞϓϦͷσϓϩΠΛߦ͏Concourse CIࣗ਎ΛͲ͜Ͱӡ ༻͢Δͷ͔໰୊

36. © Chatwork (JU0QTʂ 36 https://www.weave.works/technologies/gitops/

37. © Chatwork (JU0QTʂ!$IBUXPSL 37 Argoͷ෦෼͸·ͩରԠͰ͖͍ͯͳ͍

38. © Chatwork $IBUXPSLͷ(JU0QTߏ૝ 38 ▸ Flux + Argo CDͷϋΠϒϦουߏ੒ ▸

    Flux ▸ ѻ͑Δͷ͸ݪଇmanifestͷΈͰ, 1flux - 1repo - 1branch ▸ Argo CDࣗ਎ͷΠϯετʔϧ΍namespace, aws-auth(eksಠࣗ)ͳͲ manifestద༻Ͱ͋·Γมߋ͕ͳ͍΋ͷ ▸ Argo CD ▸ GUI͕͋ͬͨΓ, helmʹରԠ͍ͯͨ͠Γɺͱػೳ͕ଟ͘࢖͍΍͍͢

39. © Chatwork $IBUXPSLͷ(JU0QTߏ૝ 39 ▸ αʔϏεͷΞϓϦέʔγϣϯ͚ͩͰ͸ͳ͘ɺ؅ཧܥͷΞϓϦͷΞϓϦέʔ γϣϯ΋GitOpsͰద༻͢Δํࣜʹมߋ༧ఆ ▸ Kubernetes͕Ͱ͖ΔͱɺඞཁͳΞϓϦέʔγϣϯ͸͢΂ͯউखʹೖ͍ͬͯ Δঢ়ଶ

    ▸ αʔϏεͷΞϓϦέʔγϣϯʹؔͯ͠͸΋ͪΖΜ੾Γସ͑ͳͲ͕ඞཁ

40. © Chatwork (JU0QTΛΊ͙Δ؀ڥ 40

41. © Chatwork ,VCFSOFUFTͷӡ༻Ͱେมͳ͜ͱ 41 ▸ Kubernetes version up ͸ 3ϲ݄ʹ1ճ

    ▸ ͢΂ͯʹ௥ै͠ͳ͍ͱͯ͠΋, ൒೥ʹ1ճ͙Β͍͸version up ▸ test, stg, production؀ڥͷ࡞Γม͑ɺσϓϩΠ͕සൃʂ ▸ EKSʹ͸ϩʔϦϯάΞοϓάϨʔυ͋Δ͚ΕͲ.... ▸ खಈ(υΩϡϝϯτରԠ)Ͱ͸ӡ༻ෛՙ͕ߴ͗͢Δ

42. © Chatwork ·ͱΊ 42 ▸ Chatworkʹ͓͚ΔKubernetesͷӡ༻(ߏங) ▸ eksctl, variant, helm,

    helmfile ▸ GitOpsߏ૝ ▸ EKSɺ൒ֹʂ