Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Adventure around Kubernetes at Chatwork

Ryo Sakamoto
January 23, 2020

Adventure around Kubernetes at Chatwork

Ryo Sakamoto

January 23, 2020
Tweet

More Decks by Ryo Sakamoto

Other Decks in Technology

Transcript

  1. ,VCFSOFUFTΛΊ͙Δ๯ݥ
    +"846(ίϯςφࢧ෦
    $IBUXPSL43&νʔϜࡔຊ

    View Slide

  2. © Chatwork
    ࣗݾ঺հ
    2
    ▸ ࡔຊ ྒ (͔͞΋ͱɹΓΐ͏)
    ▸ Chatwork SREΤϯδχΞ
    ▸ ϥϯχϯάΤϯδχΞ
    ▸ ϑϧϚϥιϯ͕Α͏΍͘3࣌ؒ17෼͙Β͍(2019೥3݄)
    ▸ 2019೥ͷ૸ߦڑ཭͸2700km͙Β͍
    ▸ NY- LA͸4000kmͳͷͰ, ࢒೦ͳ͕ΒΞϝϦΧԣஅ͸Ͱ͖ͣ

    View Slide

  3. ▸ ϏδωενϟοταʔϏε
    ▸ άϧʔϓνϟοτɺλεΫ؅ཧɺ

    ϑΝΠϧڞ༗ɺϏσΦɾԻ੠௨࿩
    ▸ ಋೖاۀ 246,000 ࣾ

    (2019೥12݄ݱࡏ)

    View Slide

  4. © Chatwork
    ΞδΣϯμ
    4
    ▸ ChatworkͷKubernetesͷྺ࢙ΛৼΓฦΓͳ͕Βݱࡏͷӡ༻ʹ͍ͭͯ
    ▸ ྺ࢙
    ▸ ӡ༻πʔϧ
    ▸ ChatworkͷKubernetes CI/CD

    View Slide

  5. © Chatwork
    ஫ҙ
    5
    Fargateͷ࿩͸͋Γ·ͤΜ

    View Slide

  6. © Chatwork
    લఏ$IBUXPSLͷ,VCFSOFUFTར༻ܗଶ
    6
    ▸ Ϛϧνςφϯτ
    ▸ Ϋϥελࣗମ͸SRE͕࡞Δ
    ▸ ؅ཧܥΞϓϦέʔγϣϯ(Datadog, Fluentd ͳͲ) -> SRE
    ▸ αʔϏεΞϓϦέʔγϣϯ -> DevνʔϜ

    View Slide

  7. © Chatwork
    $IBUXPSLͷ,VCFSOFUFTͷྺ࢙
    7
    ▸ ಋೖ͸2016೥
    ▸ ࠷ॳͷProductionͷόʔδϣϯ͸1.5
    ▸ ϝοηʔδϯάγεςϜͷ࡮৽ϓϩδΣΫτ
    ▸ AWS Summit Tokyo 2017 Ͱൃද
    ▸ ͦͷޙԿ౓͔ΫϥελҠߦ(όʔδϣϯΞοϓ)Λͭͭ͠ɺݱࡏʹࢸΔ
    ▸ ࣗલKubernetes, EKSͷࠞ߹ঢ়ଶ

    View Slide

  8. © Chatwork
    &,4ͷಋೖ
    8
    ▸ EKSར༻લ͸kube-awsͱ͍͏ͷΛར༻ͯ͠, ࣗલͰϗεςΟϯά
    ▸ https://github.com/kubernetes-incubator/kube-aws
    ▸ EKSͷొ৔ʹΑΓkube-awsͷϢʔβݮ…
    ▸ ࡉ͔͍ػೳʹόά͕ଟ͘, దٓPRରԠͭͭ͠΋ࣗલύονӡ༻
    ▸ EKSࣗମ͸ίϯτϩʔϥʔ෦෼͚ͩͰ, ࢖͍ʹ͍͘ͳ͊…ͱϓϨϏϡʔͰ
    ࢥ͍͕ͬͯͨeksctlʹΑΓͦ͏͍͏ͱ͜Ζ͕·Δͬͱָʹ

    View Slide

  9. © Chatwork
    FLTDUMͱLVCFBXTͷൺֱ
    9
    ▸ eksctl
    ▸ ։ൃ͕ૣ͍
    ▸ جຊతʹ͸Ϋϥελͷߏ੒͸updateͰ͖ͳ͍
    ▸ ϊʔυάϧʔϓͷઃఆมߋΛ͍ͨ͠৔߹͸ɺ৽ن࡞੒ɾچϊʔυά
    ϧʔϓ࡟আ
    ▸ ϊʔυάϧʔϓͷߏ੒Λࢼ͍ͯ͠Δͱ͖͸ͪΐͬͱखؒ

    View Slide

  10. ΦϑΟγϟϧͳͷͰɺEKS৽͍͠ػೳ͸େମAWSͷൃදͱಉ࣌ʹରԠ͞Ε͍ͯΔ

    View Slide

  11. © Chatwork
    FLTDUMͱLVCFBXTͷൺֱ
    11
    ▸ kube-aws
    ▸ Ϋϥελͷߏ੒͸updateՄೳ
    ▸ ϊʔυάϧʔϓͷઃఆมߋΛ͍ͨ͠৔߹͸ɺچϊʔυάϧʔϓ͔Β
    ϩʔϦϯάΞοϓσʔτ(CloudFormationͷupdateͳͷͰ...)
    ▸ EKSΛར༻͢Δ΋ͷͰ͸ͳ͘, controller, etcdؚΊͯ࡞੒
    ▸ ੜKubernetesͳͷͰɺΧελϚΠζੑ͸͔ͳΓߴ͍
    ▸ KubernetesͷઃఆϑΝΠϧ͕௕େͳyamlʹͳΔ(Productionͷ686ߦ)

    View Slide

  12. © Chatwork
    FLTDUMͱLVCFBXTͷൺֱ
    12
    ▸ ௕ظతʹݟΔͱ, eksctlͷ΄͏͕ϝϦοτ͸େ͖͍
    ▸ Ϣʔβೝূ͕iamͰͰ͖ͨΓ, podʹiam roleΛ෇͚ΒΕͨΓ, eksͦͷ΋ͷ
    ͷϝϦοτ΋େ͖͍
    ▸ iam role for pod͸ͱͯ΋خ͍͠
    ▸ kube2iam, kiam, kube-aws-iam-controller͢΂ͯӡ༻͖ͯͨ͠ܦݧ
    ▸ ͜Ε͸͜ΕͰखܰ͞΋͋Δ͚ΕͲ

    View Slide

  13. ΫϥελΛ࡞Δ͚ͩͳΒ͜ΕͰOK

    ͔͠͠ɾɾ

    View Slide

  14. © Chatwork
    ,VCFSOFUFTͷӡ༻Ͱେมͳ͜ͱ
    14
    ▸ Kubernetes version up ͸ 3ϲ݄ʹ1ճ
    ▸ ͢΂ͯʹ௥ै͠ͳ͍ͱͯ͠΋, ൒೥ʹ1ճ͙Β͍͸version up
    ▸ test, stg, production؀ڥͷ࡞Γม͑ɺσϓϩΠ͕සൃʂ
    ▸ EKSʹ͸ϩʔϦϯάΞοϓάϨʔυ͋Δ͚ΕͲ....
    ▸ खಈ(υΩϡϝϯτରԠ)Ͱ͸ӡ༻ෛՙ͕ߴ͗͢Δ

    View Slide

  15. ͜ΕΛ৐Γӽ͑Δӡ༻ɾ࢓૊Έ͕ඞཁ

    View Slide

  16. © Chatwork
    ,VCFSOFUFTͷӡ༻Λࢧ͑Δπʔϧͨͪ
    16
    ▸ eksctl (kube-aws)
    ▸ Ϋϥελࣗମ͸͜ΕͰ࡞੒
    ▸ Variant
    ▸ https://github.com/mumoshu/variant
    ▸ helm, helmfile
    ▸ https://github.com/roboll/helmfile

    View Slide

  17. © Chatwork
    7BSJBOU
    17
    ▸ task runner
    ▸ eksctlͳͲͷίϚϯυΛϥοϓ͢Δ
    ▸ wrapper shellͰ΋͍͍͕ɺͪ͜Β͸yamlͰॻ͚ͯɺύϥϝʔλͳͲͷࢦ
    ఆ΋΍Γ΍͍͢
    ▸ dockerϕʔεͷtask runner͕ఆٛͰ͖ɺ؀ڥґଘΛݮΒͤΔ
    ▸ ؀ڥͱίϚϯυΛηοτʹͰ͖Δ

    View Slide

  18. © Chatwork
    FLTDUM͚ͩͰ͸ͳͥͩΊͳͷ͔
    18
    ▸ eksctl͚ͩͰ΋े෼ͳ৔߹΋ଟ͍
    ▸ ͱΓ͋͑ͣࢼݧ؀ڥΛىಈͤ͞Δʹ͸ίϚϯυ͚ͩͰे෼
    ▸ ͨͩ͠ࡉ͔͍ઃఆΛ͠Α͏ͱ͢Δͱ, ϑΝΠϧͰͷઃఆͷ΄͏ָ͕
    ▸ sampleҎ֎ͷυΩϡϝϯτ͸ͳ͍ͷͰ, ஫ҙ
    ▸ https://github.com/weaveworks/eksctl/blob/master/pkg/apis/eksctl.io/
    v1alpha5/types.go ͔Β୳͔͢͠ͳ͍ͱ͖΋͋Δ
    ▸ ઃఆΛ౉ͤΔVariantͰϥοϓͯ͠, eksctl༻ઃఆϑΝΠϧͷ஋Λஔ׵

    View Slide

  19. # Ϋϥελ໊
    # αϒωοτ
    # cluster-autoscalerͷiam roleͭ͘Δ
    # eksຊମ(ϊʔυάϧʔϓআ͘)ͷeksctlͷઃఆϑΝΠϧ

    View Slide

  20. © Chatwork
    FLTDUM͚ͩͰ͸ͳͥͩΊͳͷ͔
    20
    ▸ Ϋϥελ࡞Δ͚ͩͳΒ, ୯ମར༻Ͱ΋͍͚ͳ͘͸ͳ͍
    ▸ FluxͩΕ͕ೖΕΔͷ໰୊ɺ΁ͷରԠ
    ▸ Flux ... GitOps tool ޙड़͠·͢
    ▸ GitOpsͰઃఆͷ൓ө΍ΞϓϦέʔγϣϯͷσϓϩΠΛ͢Δ
    ▸ Ͱ͸, GitOpsΛ࣮ݱ͢ΔͨΊͷπʔϧ͸Ұମ୭͕ೖΕΔͷ͔...!
    ▸ GitOps ͸Kubernetes಺෦ʹ࣮ݱ͢ΔͨΊͷΞϓϦέʔγϣϯ͕͍Δ

    View Slide

  21. © Chatwork 21
    ▸ eksctlίϚϯυͰFluxΛ͍ΕΔ͜ͱ͸Մೳ
    ▸ υΩϡϝϯτϕʔεͰ΋ରԠͰ͖Δ͕ɺԼهͷϑϩʔΛ·ͱΊ͍ͨ
    ▸ eksctlͷઃఆϑΝΠϧ࡞੒
    ▸ eksctl create cluster
    ▸ eksctl enable repo ... (Fluxͷinstall)
    ▸ ͜ΕΒΛvariantͰλεΫ੍ޚ(Infrastructure as codeͷҰ؀)
    FLTDUM͚ͩͰ͸ͳͥͩΊͳͷ͔

    View Slide

  22. © Chatwork 22
    FLTDUMΛ7BSJBOUͰXSBQͨ݁͠Ռ
    ͜Ε͚ͩͰEKS؀ڥOK
    https://en.wikipedia.org/wiki/Achlis
    EKS࡞੒ίϚϯυΛachlisͱ໋໊

    View Slide

  23. © Chatwork
    ,VCFSOFUFTͷӡ༻Ͱେมͳ͜ͱ
    23
    ▸ Kubernetes version up ͸ 3ϲ݄ʹ1ճ
    ▸ ͢΂ͯʹ௥ै͠ͳ͍ͱͯ͠΋, ൒೥ʹ1ճ͙Β͍͸version up
    ▸ test, stg, production؀ڥͷ࡞Γม͑ɺσϓϩΠ͕සൃʂ
    ▸ EKSʹ͸ϩʔϦϯάΞοϓάϨʔυ͋Δ͚ΕͲ....
    ▸ खಈ(υΩϡϝϯτରԠ)Ͱ͸ӡ༻ෛՙ͕ߴ͗͢Δ

    View Slide

  24. ͜͜·Ͱ͸Kubernetesࣗମͷߏங

    ͔͜͜Β͸Kubernetes಺෦ͷߏங

    View Slide

  25. © Chatwork 25
    ▸ https://toris.io/2019/12/what-i-think-about-when-i-think-about-kubernetes-
    and-ecs/
    ▸ (ͳΜ͔ଜ্य़थͷλΠτϧͬΆ͍...)
    ▸ EKS͸ϚωʔδυͰ͕͢ɺϚωʔδϝϯτίϯιʔϧͰ࡞੒͚ͨͩ͠Ͱ͸
    ΄΅Կ΋Ͱ͖·ͤΜ
    ▸ ϊʔυάϧʔϓ(࣮ࡍʹίϯςφ͕ಈ͘؀ڥ)Λ࡞ͬͯ΋ɺ·ͩϩά΋ૹΕ
    ͣɺϝτϦΫε΋ݟΕͣɺͰ͢
    ▸ ͳͷͰɺEKSͰ΋؅ཧܥͷΞϓϦέʔγϣϯ(ex. Datadog)ͷӡ༻͕ඞਢ
    &$4ͱͷҧ͍

    View Slide

  26. © Chatwork
    IFMN IFMNpMF
    26
    ▸ helm
    ▸ The package manager for Kubernetes
    ▸ KubernetesʹΞϓϦέʔγϣϯΛೖΕΔࡍʹɺඞཁͳ΋ͷΛύοέʔδ
    Խͯ͠ɺೖΕͯ͘ΕΔπʔϧ
    ▸ ChatworkͰ͸΄ͱΜͲΞϓϦέʔγϣϯΛHelmΛར༻ͯ͠σϓϩΠ
    ▸ Chartࣗମ͸؀ڥʹґଘ͠ͳ͍ʂ
    ▸ σϓϩΠͷ࠶ݱੑ(ࣗલKubernetes(kube-aws) -> EKSͰ΋ͦͷ··)

    View Slide

  27. © Chatwork
    IFMN IFMNpMF
    27
    ▸ helmfile
    ▸ helm chartΛͲ͏΍ͬͯద༻͢Δͷ͔Λએݴతʹॻ͚Δ
    ▸ helmͰσϓϩΠ͢ΔࡍͷΞϓϦέʔγϣϯͷઃఆ஋΍ɺhelmίϚϯυ
    ͷΦϓγϣϯͳͲΛҰׅͰهࡌ͠ɺhelmͰσϓϩΠͰ͖Δπʔϧ
    ▸ https://github.com/helm/charts/tree/master/stable
    ▸ ͍ΖΜͳΞϓϦέʔγϣϯͷhelm chart
    ▸ https://developers.freee.co.jp/entry/2019/12/03/122657

    View Slide

  28. © Chatwork
    ͳͥIFMNpMF͕ඞཁͳͷ͔
    28
    ▸ ΞϓϦέʔγϣϯ͝ͱʹ࢖͍͍ͨΦϓγϣϯ(΍લॲཧ)͕มΘΔ
    ▸ ྫ͑͹)ΞϓϦέʔγϣϯಉ࢜Ͱґଘؔ܎ͷ͋Δ΋ͷ, ͳͲ
    ▸ helmͷઃఆϑΝΠϧ͕؀ڥ͝ͱ(test, stg, prod)ͰมΘΔ
    ▸ ྫ͑͹)AWSͷΞΧ΢ϯτ͝ͱʹҧ͏, ϩάͷు͖ग़͠ઌ͕ҧ͏
    ▸ go template ΋࢖͑ΔͷͰ, helmͷઃఆ஋Λಈతʹੜ੒͢Δ͜ͱ΋Մೳ
    ▸ ChatworkͰ͸ɺ͞ΒʹVariantͰhelmfileΛแΈࠐΈ

    View Slide

  29. © Chatwork 29
    IFMNpMF IFMN
    Λ7BSJBOUͰXSBQͨ݁͠Ռ
    ͜Ε͚ͩͰKubernetes಺෦΋OK

    View Slide

  30. © Chatwork 30
    ͍Ζ͍Ζ7BSJBOUͰXSBQͨ݁͠Ռ

    View Slide

  31. © Chatwork
    ,VCFSOFUFTͷӡ༻Ͱେมͳ͜ͱ
    31
    ▸ Kubernetes version up ͸ 3ϲ݄ʹ1ճ
    ▸ ͢΂ͯʹ௥ै͠ͳ͍ͱͯ͠΋, ൒೥ʹ1ճ͙Β͍͸version up
    ▸ test, stg, production؀ڥͷ࡞Γม͑ɺσϓϩΠ͕සൃʂ
    ▸ EKSʹ͸ϩʔϦϯάΞοϓάϨʔυ͋Δ͚ΕͲ....
    ▸ खಈ(υΩϡϝϯτରԠ)Ͱ͸ӡ༻ෛՙ͕ߴ͗͢Δ

    View Slide

  32. ͜͜·Ͱ͸Kubernetesӡ༻

    ͔͜͜Β͸CI/CD

    View Slide

  33. © Chatwork
    ,VCFSOFUFTͷӡ༻ͰҰ൪େมͳ͜ͱ
    33
    ▸ Kubernetes version up ͸ 3ϲ݄ʹ1ճ
    ▸ ͢΂ͯʹ௥ै͠ͳ͍ͱͯ͠΋, ൒೥ʹ1ճ͙Β͍͸version up
    ▸ test, stg, production؀ڥͷ࡞Γม͑ɺσϓϩΠ͕සൃʂ
    ▸ EKSʹ͸ϩʔϦϯάΞοϓάϨʔυ͋Δ͚ΕͲ....
    ▸ खಈ(υΩϡϝϯτରԠ)Ͱ͸ӡ༻ෛՙ͕ߴ͗͢Δ

    View Slide

  34. © Chatwork
    ,VCFSOFUFT؀ڥͷ$*$%
    34
    ▸ KubernetesͰͷCI/CD
    ▸ αʔϏεͷΞϓϦέʔγϣϯ
    ▸ Concourse CIΛར༻ͯ͠CI/CD
    ▸ ͢΂ͯΛyamlܗࣜͷpipelineͰఆٛͰ͖Δπʔϧ

    View Slide

  35. © Chatwork
    ,VCFSOFUFT؀ڥͷ$*$%
    35
    ▸ ࠔͬͨ
    ▸ CIͳΓϩʔΧϧ͔ΒσϓϩΠ͢Δ৔߹, ςϯϓϨʔτԽ͍ͯ͠Δͱͯ͠
    ΋ɺΫϥελ͕Ͱ͖Δͨͼʹ޲͖ઌΛม͑Δඞཁ͕͋Δ
    ▸ ͦ΋ͦ΋αʔϏεΞϓϦͷσϓϩΠΛߦ͏Concourse CIࣗ਎ΛͲ͜Ͱӡ
    ༻͢Δͷ͔໰୊

    View Slide

  36. © Chatwork
    (JU0QTʂ
    36
    https://www.weave.works/technologies/gitops/

    View Slide

  37. © Chatwork
    (JU0QTʂ!$IBUXPSL
    37
    Argoͷ෦෼͸·ͩରԠͰ͖͍ͯͳ͍

    View Slide

  38. © Chatwork
    $IBUXPSLͷ(JU0QTߏ૝
    38
    ▸ Flux + Argo CDͷϋΠϒϦουߏ੒
    ▸ Flux
    ▸ ѻ͑Δͷ͸ݪଇmanifestͷΈͰ, 1flux - 1repo - 1branch
    ▸ Argo CDࣗ਎ͷΠϯετʔϧ΍namespace, aws-auth(eksಠࣗ)ͳͲ
    manifestద༻Ͱ͋·Γมߋ͕ͳ͍΋ͷ
    ▸ Argo CD
    ▸ GUI͕͋ͬͨΓ, helmʹରԠ͍ͯͨ͠Γɺͱػೳ͕ଟ͘࢖͍΍͍͢

    View Slide

  39. © Chatwork
    $IBUXPSLͷ(JU0QTߏ૝
    39
    ▸ αʔϏεͷΞϓϦέʔγϣϯ͚ͩͰ͸ͳ͘ɺ؅ཧܥͷΞϓϦͷΞϓϦέʔ
    γϣϯ΋GitOpsͰద༻͢Δํࣜʹมߋ༧ఆ
    ▸ Kubernetes͕Ͱ͖ΔͱɺඞཁͳΞϓϦέʔγϣϯ͸͢΂ͯউखʹೖ͍ͬͯ
    Δঢ়ଶ
    ▸ αʔϏεͷΞϓϦέʔγϣϯʹؔͯ͠͸΋ͪΖΜ੾Γସ͑ͳͲ͕ඞཁ

    View Slide

  40. © Chatwork
    (JU0QTΛΊ͙Δ؀ڥ
    40

    View Slide

  41. © Chatwork
    ,VCFSOFUFTͷӡ༻Ͱେมͳ͜ͱ
    41
    ▸ Kubernetes version up ͸ 3ϲ݄ʹ1ճ
    ▸ ͢΂ͯʹ௥ै͠ͳ͍ͱͯ͠΋, ൒೥ʹ1ճ͙Β͍͸version up
    ▸ test, stg, production؀ڥͷ࡞Γม͑ɺσϓϩΠ͕සൃʂ
    ▸ EKSʹ͸ϩʔϦϯάΞοϓάϨʔυ͋Δ͚ΕͲ....
    ▸ खಈ(υΩϡϝϯτରԠ)Ͱ͸ӡ༻ෛՙ͕ߴ͗͢Δ

    View Slide

  42. © Chatwork
    ·ͱΊ
    42
    ▸ Chatworkʹ͓͚ΔKubernetesͷӡ༻(ߏங)
    ▸ eksctl, variant, helm, helmfile
    ▸ GitOpsߏ૝
    ▸ EKSɺ൒ֹʂ

    View Slide