ニコニコ生放送におけるWebフロントエンドBFFサーバーのKubernetes移行事例の紹介

2022/06/13に「ニコニコ生放送 WebフロントエンドのKubernetes移行ハンドブック 2022」を公開しました。本発表ではハンドブックの内容とともに、WebフロントエンドにおけるKubernetesへの移行と運用がどのようなものか紹介します。

Playground

Transcript

1. χίχίੜ์ૹʹ͓͚Δ
   WebϑϩϯτΤϯυBFFαʔόʔͷ
   KubernetesҠߦࣄྫͷ঺հ
   Kubernetes Meetup Tokyo #52
   2022/08/17 Wed 19:00 ʙ
   Online
   

   View Slide

2. Name
   ● Himeno Kosei
   Recent Career
   ● 2022/07 - גࣜձࣾϋΠϠʔϧʔ
   ● 2018/06 - 2022/06 גࣜձࣾυϫϯΰ
   Focus
   1. Web Frontend
   2. Go / Kubernetes / LB / Proxy
   GitHub
   ● https://github.com/Himenon
   ࣗݾ঺հ
   

   View Slide

3. ໨࣍
   1. Introductionʢʙ 3minʣ
   2. ҠߦϋϯυϒοΫͷ঺հʢʙ 3min ʣ
   3. എܠઆ໌ͱ՝୊ʢʙ3minʣ
   4. TypeScriptͰManifestΛॻ͘ʢʙ3minʣ
   5. ArgoCD + SlackBotʢʙ3minʣ
   6. ݕূ࡞ۀʢʙ 5minʣ
   7. ΫϥελʔͷҠߦʢʙ 2minʣ
   8. ࣭ٙʢʙ remain timeʣ
   a. Handbookதͷ಺༰Ͱ΋Մ
   

   View Slide

4. ಡऀ૚ ࣋ͪؼΓͰ͖Δͱࢥ͏΋ͷ
   KubernetesΛීஈ৮͍ͬͯΔਓ ● ManifestΛTypeScriptͰӡ༻͢Δํ๏͕͋ΔΒ͍͠
   ○ ීஈ׳Ε͍ͯΔݴޠͰॻ͘͜ͱ΋Ͱ͖Δ
   ● ൚༻ݴޠͰManifestΛੜ੒͢ΔͱYAMLΛॻ͘ͱຊ࣭తͳ࡞ۀʹूதͰ͖Δ
   TypeScriptΛීஈར༻͍ͯ͠Δਓ ● KubernetesͷManifest͸ܕ҆શʹॻ͘͜ͱ΋Ͱ͖Δ
   ● NodejsपΓͷΤίγεςϜʹ৐ΕΔ
   ͍Ζ͍Ζৄ͍͠ํʑ ● ൚༻ݴޠΛ࢖ͬͨݕূαΠΫϧͷߴ଎Խͷํ๏
   ຊൃදͰ࣋ͪؼΕΔ΋ͷ
   

   View Slide

5. 2021೥6݄͔Β2022೥3݄ͷ9ϲ݄ؒͰχίχίੜ์ૹͷϑϩϯτΤϯυʹؔ܎͢ΔϚΠΫϩαʔϏεΛDocker Swarm͔Β
   Kubernetes΁Ҡߦͨ͠࿩Λܝࡌ͍ͯ͠·͢ɻ
   ͜ͷαʔϏεͰ͢ → https://live.nicovideo.jp/
   ϋϯυϒοΫͷ঺հ
   χίχίੜ์ૹ WebϑϩϯτΤϯυͷKubernetesҠߦϋϯυϒοΫ 2022
   GitHub https://github.com/dwango/nicolive-kubernetes-migration-handbook-2022
   Web Site https://dwango.github.io/nicolive-kubernetes-migration-handbook-2022/
   PDF https://dwango.github.io/nicolive-kubernetes-migration-handbook-2022/docs/about/
   

   View Slide

6. ҎԼͷ՝୊Λ࠷খݶʹ͍ͨ͠
   ● KubernetesΛӡ༻͢Δʹ͸େྔͷ஌͕ࣝඞཁ
   ● ࣮ӡ༻͢Δʹ͸அยతͳ஌ࣝͰ͸ͳ͘Ұ؏ͨ͠৘ใ͕ٻΊΒΕΔ
   ● ઐ໳ੑ͕ߴ͍ͱଐਓԽͷϦεΫ͕ߴ͍
   ● ֶशίετ͕ߴ͍ʢ= આ໌ʹ΋͕͔͔࣌ؒΔʣ
   ࣄྫͱͯ͠࢒͢ҙٛ
   ● ઌߦࣄྫ͕͋Δͱଞͷ։ൃऀ͕ํ๏Λਅࣅ͠΍͘͢ͳΔ
   ● ΋ͬͱ͍͍ํ๏Λࢥ͍ͭ͘ਓ͕ग़ͯ͘Δ
   ϋϯυϒοΫΛެ։ͨ͠໨త
   

   View Slide

7. ϋϯυϒοΫͰ঺հ͍ͯ͠ΔτϐοΫ
   ঺հτϐοΫ πʔϧ or ߏ੒ ؆қ·ͱΊ
   ✅ Manifest؅ཧ TypeScript ܕγεςϜͱ൚༻ݴޠͷΤίγεςϜʹॾʑΛ৐ͤΔ
   ✅ Continuous Delivery Argo CD + Argo Rollouts + Slack Bot ෳ਺؀ڥରԠ + Canary Releaseɺӡ༻ࣗಈԽ
   Service Mesh Istio Observability޲্
   Rate Limit Local Ratelimit / Global Ratelimit ෛՙରࡦ
   εέʔϦϯά Horizontal Pod Autoscaler HPAͷݟੵ΋Γํ๏
   ✅ ෛՙࢼݧ vegetaʢHTTP Load Testing Toolʣ ҠߦͷͨΊͷ؆қੑೳධՁ
   ✅ ϞχλϦϯά dd-agent → DataDog rpsͷूܭɺPodͷߋ৽ՄࢹԽ
   Logging nginx → fluent bit → Volume → fluentd ϩάऩूͷҠߦͱকདྷ΁ͷ෍ੴ
   ✅ Kubernetes΁ͷҠߦ ετϥϯάϥʔϑΟάύλʔϯ Docker Swarm͔ΒKubernetes΁ͷҠߦ
   ✅ ͕͍͍ͭͯΔ߲໨ΛຊൃදͰ঺հ͢Δ
   

   View Slide

8. എܠઆ໌ͱ՝୊
   

   View Slide

9. എܠ ߟྀࣄ߲
   Kubernetes͸ΦϯϓϨ؀ڥʹߏங ● ໰୊͕ੜͨ͡৔߹͸VMͷઃఆ΋ߟྀʹೖΕΔඞཁ͕͋Δ
   ● Kubernetesߏங࣌ͷઃఆ΋৔߹ʹΑͬͯ͸ݟΔ
   Kubernetesͷӡ༻ऀ͸σϓϩΠ͠
   ͨνʔϜࣗ਎
   ● WebϑϩϯτΤϯυͷνʔϜ͕ࣗ෼ͨͪͰσϓϩΠ͍ͯ͠Δίϯςφͷ໘౗ΛݟΔ
   ● Backend For Frontendͷߏ੒ͳͨΊɺ༷ʑͳϚΠΫϩαʔϏεʹ઀ଓ
   ӡ༻ऀʹे෼ͳKubernetesͷ஌ࣝ
   ͕ٻΊΒΕΔ
   ● ଞͷνʔϜͱ΋ڠྗ͸͍ͯ͠Δ͕ඞͣ͠΋ඞཁͳͱ͖ʹ͕࣌ؒऔΕΔΘ͚Ͱ͸ͳ͍
   ● ΞϓϦέʔγϣϯͷ໰୊ͳͷ͔ɺΠϯϑϥͷઃఆͷ໰୊ͳͷ͔ݟۃΊΔඞཁ͕Ͱͯ͘Δ
   ΠϯϑϥͷGitOpsʹ׳Ε͍ͯͳ͍ ● ैདྷ͸Jenkinsܦ༝Ͱߋ৽ɻύϥϝʔλʔΛೖΕͨΒϦϦʔε͞ΕΔੈք
   ● Ҡߦ࣌ͷӡ༻Λ؆୯ʹͯ͠GitOpsʹνʔϜ͕׳ΕΔ·Ͱͷ͕࣌ؒඞཁ
   ։ൃ؀ڥ؀ڥ͕ෳ਺ ● ෳ਺ϨʔϯͰҊ͕݅ಈ͍͍ͯΔ͜ͱ΋͋ΔͨΊૣ͍΋ͷ͕ͪ
   ● ։ൃ؀ڥ΋εέʔϧ͍͖͍ͯͨ͠
   എܠઆ໌ (1/2)
   

   View Slide

10. എܠઆ໌ (2/2)
    Backend For Frontedͱ͸
    nodejs
    MicroServie A
    MicroService B
    MicroService C
    Browser
    ᶆ HTML / JSON
    ᶃ Access ᶄFetch
    ᶅ
    ɾServe Side Rendering
    ɾData Formating
    Load Balancer
    ● ΫϥΠΞϯτʢWebϑϩϯτΤϯυʣͷͨΊͷαʔόʔ
    ● Server Side RenderingʢHTMLΛฦ٫ʣ
    ● ੔ܗ͞ΕͨσʔλΛฦ͢APIͷఏڙΛߦ͏ʢ෗ഊ๷ࢭ૚ͱͯ͠ͷ໾ׂʣ
    ● جຊతʹεςʔτϨε
    Reverse Proxy
    

    View Slide

11. ༧૝͞ΕΔ՝୊ ͳͥ
    ͲͷύϥϝʔλʔΛ͍ͭߋ৽ͯ͠Α͍ͷ͔Θ͔Βͳ͍ ● ӡ༻υΩϡϝϯτ͕ͳ͍
    ● ύϥϝʔλʔ͕ଟ͍ɾґଘؔ܎͕ෳࡶ
    ࡉ͔͍ϨϏϡʔ͕ൃੜ͢Δ ● YAMLͷه๏͕༷ʑ
    ● ໋໊نଇ / ॱং … etc
    શମ࠷దԽ͕೉͍͠ ● ྔ͕ଟ͍ɾݕূ߲໨͕ଟ͍
    ● ࠶ݱੑͷ͋Δߏ੒Λҡ࣋ɺల։͢Δํ๏Λ͍࣋ͬͯͳ͍
    υΩϡϝϯτΛॻ͍ͯ΋อक͠ͳ͘ͳΔ ● ࣮ଶͱဃ཭͍ͯ͘͠
    ● υΩϡϝϯτΛϨϏϡʔର৅ʹීஈೖΕͯͳ͍
    ϞχλϦϯάʹඞཁͳύϥϝʔλʔΛ෇༩͠๨ΕΔ ● ීஈ͔Βҙࣝతʹ৮Δ͜ͱ͕গͳ͍
    എܠΛ౿·্͑ͨͰ...
    KubernetesΛΞϓϦέʔγϣϯ։ൃऀ͕ແରࡦͰӡ༻͢Δͱൃੜ͢Δʢͨ͠ʣͰ͋Ζ͏՝୊
    

    View Slide

12. ՝୊͕ൃੜ͢Δͱ͜Ζ͸େ͖͘2Օॴ
    ManifestΛॻ͘ͱ͖ͷ໰୊ σϓϩΠ͢Δͱ͖ͷ໰୊
    Manifest
    (YAMLϑΝΠϧ)
    Developer Kubernetes
    ฤू Apply
    

    View Slide

13. YAMLΛੜ੒
    ࠓճͷ՝୊ղܾͷΞϓϩʔν
    Manifest
    (YAML)
    σϓϩΠͷ՝୊Λղܾ͢Δ
    Manifest؅ཧ
    ਺ͷ՝୊Λղܾ͢Δ
    ӡ༻ͷ՝୊Λղܾ͢Δ
    Developer͔ΒݟΔͱ׳Ε͍ͯΔπʔϧͰKubernetesͷઃఆม
    ߋΛͰ͖Δ͜ͱʹͳΔ
    Developer
    Deploy
    webhook
    ࡉ͔͍ߏ੒มߋ
    ೔ৗͷߋ৽࡞ۀ
    

    View Slide

14. TypeScriptͰ
    ManifestΛॻ͘
    

    View Slide

15. DEMO: ManifestΛॻ͍͍ͯΔྫ
    Reference:
    Playground: https://codesandbox.io/s/typescript-to-kubernetes-manifest-playground-nco13z?file=/src/manifests/deployment.ts
    SampleCode: https://gist.github.com/Himenon/ec9ab8681a779daf746903ab3d8cddab
    

    View Slide

16. TypeScriptΛॻ͘͜ͱͰޮ཰ԽͰ͖Δ࡞ۀ
    ● KubernetesͷTypeScriptܕఆٛ͸OpenAPI͔Βੜ੒͢Δ
    ○ Schema: https://github.com/kubernetes/kubernetes/tree/master/api/openapi-spec
    ○ ੜ੒ͨ͠TypeScript: https://github.com/Himenon/kubernetes-typescript-openapi
    ■ ࣗಈੜ੒
    ■ npmʹpublish
    ● ܕͷิ׬Λޮ͔ͤͳ͕Β֤छύϥϝʔλʔΛೖྗ͢Δ
    ● ManifestͷϓϩύςΟͷҙຯ͕Θ͔Βͳ͍ͱ͖͸TSDocΛಡΉ
    ○ OpenAPI Schemaͷdescription͕EditorͰݟΕΔͨΊWebݕࡧ͠ͳͯ͘΋͍͍
    ● ࣗ෼ͨͪͷϚΠΫϩαʔϏεͷ৘ใΛTSDocΛॻ͘͜ͱ΋Ͱ͖Δ
    ● ม਺Λར༻ͯ͠ύϥϝʔλʔԽ
    ● ࢦఆ͞Εͨ͘ͳ͍஋͸ExceptionΛ౤͛Δ
    ○ CIͰݕ஌ͤ͞ΔͨΊɺcatch͠ͳͯ͘Α͍
    

    View Slide

17. TypeScriptͰManifestΛग़ྗ͢ΔΞʔΩςΫνϟͷղઆ
    Reference:
    Handbook: https://dwango.github.io/nicolive-kubernetes-migration-handbook-2022/docs/manifest/kubernetes-manifest-generator-architecture/
    SlackBot͔Βௐ੔͢Δ৔ॴ
    ؀ڥ͝ͱͷόʔδϣϯ৘ใͳͲɻ ManifestҎ֎ʹ΋υΩϡϝϯτ
    ΋ग़ྗ͢Δ
    ܕఆٛ͸OpenAPI͔Βੜ੒͠
    ͨ΋ͷΛར༻͢Δ
    ֤ϚΠΫϩαʔϏε͝ͱͷςϯ
    ϓϨʔτΛੜ੒͓ͯ͘͠
    ςϯϓϨʔτͷ੾Γସ͑΍ࡉ͔
    ͍ύϥϝʔλʔΛௐ੔͢Δ
    ग़ྗ͢ΔϑΝΠϧߏ଄ͷ؅ཧ
    ༷ʑͳϑΥʔϚοτ
    ʹग़ྗ͢Δॲཧ
    ߹੒ͷ৔ॴ
    Port,Namespace,
    ଞͷMSͷ৘ใͳ
    Ͳͷݻఆ஋
    

    View Slide

18. labelɾannocationͷࣗಈ෇༩
    ● app: micro-service-aͱ͍͏ܗࣜͰؔ࿈͢Δίϯϙʔ
    ωϯτʹશͯ෇༩
    kubectl get service,deployment,configmap,virtualservice -l
    app=microservice-a
    ͕Մೳ
    ● ϚΠΫϩαʔϏεͷόʔδϣϯΛߋ৽͢Δͨͼʹ
    annotationͷ஋΋มԽ͢ΔͨΊɺDataDogͷϩάͷ෼
    ղೳ͕޲্
    TypeScriptͰॻ͘͜ͱʹΑͬͯԿΛ࣮ݱ͔ͨ͠ʢ1/2ʣ
    kube_app_versionͰάϧʔϓԽͯ͠ՄࢹԽͨ͠ྫ
    ͲͷλΠϛϯάͰόʔδϣϯ͕੾ΓସΘͬͨͷ͔ՄࢹԽ͞Εɺ
    ໰୊ͷૣظൃݟʹͭͳ͕Δɻ
    Reference:
    Handbook: https://dwango.github.io/nicolive-kubernetes-migration-handbook-2022/docs/performance/monitoring/
    DataDogͷλάநग़: https://docs.datadoghq.com/ja/containers/kubernetes/tag/?tab=containerizedagent
    

    View Slide

19. Reference:
    Handbook: https://dwango.github.io/nicolive-kubernetes-migration-handbook-2022/docs/manifest/kubernetes-manifest-generator-architecture/
    ߏ੒มߋ͕؆୯͔ͭਝ଎ʹͰ͖Δ
    ● ߏ੒͕ςϯϓϨʔτԽ͞Ε͍ͯΔͨΊɺύϥϝʔλʔ̍ͭͷߋ৽Ͱେ୾ͳߏ੒มߋ͕ՄೳʹͳΔ
    શମ࠷దԽ͕୹࣌ؒͰՄೳ
    ● શͯͷManifest͸ੜ੒͞ΕΔͨΊ਺ͷ໰୊Ͱ͸ͳ͘ͳͬͨɻ
    Manifest͸શͯ୯ҰϦϙδτϦ಺ʹ׬શͳঢ়ଶͰग़ྗ͢Δ
    ● Manifest͸׬શͳঢ়ଶͰϦϙδτϦʹอଘ͞ΕΔͨΊࠩ෼͕ͦͷ··ͷҙਤΛද͢ʢม਺͕հࡏ͠ͳ͍ͷͰ೴಺Ͱ૊ΈཱͯΔඞཁ͕
    ͳ͍ʣ
    ● GitHub্ͰͷݕࡧʹHit͠΍͘͢ͳΔ
    ൚༻ݴޠΛͦͷ··ར༻͢ΔͨΊ඼࣭Λҡ࣋͢Δख๏Λ༻ҙ͠΍͍͢
    ● ໋໊نଇ΍஋ͷ্ݶɺԼݶͷςετͳͲ
    TypeScriptͰॻ͘͜ͱʹΑͬͯԿΛ࣮ݱ͔ͨ͠ʢ2/2ʣ
    

    View Slide

20. ArgoCD + Slack Bot
    

    View Slide

21. ArgoCDͷSyncͷ༏ઌ౓
    Parent Apps
    Appsͷॱং؅ཧ
    ᶅApps
    ֤ϚΠΫϩαʔϏε
    ᶄApps
    Secretͷ࡞੒
    Root
    ج఺
    ᶃNamespace
    Namespaceͷ࡞੒
    App of AppsύλʔϯΛར༻ͯ͠੹຿෼ׂ
    ● ᶃ Namespaceͷ࡞੒
    ● ᶄ Secret
    ● Secret͸ϚΠΫϩαʔϏεͰར༻͢Δ஋͕֨ೲ͞Ε͍ͯΔ͕ɺ
    ؅ཧऀΛݶఆ͢ΔͨΊ෼ׂ
    ● ᶅ ֤ϚΠΫϩαʔϏε
    ● NamespaceͱSecret͕લஈ֊Ͱ࡞੒͞Ε͍ͯΔͨΊؾ݉Ͷͳ͘σϓϩΠͰ͖Δ
    ● ͜ͷ഑ԼʹσϓϩΠ͍ͨ͠ΞϓϦέʔγϣϯ͕࿈ͳΔ
    ● ׆ൃʹมߋ͕ߦΘΕΔϚΠΫϩαʔϏε܈
    Reference:
    Handbook: https://dwango.github.io/nicolive-kubernetes-migration-handbook-2022/docs/ci/argo-cd/
    How Do I Configure Waves?: https://argo-cd.readthedocs.io/en/stable/user-guide/sync-waves/#how-do-i-configure-waves
    

    View Slide

22. PcWeb
    path: /overlays/[env]/pc-web
    ArgoCD AppsͷࢀরઌϦϙδτϦ
    ᶅApps
    ֤ϚΠΫϩαʔϏε
    BasicPcWeb
    path: /overlays/[env]/basic-pc-web
    BasicSpWeb
    path: /overlays/[env]/basic-sp-web
    IstioIngressGateway
    path: /overlays/[env]/istio-ingress-gateway
    ϞϊϨϙ಺ͷσΟϨΫτϦߏ੒
    ● overlays/[σϓϩΠ؀ڥ]/[ϚΠΫϩαʔϏε໊]
    ※ NamespaceͱSecretͷManifest͸আ͘
    Reference:
    Handbook: https://dwango.github.io/nicolive-kubernetes-migration-handbook-2022/docs/ci/argo-cd/
    Sample Repositroy: https://github.com/Himenon/kubernetes-template
    

    View Slide

23. @bot update version:2.0.0 app:pc-web
    ● Slack্ͰBotʹMentionΛඈ͹͢
    @bot merge pr:123
    ● ࡞੒͞ΕͨPull RequestΛϚʔδ͢Δ
    @bot create next release
    ● ຊ൪޲͚ʹ࣍ͷϦϦʔεΛ࡞੒
    ○ ࣍ͷόʔδϣϯͷܾఆʢSemantic Releaseʣ΍
    νέοτͷ࡞੒ɺϦϦʔεϊʔτͷ࡞੒ͳͲ
    ॾʑͷ࡞ۀ΋ؚΉ
    → ೔ৗͷߋ৽͸git clone͠ͳͯ͘΋ྑ͍
    Slack BotʹΑΔόʔδϣϯߋ৽ͷࣗಈԽ
    Reference:
    Handbook: https://dwango.github.io/nicolive-kubernetes-migration-handbook-2022/docs/ci/slack-bot/
    Sample Repositroy: https://github.com/Himenon/template-slack-bolt
    

    View Slide

24. ݕূ࡞ۀ
    

    View Slide

25. ● ૄ௨֬ೝ
    ○ ϚΠΫϩαʔϏεؒͷ઀ଓ͕Ͱ͖Δ͔
    ○ Ϋϥελʔͷ಺֎ͱͷૄ௨͕Ͱ͖Δ͔
    ● ෛՙ଱ੑ
    ○ େྔͷΞΫηε͕͋ͬͨ৔߹ʹ௨৴Λҡ࣋Ͱ͖Δ͔
    ○ ௨৴༰ྔͷΩϟύγςΟΛ௒͑ͳ͍͔
    ● ηΩϡϦςΟ
    ○ ඞཁ࠷௿ݶHeader৘ใͷΈΛར༻͢Δ
    ○ Ϋϥελʔ֎ʹର͢ΔΞΫηε੍ݶ
    ͜ΕΒΛ֬ೝ͍ͯͨ͘͠Ί͍ʹ͸ϩΪϯάͱϞχλϦϯάج൫͕ඞཁ
    ● Grafana / Prometheus / DataDogΛར༻
    ԿΛݕূ͢Δ͔
    Reference:
    Handbook: https://dwango.github.io/nicolive-kubernetes-migration-handbook-2022/docs/performance/load-test/
    

    View Slide

26. ݕূͷਐΊํʢ1/3ʣ
    γϯϓϧͳHTTPαʔόʔΛ༻ҙͯ͠ຊ൪؀ڥ·ͰઌߦϦϦʔε͢Δ
    ● ඞཁͳݴޠʢgo, nodejs, ruby, python, etc…)Ͱ༻ҙ͢Δͱͳ͓ྑ͍
    ● ͨͱ͑૯౰ΓͰΞΫηε͞Εͯ΋໰୊͕ͳ͍Α͏ͳαʔϏεΛຊ൪ͷKubernetesʹσϓϩΠ
    HTTP Server
    Browser
    ࣍ͷΑ͏ͳ΋ͷΛ࣮૷͓ͯ͘͠ͱݕূ͕ḿΔ
    ΫΤϦύϥϝʔλʔ΍HeaderͰઃఆͰ͖Δͱศར
    ● Logग़ྗ
    ● Metricsͷग़ྗ
    ● ٙࣅߴෛՙॲཧ
    ○ CPU Intensive
    ○ MEM Intensive
    ● ஗ԆϨεϙϯε
    ○ setTimeoutͰ3000msec଴ͨͤΔͳͲ
    ● αΠζͷେ͖͍Ϩεϙϯε
    ○ ඇѹॖͷը૾΍ڊେͳJSONΛΘ͟ͱฦ͢ͳͲ
    ● HTTP2ͷ௨৴
    Reference:
    Sample Repositroy: https://dwango.github.io/nicolive-kubernetes-migration-handbook-2022/docs/performance/load-test/#ࢼݧํ๏
    

    View Slide

27. ݕূͷਐΊํʢ2/3ʣ
    ݕূ؀ڥͰ͸࿈ଓతʹϦΫΤετΛૹΓଓ͚Δ
    ● ࿈ଓతͳϦιʔεͷมԽ΍ϩά͕ه࿥ͱͯ͠࢒Δ
    ○ ։ൃ؀ڥͰ͸ؾܰʹ͋ͪͪ͜Ͱߋ৽͍ͯ͠ΔͨΊɺߋ৽࣌ͷෆ۩߹͕ૣظʹᖰΓग़ͤΔ
    ● ݕূ؀ڥ༻ͷΞϥʔτ΋ਖ਼͘͠ൃใͤ͞Δ
    ● Horizontal Pod Autoscalar͕࣮ߦ͞ΕΔλΠϛϯά͕ഽؒͰ͔ͭΊΔ
    ○ → ͜ͷઃఆ͸ةͳ͍ͳͱ͍͏ײ͕֮਎ʹͭ͘
    ● Kubernetes্ͰNodejsΛಈ͔ͨ͠ͱ͖ͷৼΔ෣͍͕͔ͭΊΔ
    → ϦΫΤετͷঢ়گ΋։ൃ/ຊ൪ҰகΛ໨ࢦ͢
    Reference:
    Sample Repositroy: https://github.com/Himenon/http-echo
    X. ։ൃ/ຊ൪Ұக: https://12factor.net/ja/dev-prod-parity
    

    View Slide

28. ݕূͷਐΊํʢ3/3ʣ
    ຊ൪؀ڥͰϦΫΤετͷϛϥʔϦϯάΛ࣮ࢪ͢Δ
    ● nginxͷϦΫΤετͷϛϥʔϦϯάػೳΛར༻
    ● ຊ൪ͱಉ౳ͷϦΫΤετΛड͚෇͚ͯૄ௨ͱෛՙͷ
    ঢ়ଶͷಈ࡞֬ೝΛ࣮ࢪ
    → ͜͜·Ͱ΍Ε͹ෛՙͱૄ௨ͷݒ೦͕΄ͱΜͲແ͘ͳΔɻ
    Reference:
    Handbook: https://dwango.github.io/nicolive-kubernetes-migration-handbook-2022/docs/performance/load-test/#proxy͔ΒϦΫΤετΛmirroring͢Δ
    ngx_http_mirror_module: http://nginx.org/en/docs/http/ngx_http_mirror_module.html
    

    View Slide

29. ݕূΛͯ͠Θ͔Δ͜ͱͷ·ͱΊ
    ݕূͯ͠Θ͔Δ͜ͱ ݪҼ / ରࡦ
    ։ൃ/ຊ൪؀ڥͰͷ݁߹֬ೝ ඞཁͳࣄ຿࡞ۀɺࣄલͷઃఆͷߋ৽ɺ࠶ىಈͷ༗ແͳͲ͕Θ͔Δ
    ೔ৗͷϦϦʔε࡞ۀͰϘτϧωοΫʹͳΔλεΫ Slack BotͰࣗಈԽରԠ
    Graceful ShutdownΛ࣮૷͍ͯ͠ͳ͍ Retryॲཧͷ͓͔͛ͰͰϢʔβʔӨڹ͸ͳ͍͕ɺμογϡϘʔυ্ͰΤϥʔͱͯ͠؍ଌ͞Ε͍ͯΔ
    ௨৴ͷϘτϧωοΫ Pod਺͕଍Γͳ͍ɻRate Limit͕؇͍ͳͲ
    HPA͕ىಈ͢Δ͔Ͳ͏͔ େ఍ͷ৔߹ܭࢉϛε
    Memory Leak MEMͷ࢖༻ྔͷඍ෼ΛऔΔͱϝϞϦϦʔΫ͍ͯ͠Δ͜ͱΛ͕Θ͔Δɻ
    1ϲ݄ޙʹOut of memory͢Δ͕ͦΕ·Ͱʹߋ৽͞Ε͍ͯΔͨΊϦηοτ͞Ε͍ͯͨɻ
    ΞϓϦέʔγϣϯͷੑೳݶքʢԼݶ/্ݶʣ Nodejs(expressjs)ΛKubernetes্ͰՔಇͤͨ͞৔߹ͷͦ΋ͦ΋ͷ্ݶ͕Θ͔Δɻ
    ্ݶ = ͋Δͱ͜Ζ͔Βrps͕৳ͼͳ͘ͳΔͱ͜Ζ
    Nodeͷ࢖༻཰͕ภ͍ͬͯΔ PodAntiAffinityΛར༻ͯ͠PodΛ෼ࢄ഑ஔͤ͞Δ
    Server Side RenderingͱAPI͕ಉډ͍ͯ͠ΔΑ͏ͳPod͸ޓ͍ʹϦι
    ʔεΛ৯͍߹ͬͯΔ
    VirtualServiceΛར༻ͯ͠URI୯ҐͰPodΛ෼ׂ͢ΔΑ͏ʹରԠ
    ϞχλϦϯάͷ݁Ռ͔ΒϦΫΤετ਺ͱϦιʔε࢖༻ྔͷ૬ؔΛ༧ଌ
    Ͱ͖Δ
    େن໛ͳΠϕϯτͰ͸Ͳͷఔ౓ඞཁ͔ɺϛʔςΟϯάͰૈݟੵ΋ΓΛग़ͤΔΑ͏ʹͨ͠
    

    View Slide

30. ΫϥελʔͷҠߦ
    

    View Slide

31. Ҡߦͷํ๏
    ετϥϯάϥʔ ϑΟά ύλʔϯ
    Stranglerύλʔϯ
    ● ετϥϯάϥʔ ϑΟά ύλʔϯ - Azure Architecture Center | Microsoft Docs
    ػೳͷಛఆͷ෦෼Λ৽͍͠ΞϓϦέʔγϣϯ΍αʔϏεʹঃʑʹஔ͖׵͑Δ͜ͱͰɺϨΨγ γεςϜΛஈ֊తʹҠߦ͠·͢ɻ
    ϨΨγ γεςϜ͔Βͷػೳ͕ஔ͖׵͑ΒΕ͍ͯ͘ͱɺ৽͍͠γεςϜ͸࠷ऴతʹݹ͍γεςϜͷػೳ͢΂ͯΛஔ͖׵͑ɺݹ͍
    γεςϜΛ཈ѹͯ͠࢖༻ఀࢭͰ͖ΔΑ͏ʹ͠·͢ɻ
    ઌߦࣄྫ
    ● ৽نߏங΍Ҡߦ࣌ͷϦεΫΛܰݮɺʮετϥϯάϥʔύλʔϯʯͱ͸ʁɿಛूɿϚΠΫϩαʔϏεೖ໳ʢ5ʣ - ˏIT
    ● EnvoyΛFront Proxyͱͯ͠ར༻͢Δ - Uzabase for Engineers
    

    View Slide

32. γϯϓϧͳHTTPαʔόʔͱϛϥʔϦϯάͷݕূΛ׬ྃͨ͠ޙ
    Phase 1: Ҡߦલͷঢ়ଶ
    Phase 2: Ҡߦதͷঢ়ଶ
    ● IngressGateway͔ΒDocker SwarmʹProxy
    ● IngressGatewayͷෛՙΛ֬ೝ͢Δ
    Phase 3: Ҡߦޙͷঢ়ଶ
    Phase 2ɺ3͸ݕূʹΑͬͯݟੵ΋Γ݁Ռ͕͋ΔͨΊɺ͔ͦ͜Βେ
    ͖ͣ͘Εͳ͍ݶΓ͸PhaseΛ໭͢͜ͱ͸ͳ͍ɻ
    χίχίੜ์ૹͷBFFαʔόʔ͸̏ͭͷҠߦϑΣʔζͰҠߦ
    

    View Slide

33. ϚΠΫϩαʔϏεͷҠߦͷखॱ
    Ҡߦͷॱ൪
    ● جຊઓུͱͯ͠ϦΫΤετ਺ͷখ͞ͳϚΠΫϩαʔϏε͔Β։࢝͢Δ
    ● ӡ༻ʹඞཁͳσʔλ͕ෆ଍͍ͯ͠Δ৔߹͸Phase 3͔ΒPhase 1΍2·Ͱ໭͢͜ͱ΋͋Δ
    Ҡߦͷप஌
    ● εϓϨουγʔτͰͲͷΞϓϦέʔγϣϯΛͲͷ࣌ظʹɺͲͷPhaseʹ͢͢ΊΔ͔ΛνʔϜ಺֎ʹप஌
    ● มߋ͕͋ͬͨ৔߹͸೔࣌ͷϛʔςΟϯάͰ࠶౓प஌
    ※ Ҡߦঢ়ଶʹ໊લΛ͚ͭͯ఻ୡ͢Δ͜ͱͰೝࣝ߹Θͤɺਐḿ؅ཧΛ༰қʹ͍ͯ͠Δɻ
    ࡾาਐΜͰೋาԼ͕Δײ͡ͰҠߦΛ׬਱ɻ
    

    View Slide

34. શମ·ͱΊ
    TypeScriptͰManifestΛ؅ཧ͍ͯ͠Δ
    ● WebϑϩϯτΤϯυͷνʔϜ͕ӡ༻Ͱ͖Δঢ়ଶʹɻυΩϡϝϯτ͸TSDocʹهࡌɻ
    ● ExceptionΛthrow͢Ε͹ͦΕ͕σετʹɻ
    ● ։ൃ࣌͸ؾʹͯ͠ͳ͍ӡ༻ʹඞཁͳύϥϝʔλʔ͸ࣗಈతʹ෇༩͞ΕΔΑ͏ͳ࢓૊Έʹͨ͠
    ● μΠφϛοΫͳߏ੒มߋ͕؆୯ʹ
    ● ຊ࣭తͳ࡞ۀʹूதͰ͖Δ
    ArgoCD + Slack Bot
    ● ArgoCD͸৽نͷΫϥελʔ্ཱͪ͛௚࣌͠ʹށ࿭Θͳ͍Syncͷઃܭʹ
    ● SlackBotʹΑͬͯҠߦதɺҠߦޙͷӡ༻࣌ͷෛՙΛԼ͛Δ
    ݕূ࡞ۀ
    ● ϝτϦΫεͷऩूͱՄࢹԽ
    ● γϯϓϧͳHTTPαʔόʔΛຊ൪ϦϦʔεͯ݁͠߹֬ೝ
    ● ։ൃ؀ڥͰ͸ϦΫΤετΛૹΓଓ͚ɺϩάɾϝτϦΫεɾߋ৽࣌ͷॠஅͷ৘ใͳͲΛऩू͠ରࡦʹճ͢
    Ҡߦ࡞ۀ
    ● ετϥϯάϥʔ ϑΟά ύλʔϯͰ࣮ࢪ
    ● ϞχλϦϯάͭͭ͠૝ఆ಺ͷڍಈΛ͍ͯ͠Ε͹ϦϦʔε
    ● νʔϜ಺ͷप஌
    

    View Slide