Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ECSからEKSへの移行への移行事例の紹介

 ECSからEKSへの移行への移行事例の紹介

発表資料の中に出てくるURLです。

* [Amazon EKSでのArgoCDを使ったGitOps CD](https://tech.recruit-mp.co.jp/infrastructure/gitops-cd-by-using-argo-cd-at-eks/)
* [EnvoyとAmazon ECS Service Discoveryを利用したgRPCの負荷分散](https://tech.recruit-mp.co.jp/infrastructure/post-17098/)
* [Amazon EKSでAWS App Meshを利用してgRPCサーバーを運用する](https://tech.recruit-mp.co.jp/infrastructure/post-20765/)
* [https://tech.recruit-mp.co.jp/infrastructure/post-19364/](https://tech.recruit-mp.co.jp/infrastructure/post-19364/)
* [JenkinsのJobを CronJobに移した話](https://speakerdeck.com/yutachaos/the-story-of-moving-jenkins-job-to-cronjob)
* [EKS クラスタの CronJob を Fargate 上で実行する方法](https://tech.recruit-mp.co.jp/infrastructure/post-20631/)

E3d96b25e645921d9e5acbaa36c507a5?s=128

Masato Oshima

November 20, 2020
Tweet

Transcript

  1. &$4͔Β&,4΁ ͷҠߦࣄྫͷ঺հ !@NQPO +"846(ίϯςφࢧ෦

  2. Masato Oshima github.com/mpon @_mpon Software Engineer

  3. ίϯςφࢧ෦ͷൃද͸3ճ໨Ͱ͢ 2017/12 2018/12

  4. ίϝϯτ΍࣭໰ੋඇ͓ئ͍͠·͢ ➤ ਖ਼௚ɺ΋ͬͱ͍͍ํ๏͋Δ͔΋ͱ͍͏ෆ҆ɾɾ ➤ ͳͷͰποίϛ΋Β͑Δͱخ͍͠Ͱ͢ʂ ➤ ͦΜͳ͜ͱ͠ͳͯ͘΋ࠓͳΒ͜ΕͰͰ͖Δͷʹɻͱ͔ ➤ ͬͪ͜ͷπʔϧ࢖͑͹Α͔ͬͨͷʹɻͱ͔ ➤

    ͳΜͰ͜͏͍͏ߏ੒ʹͨ͠ͷʁͱ͔ ➤ ͜͏͍͏έʔεͬͯରԠͰ͖ͯ·͔͢ʁͱ͔ ➤ ͕͜͜Α͘෼͔Βͳ͔ͬͨɻͱ͔
  5. AGENDA ➤ ελσΟαϓϦENGLISHʹ͍ͭͯ ➤ ୈҰ෦: ECS͔ΒEKS΁Ҡߦͷഎܠ ➤ ՝୊1: ؀ڥ૿΍͢ͷ͕ਏ͔ͬͨ ➤

    ՝୊2: Ͳ͜ʹԿ͕σϓϩΠ͞ΕͯΔ͔೺Ѳ͠ʹ͔ͬͨ͘ ➤ ՝୊3: gRPCͷෛՙ෼ࢄ ➤ ୈೋ෦: EKSҠߦޙͷߏ੒ʹ͍ͭͯ ➤ Spot OceanΛ࢖ͬͨΫϥελʔ؅ཧ ➤ JenkinsͷJobΛCronJobʹҠߦʢFargateͷ࿩΋͋ΔΑʣ ➤ ·ͱΊɿίϯςφ͸͍͍ͧ
  6. ελσΟαϓϦENGLISH ➤ ΦϯϥΠϯӳޠֶशαʔϏε ➤ ӳձ࿩Ի੠΍ը૾ɾಈըͳͲΛ഑৴ ➤ ΫΠζܗࣜͰֶशσʔλɺཤྺΛอଘ ➤ iOS, Android,

    WebͰར༻Մೳ
  7. αʔϏεߏ੒ͷ֓ཁ

  8. αʔϏεߏ੒ͷ֓ཁ ໿50αʔϏε ɾɾɾ ɾ ɾ ɾ dev1, dev2... ໿20؀ڥ

  9. ୈҰ෦ ECS͔ΒEKS΁Ҡߦ͠ ͨഎܠʹ͍ͭͯ

  10. ՝୊1 ؀ڥ૿΍͢ͷ͕ਏ ͔ͬͨ

  11. ECSͷͱ͖ͷߏ੒؅ཧํ๏ ෼ྨ ίϯϙʔωϯτ ઃఆํ๏ Ծ૝Ϋϥελʔ ECS Cluster Terraform Πϯελϯε Autoscaling

    Group Terraform αʔϏεεέδϡʔϥ ECS Service Terraform αʔϏεσΟεΧόϦ Cloud Map Terraform ίϯςφఆٛ ECS Task Definition Terraform + ಠࣗπʔϧ ϩʔυόϥϯαʔ ALB Terraform αʔϏεͻ΋͚ͮ ALB Target Group Terraform ϧʔςΟϯά ALB Listenr Rule Terraform
  12. ECSͰ؀ڥΛͨ͘͞Μ૿΍͢ʹ͸ʁ ➤ TerraformͰECS ClusterɺECS ServiceɺALBͳͲ৭ʑ࡞Δ ➤ Task Definitionʢಠࣗπʔϧͷ࢓༷ͷYAMLʣΛେྔʹίϐʔ ➤ ςϯϓϨʔτͷػೳ͕ͳ͘؀ڥ͝ͱʹ؀ڥݻ༗ͷ஋ʹॻ͖׵͑Δ

    ➤ ؀ڥ૿΍͢ͷʹTerraformͰΠϯελϯε΍ALBΛ૿΍͞ͳ͍ͱ͍͚ͳ͍ ➤ Terraform͸ModuleԽ͸͚ͨ͠Ͳɺ͋ͱ͔ΒมߋೖΕΔͱޓ׵ੑอͭͷ ʹେมͳ͜ͱ͕͋ͬͯؾܰʹmoduleΛมߋͰ͖ͳ͍ ➤ σϓϩΠͰมߋ͕ى͖Δ΋ͷΛTerraformͰ؅ཧͯ͠͠·͍ͬͯͨͷͰ ignore_changesͰແཧ΍Γແࢹ ➤ ϥΠϑαΠΫϧͷҧ͍ΛݟۃΊΒΕ͍ͯͳ͔ͬͨɾɾ
  13. EKSͷߏ੒؅ཧํ๏ ෼ྨ ίϯϙʔωϯτ ઃఆํ๏ Ծ૝Ϋϥελʔ Namespace YAML(k8s) Πϯελϯε EC2Πϯελϯε Terraform

    + Spot Ocean αʔϏεεέδϡʔϥ Deployment YAML(k8s) αʔϏεσΟεΧόϦ Service YAML(k8s) ίϯςφఆٛ Pod YAML(k8s) ϩʔυόϥϯαʔ Ingress YAML(k8s) αʔϏεͻ΋͚ͮ Ingress YAML(k8s) ϧʔςΟϯά Ingress YAML(k8s)
  14. EKSͰ؀ڥΛͨ͘͞Μ૿΍͢ʹ͸ʁ ➤ Terraformͷग़൪͸EKS ClusterΛ࠷ॳʹ࡞Δ͚ͩͰ؀ڥ૿΍͢ͱ ͖ʹ͸͍Βͳ͍ ➤ ؀ڥ૿΍͢৔߹͸NamespaceΛ૿΍͢ ➤ KustomizeΛ࢖ͬͯ؀ڥ͝ͱʹมΘΔ෦෼͚ͩΛ௥Ճ ➤

    ֎෦͔ΒͷϦΫΤετͷϚοϐϯά͸Ingress ➤ ಺෦௨৴͸ServiceΦϒδΣΫτͰ؆୯αʔϏεσΟεΧόϦɺ CoreDNS͕͋ΔͷͰRoute53ͳ͠Ͱ΋಺෦ͰDNS͕༻ҙ ➤ ΄ͱΜͲKubernetesͷΦϒδΣΫτͰ׬݁͢ΔͷͰએݴతʹ؅ཧ ͠΍͍͢
  15. ΫϥελʔΛ࿦ཧతʹ෼ׂͰ͖Δ Namespace dev1 EC2 EC2 EC2 Cluster EC2 EC2 EC2

    Namespace dev2 Namespace dev3 ➤ Kubernetesʹ͸Namespace ͱ͍͏ΦϒδΣΫτ͕͋Δ ➤ Cluster͑͞࡞ͬͯ͠·͑͹ ͋ͱ͸NamespaceΛ૿΍͢ ͚ͩͰ؀ڥ͕૿΍ͤΔ ➤ Namespace͝ͱͷDNSΤϯ τϦ΋࡞੒͞ΕΔ ➤ ͜Μͳ୯७ͳYAMLͰOK
  16. Kustomize, HelmͳͲσϑΝΫτͳߏ੒؅ཧ͕͋Δ

  17. Kustomizeͷoverlays

  18. ՝୊2 Ͳ͜ʹԿ͕σϓϩΠ͞Ε ͯΔ͔೺Ѳ͠ʹ͔ͬͨ͘

  19. ECSͷͱ͖ʹײ͍ͯͨ͡՝୊ ➤ hoge؀ڥͷfugaαʔϏεͬͯࠓσϓϩΠ͞ΕͯΔͷ͸Ͳͷϒϥϯνʁ ➤ ίϯςφͷ؀ڥม਺͸Կ͕ઃఆ͞ΕͯΔʁ ➤ aws ecs list-servicesͱ͔ͰҰൃͰͲͷtag͕σϓϩΠ͞ΕͯΔ͔೺ѲͰ͖ ͳ͍ͷͰࣗ࡞πʔϧ࡞͙ͬͯ྇

    ➤ σϓϩΠͷઃఆʢJenkinsfileʣʹઃఆ͕ͲΜͲΜ૿͍͑ͯͬͯΧΦεʹ ͳ͖ͬͯͨ ➤ Α͘σϓϩΠ͢ΔAPIͳͲͷαʔόʔΞϓϦέʔγϣϯͱ୯ͳΔnginxͱ ͔ͷσϓϩΠ͸σϓϩΠํ๏͕ผʑʹ༻ҙͯ͠ϝϯς͍ͯͨ͠ ➤ gitϦϙδτϦΛݟΕ͹Կ͕σϓϩΠ͞ΕͯΔ͔෼͔ΔΑ͏ʹ͍ͨ͠ ➤ gitops͍ͨ͠
  20. GITOPS͍ͨ͠

  21. gitopsπʔϧͱͯ͠Argo CDͷಋೖ https://tech.recruit-mp.co.jp/infrastructure/gitops-cd-by-using-argo-cd-at-eks/

  22. Argo CDͷArchitecture - k8sͷReconciliation Loop gitϦϙδτϦΛ ݟΕ͹Կ͕σϓϩΠ͞Ε ͍ͯΔ͔೺ѲͰ͖Δ gitϦϙδτϦͱ ࠩ෼Λఆظతʹ

    νΣοΫͯ͠ ࠩ෼͕͋Ε͹ σϓϩΠ
  23. Dashboard͕͋ΔͷͰDeveloper Friendly GUI͕͋Δͱ։ൃऀʹ΋ ֓೦Λཧղͯ͠΋Β͍΍͍͢

  24. Sync Phase and WavesͰσϓϩΠͷґଘؔ܎ΛදݱͰ͖Δ

  25. Clusterʹඞཁͳresource΋gitopsͰ ➤ Ingress Controller΍ Datadog AgentͳͲ΋؅ཧ Ͱ͖Δ ➤ Helm Chart

    Repositoriesͷ ػೳ͕͋ΔͷͰArgo CDͰ Helm ChartΛιʔεʹ؅ཧ Մೳ ➤ ArgoCD ApplicationΛ࡞੒ ͢Ε͹ߏங׬ྃ
  26. ެࣜπʔϧͷkubectlͰ͙͢ʹ೺ѲͰ͖Δ ➤ ྫ͑͹ͲͷImage͕σϓϩΠ͞ΕͯΔ͔֬ೝ͢Δͷʹ͜Ε͚ͩͰOK ➤ `kubectl get deploy -o wide`

  27. ՝୊3 GRPCͷෛՙ෼ࢄ

  28. ౰࣌ͷgRPCͷෛՙ෼ࢄ https://tech.recruit-mp.co.jp/infrastructure/post-17098/

  29. ECSͷͱ͖ͷߏ੒

  30. σϓϩΠ࣌ʹϦΫΤετ͕མͪΔ໰୊ Blue gRPC Green gRPC Blue gRPC Green gRPC blue.grpc.internal

    green.grpc.internal Cloud Map lb.grpc.internal ➤ grpcαʔόʔΛRolling Update͢Δ ͱSTOPࡁΈͷίϯςφͷIP͕ฦͬ ͖ͯͯϦΫΤετ͕མͪͯ͠·͏ ➤ JenkinsͷpipelineͱshΛ૊Έ߹Θ ͤͯBlue/GreenσϓϩΠ ➤ ECS Task Definitionͷ؀ڥม਺Λ sedͰBlue->Greenʹॻ͖׵͑ͯ envoyΛRolloing Update ➤ Sleep 30͔ͯ͠ΒBlueΛ࡟আ ➤ ϫʔΫΞϥ΢ϯυײຬࡌͳͷͰͳ Μͱ͔͔ͨͬͨ͠
  31. App Meshͷ࠾༻ https://tech.recruit-mp.co.jp/infrastructure/post-20765/

  32. App Meshͷ࠾༻ ➤ gRPC routingΛαϙʔτ͍ͯͨ͠ ➤ কདྷతʹαʔϏεϝογϡͱͯ͠׆༻Ͱ͖Δ ➤ ͲͪΒʹ͠Ζenvoyͷ؅ཧ͸୭͔͕΍Δඞཁ͸͋Δ ➤

    envoyͷretryPolicyɺkubernetesͷPod LifecycleΛ׆༻ͯ͠ϦΫ ΤετΛམͱͣ͞ʹσϓϩΠͰ͖Δ
  33. App Meshͷߏ੒؅ཧ ➤ AWS App Mesh Controller for k8s͕༻ҙ͞Ε͍ͯΔ ➤

    APIͷίϯςφͷYAMLΛ͍ ͡Δ͜ͱͳ͘envoyΛInject ͯ͘͠ΕΔ ➤ k8sͷCRDͱͯ͠؅ཧͰ͖ ΔͷͰgitopsͰ͖Δ ➤ ։ൃ؀ڥΛ૿΍͢ͷ΋؆୯
  34. σϓϩΠ࣌ʹϦΫΤετ͕མͪΔ໰୊ ➤ App Mesh best practicesͱ ͍͏υΩϡϝϯτ͕༻ҙ͞ Ε͍ͯͯͦͷ௨Γʹ RetryPolicyΛઃఆ

  35. σϓϩΠ࣌ʹϦΫΤετ͕མͪΔ໰୊ ➤ PodͷpreStopͰsleepΛೖΕΔ͜ͱͰ Service͔Β੾Γ཭͞ΕΔͷΛ଴ͭ ➤ AWSαϙʔτͷํʹΞυόΠεΛ΋ Β͍ղܾ͠·ͨ͠ɻେײँ ➤ AWSαʔϏεͷ͜ͱ͔͠ฉ͍ͪΌ͍ ͚ͳ͍ͱࢥ͚ͬͯͨͲͦΜͳ͜ͱ͸ͳ

    ͔ͬͨ ➤ Kubernetes΍Envoyͷ͜ͱ΋αϙʔτ ͯ͘͠Εͨ ➤ Argo CDʹΑΔSync(Deploymentͷ Rolling Update)Ͱγϯϓϧʹσϓϩ ΠͰ͖ΔΑ͏ʹͳͬͨ
  36. ༨ஊ: ਓྨ͕଴๬͍ͯͨ͠ALBͷgRPCαϙʔτʂʂ ΋ͬͱૣ͘ཉ͔ͬͨ͠

  37. ୈೋ෦ EKSҠߦޙͷ ߏ੒ʹ͍ͭͯ

  38. EKSҠߦޙͷߏ੒ ECS EKS

  39. EKSҠߦޙͷߏ੒ ALB Ingress Controllerͱ Nginx Ingrss Controllerͷ ૊Έ߹Θͤ App MeshʹΑΔ

    EnvoyͷInject Cloud MapͰ Service Discovery
  40. SPOT OCEANΛ࢖ͬͨ Ϋϥελʔ؅ཧ

  41. ৄࡉ͸ͪ͜Βͷϒϩά https://tech.recruit-mp.co.jp/infrastructure/post-19364/

  42. Spot Ocean ੲ͸Spotinstͱ͍͏αʔϏεͩͬͨ ࠓ͸NetAppʹങऩ͞ΕͯSpotͱ͍͏໊લʹ

  43. Pod-Driven Scaling ➤ Pod͕UnscheduleʹͳΒͳ͍Α͏ʹ NodeΛΦʔτεέʔϦϯάͯ͘͠ ΕΔ ➤ PodʹׂΓ౰ͯΔrequests͚ͩΛߟ ͑Ε͹OK ➤

    Node͸ҙࣝ͠ͳͯ͘Α͍ͷͰ FargateͷΑ͏ͳ࢖͍উख ➤ Spot Instance͔Β࠷దͳΠϯελϯ εΛબΜͰ͘ΕΔ ➤ ΦϯσϚϯυͰಈ͍ͯཉ͍͠podʹ ͸annotation෇༩͢Δ͚ͩͰOK
  44. Terraform΍Custom ControllerͰߏ੒؅ཧͰ͖Δ ➤ Terraformͷprovider͕༻ҙ ͞Ε͍ͯΔ ➤ Spot OceanͷCustom Controller͕Helm ChartͰ

    ༻ҙ͞Ε͍ͯΔͷͰ؆୯Π ϯετʔϧ
  45. Headroom ➤ Headroom͸༨৒ͷϊʔυ Λ֬อ͓͍ͯͯ͘͠ΕΔػ ೳ ➤ ϦΫΤετͷεύΠΫ౳Ͱ PodͷεέʔϧΛૉૣ͘͢Δ

  46. Cluster Roll ➤ Cluster Roll͸Nodeͷ҆શ ͳRolling UpdateΛͯ͘͠Ε Δ ➤ NodeͷೖΕସ͍͑ͨ͠ࣄ৘

    ͕ൃੜͨ͠৔߹ͳͲʹศར
  47. JENKINSͷJOBΛ CRONJOBʹҠߦ

  48. ৄࡉ͸ͪ͜Β https://tech.recruit-mp.co.jp/infrastructure/post-20631/ https://speakerdeck.com/yutachaos/the-story-of-moving-jenkins-job-to-cronjob

  49. όον͸JenkinsͰ΍͍ͬͯͨ ➤ αʔόʔΞϓϦέʔγϣϯ ͱಉ͡ίϯςφImageΛ ࢖ͬͯλεΫΛىಈ ➤ JenkinsͷఆظτϦΨʔͰ࣮ ߦ ➤ ௨஌΋ϦτϥΠ΋ศརɺͳ

    ΜͰ΋Ͱ͖Δɻ͔͠͠ɾɾ
  50. Jenkinsศར͗͢໰୊ ➤ ศརա͗ͯWebը໘্Ͱ ͡ΌΜ͡ΌΜδϣϒ͕࡞Β Ε͍ͯ͘ ➤ ҰମԿ͕͍ͭىಈͯ͠Δ͔ Α͘෼͔Βͳ͍ ➤ ාͯ͘updateͰ͖ͣʹԘ௮

    ͚ʹɾɾ
  51. k8sͷCronJobʹ͢Δ͜ͱͰYAML؅ཧ ➤ CronJob͸ఆظతʹJobΛ࣮ߦ ͢Δk8sͷϦιʔε ➤ EKSҠߦͷӡ༻͸·ͣδϣϒҠ ߦ͔Βߦ͕ͬͨ͏·͍ͬͨ͘ ➤ YAMLͰ؅ཧͰ͖ΔͷͰgitops Ͱ͖Δ

    ➤ kubectlͰݟͯ΋͍͍͠ɺgitϨ ϙδτϦΛgrep͢Δ͚ͩͰԿ͕ ͍ͭͲ͜ͰJob͕ಈ͍ͯΔ͔͢ ͙ʹ෼͔Δ
  52. όον࣮ߦͷͨͼʹNodeͷ૿ݮͯ͠͠·͏ ➤ ఆظతʹόον࣮ߦͷͨͼ ʹPodͷRequestsʹԠͯ͡ Node͕૿ݮ ➤ όον࣮ߦதʹNode͕མͪ ΔͱࠔΔ ➤ όονͷ࣮ߦʹΑͬͯαʔ

    ϏεʹӨڹ͸༩͑ͨ͘ͳ͍
  53. EKS FargateͰCronJobΛ࣮ߦ͢Δ ➤ όον͸ίϯςφ͕ىಈ͠ ͯऴྃ͢Ε͹Α͍ͷͰ FargateʹͽͬͨΓ ➤ kubernetes͔Β͸Fargate͸ 1ͭͷnodeͷΑ͏ʹݟ͑Δ ➤

    FargateͰىಈͨ͠όον͕ طଘͷΫϥελʔͷϦιʔ εΛ࢖͏͜ͱ͸ͳ͍
  54. Sidecarίϯςφͷఀࢭ͕೉͍͠ ➤ Datadog Agent΍FluentdͳͲΛ Sidecarʹ͓͔͘͠ͳ͍ ➤ JobͷϝΠϯͷίϯςφ͕ఀࢭͯ͠΋ sidecarίϯςφ͸ఀࢭ͠ͳ͍ͷͰJob ͕ऴྃ͠ͳ͍ʂʂ ➤

    ऴྃ࣌ʹsidecarίϯςφΛఀࢭͤ͞ ΔͳͲͷ޻෉͕ඞཁ ➤ ECSͰ͍͏essential:trueͷػೳ͕ͳ͍ ➤ Sidecar Containers͕Kubernetes v1.19ʹೖΔ༧ఆ͕ͩͬͨະఆʹͳͬ ͯ͠·ͬͨʁ
  55. ·ͱΊͱࡶஊ ➤ ECS͔ΒEKSϔҠߦͨ͠࿩Λ͠·ͨ͠ ➤ ͋͘·Ͱ΋զʑͷ৔߹ͷ࿩ ➤ gitops΍KubernetesͷΤίγεςϜͰಠࣗ࢓༷Λͳͯ࣋͘͠ଓੑͷ͋ΔγεςϜ΁ ➤ App Mesh͸ຊ൪ར༻΋໰୊ͳ͠

    ➤ ECS͸ࠓͳΒFargate Spot΋͋Δ͠ALB͕gRPCαϙʔτͨ͠͠࠶ߟͯ͠Έ͍ͨ ➤ ecspressoΛ࢖͑͹ϥΠϑαΠΫϧͱςϯϓϨʔτͱ͔΋ͬͱ͏·͘Ͱ͖͔ͨ΋ ➤ ECSͰɺALBͱͷͻ΋͚ͮɺECS ServiceɺTask Definitionͱ͔Λ౷Ұతʹѻ͑Δπʔϧ ͕͋Ε͹͍͍ͷ͔΋ ➤ copilot͕ͦ͏ͳͷ͔ʁ ➤ ίϯςφ͸͍͍ͧ ➤ ͨͩ͠ɺRDSɺClodwatch LogsɺS3ͳͲϚωʔδυαʔϏε͕͋ͬͯͦ͜