ECSからEKSへの移行への移行事例の紹介

発表資料の中に出てくるURLです。

* [Amazon EKSでのArgoCDを使ったGitOps CD](https://tech.recruit-mp.co.jp/infrastructure/gitops-cd-by-using-argo-cd-at-eks/)
* [EnvoyとAmazon ECS Service Discoveryを利用したgRPCの負荷分散](https://tech.recruit-mp.co.jp/infrastructure/post-17098/)
* [Amazon EKSでAWS App Meshを利用してgRPCサーバーを運用する](https://tech.recruit-mp.co.jp/infrastructure/post-20765/)
* [https://tech.recruit-mp.co.jp/infrastructure/post-19364/](https://tech.recruit-mp.co.jp/infrastructure/post-19364/)
* [JenkinsのJobを CronJobに移した話](https://speakerdeck.com/yutachaos/the-story-of-moving-jenkins-job-to-cronjob)
* [EKS クラスタの CronJob を Fargate 上で実行する方法](https://tech.recruit-mp.co.jp/infrastructure/post-20631/)

Transcript

1. &$4͔Β&,4΁ ͷҠߦࣄྫͷ঺հ !@NQPO
   +"846(
   ίϯςφࢧ෦
   
2. Masato Oshima github.com/mpon @_mpon Software Engineer
3. ίϯςφࢧ෦ͷൃද͸3ճ໨Ͱ͢ 2017/12 2018/12
4. ίϝϯτ΍࣭໰ੋඇ͓ئ͍͠·͢ ➤ ਖ਼௚ɺ΋ͬͱ͍͍ํ๏͋Δ͔΋ͱ͍͏ෆ҆ɾɾ ➤ ͳͷͰποίϛ΋Β͑Δͱخ͍͠Ͱ͢ʂ ➤ ͦΜͳ͜ͱ͠ͳͯ͘΋ࠓͳΒ͜ΕͰͰ͖Δͷʹɻͱ͔ ➤ ͬͪ͜ͷπʔϧ࢖͑͹Α͔ͬͨͷʹɻͱ͔ ➤ ͳΜͰ͜͏͍͏ߏ੒ʹͨ͠ͷʁͱ͔ ➤ ͜͏͍͏έʔεͬͯରԠͰ͖ͯ·͔͢ʁͱ͔ ➤ ͕͜͜Α͘෼͔Βͳ͔ͬͨɻͱ͔
5. AGENDA ➤ ελσΟαϓϦENGLISHʹ͍ͭͯ ➤ ୈҰ෦: ECS͔ΒEKS΁Ҡߦͷഎܠ ➤ ՝୊1: ؀ڥ૿΍͢ͷ͕ਏ͔ͬͨ ➤ ՝୊2: Ͳ͜ʹԿ͕σϓϩΠ͞ΕͯΔ͔೺Ѳ͠ʹ͔ͬͨ͘ ➤ ՝୊3: gRPCͷෛՙ෼ࢄ ➤ ୈೋ෦: EKSҠߦޙͷߏ੒ʹ͍ͭͯ ➤ Spot OceanΛ࢖ͬͨΫϥελʔ؅ཧ ➤ JenkinsͷJobΛCronJobʹҠߦʢFargateͷ࿩΋͋ΔΑʣ ➤ ·ͱΊɿίϯςφ͸͍͍ͧ
6. ελσΟαϓϦENGLISH ➤ ΦϯϥΠϯӳޠֶशαʔϏε ➤ ӳձ࿩Ի੠΍ը૾ɾಈըͳͲΛ഑৴ ➤ ΫΠζܗࣜͰֶशσʔλɺཤྺΛอଘ ➤ iOS, Android, WebͰར༻Մೳ
7. αʔϏεߏ੒ͷ֓ཁ
8. αʔϏεߏ੒ͷ֓ཁ ໿50αʔϏε ɾɾɾ ɾ ɾ ɾ dev1, dev2... ໿20؀ڥ
9. ୈҰ෦ ECS͔ΒEKS΁Ҡߦ͠ ͨഎܠʹ͍ͭͯ
10. ՝୊1 ؀ڥ૿΍͢ͷ͕ਏ ͔ͬͨ
11. ECSͷͱ͖ͷߏ੒؅ཧํ๏ ෼ྨ ίϯϙʔωϯτ ઃఆํ๏ Ծ૝Ϋϥελʔ ECS Cluster Terraform Πϯελϯε Autoscaling Group Terraform αʔϏεεέδϡʔϥ ECS Service Terraform αʔϏεσΟεΧόϦ Cloud Map Terraform ίϯςφఆٛ ECS Task Definition Terraform + ಠࣗπʔϧ ϩʔυόϥϯαʔ ALB Terraform αʔϏεͻ΋͚ͮ ALB Target Group Terraform ϧʔςΟϯά ALB Listenr Rule Terraform
12. ECSͰ؀ڥΛͨ͘͞Μ૿΍͢ʹ͸ʁ ➤ TerraformͰECS ClusterɺECS ServiceɺALBͳͲ৭ʑ࡞Δ ➤ Task Definitionʢಠࣗπʔϧͷ࢓༷ͷYAMLʣΛେྔʹίϐʔ ➤ ςϯϓϨʔτͷػೳ͕ͳ͘؀ڥ͝ͱʹ؀ڥݻ༗ͷ஋ʹॻ͖׵͑Δ ➤ ؀ڥ૿΍͢ͷʹTerraformͰΠϯελϯε΍ALBΛ૿΍͞ͳ͍ͱ͍͚ͳ͍ ➤ Terraform͸ModuleԽ͸͚ͨ͠Ͳɺ͋ͱ͔ΒมߋೖΕΔͱޓ׵ੑอͭͷ ʹେมͳ͜ͱ͕͋ͬͯؾܰʹmoduleΛมߋͰ͖ͳ͍ ➤ σϓϩΠͰมߋ͕ى͖Δ΋ͷΛTerraformͰ؅ཧͯ͠͠·͍ͬͯͨͷͰ ignore_changesͰແཧ΍Γແࢹ ➤ ϥΠϑαΠΫϧͷҧ͍ΛݟۃΊΒΕ͍ͯͳ͔ͬͨɾɾ
13. EKSͷߏ੒؅ཧํ๏ ෼ྨ ίϯϙʔωϯτ ઃఆํ๏ Ծ૝Ϋϥελʔ Namespace YAML(k8s) Πϯελϯε EC2Πϯελϯε Terraform + Spot Ocean αʔϏεεέδϡʔϥ Deployment YAML(k8s) αʔϏεσΟεΧόϦ Service YAML(k8s) ίϯςφఆٛ Pod YAML(k8s) ϩʔυόϥϯαʔ Ingress YAML(k8s) αʔϏεͻ΋͚ͮ Ingress YAML(k8s) ϧʔςΟϯά Ingress YAML(k8s)
14. EKSͰ؀ڥΛͨ͘͞Μ૿΍͢ʹ͸ʁ ➤ Terraformͷग़൪͸EKS ClusterΛ࠷ॳʹ࡞Δ͚ͩͰ؀ڥ૿΍͢ͱ ͖ʹ͸͍Βͳ͍ ➤ ؀ڥ૿΍͢৔߹͸NamespaceΛ૿΍͢ ➤ KustomizeΛ࢖ͬͯ؀ڥ͝ͱʹมΘΔ෦෼͚ͩΛ௥Ճ ➤ ֎෦͔ΒͷϦΫΤετͷϚοϐϯά͸Ingress ➤ ಺෦௨৴͸ServiceΦϒδΣΫτͰ؆୯αʔϏεσΟεΧόϦɺ CoreDNS͕͋ΔͷͰRoute53ͳ͠Ͱ΋಺෦ͰDNS͕༻ҙ ➤ ΄ͱΜͲKubernetesͷΦϒδΣΫτͰ׬݁͢ΔͷͰએݴతʹ؅ཧ ͠΍͍͢
15. ΫϥελʔΛ࿦ཧతʹ෼ׂͰ͖Δ Namespace dev1 EC2 EC2 EC2 Cluster EC2 EC2 EC2 Namespace dev2 Namespace dev3 ➤ Kubernetesʹ͸Namespace ͱ͍͏ΦϒδΣΫτ͕͋Δ ➤ Cluster͑͞࡞ͬͯ͠·͑͹ ͋ͱ͸NamespaceΛ૿΍͢ ͚ͩͰ؀ڥ͕૿΍ͤΔ ➤ Namespace͝ͱͷDNSΤϯ τϦ΋࡞੒͞ΕΔ ➤ ͜Μͳ୯७ͳYAMLͰOK
16. Kustomize, HelmͳͲσϑΝΫτͳߏ੒؅ཧ͕͋Δ
17. Kustomizeͷoverlays
18. ՝୊2 Ͳ͜ʹԿ͕σϓϩΠ͞Ε ͯΔ͔೺Ѳ͠ʹ͔ͬͨ͘
19. ECSͷͱ͖ʹײ͍ͯͨ͡՝୊ ➤ hoge؀ڥͷfugaαʔϏεͬͯࠓσϓϩΠ͞ΕͯΔͷ͸Ͳͷϒϥϯνʁ ➤ ίϯςφͷ؀ڥม਺͸Կ͕ઃఆ͞ΕͯΔʁ ➤ aws ecs list-servicesͱ͔ͰҰൃͰͲͷtag͕σϓϩΠ͞ΕͯΔ͔೺ѲͰ͖ ͳ͍ͷͰࣗ࡞πʔϧ࡞͙ͬͯ྇ ➤ σϓϩΠͷઃఆʢJenkinsfileʣʹઃఆ͕ͲΜͲΜ૿͍͑ͯͬͯΧΦεʹ ͳ͖ͬͯͨ ➤ Α͘σϓϩΠ͢ΔAPIͳͲͷαʔόʔΞϓϦέʔγϣϯͱ୯ͳΔnginxͱ ͔ͷσϓϩΠ͸σϓϩΠํ๏͕ผʑʹ༻ҙͯ͠ϝϯς͍ͯͨ͠ ➤ gitϦϙδτϦΛݟΕ͹Կ͕σϓϩΠ͞ΕͯΔ͔෼͔ΔΑ͏ʹ͍ͨ͠ ➤ gitops͍ͨ͠
20. GITOPS͍ͨ͠
21. gitopsπʔϧͱͯ͠Argo CDͷಋೖ https://tech.recruit-mp.co.jp/infrastructure/gitops-cd-by-using-argo-cd-at-eks/
22. Argo CDͷArchitecture - k8sͷReconciliation Loop gitϦϙδτϦΛ ݟΕ͹Կ͕σϓϩΠ͞Ε ͍ͯΔ͔೺ѲͰ͖Δ gitϦϙδτϦͱ ࠩ෼Λఆظతʹ νΣοΫͯ͠ ࠩ෼͕͋Ε͹ σϓϩΠ
23. Dashboard͕͋ΔͷͰDeveloper Friendly GUI͕͋Δͱ։ൃऀʹ΋ ֓೦Λཧղͯ͠΋Β͍΍͍͢
24. Sync Phase and WavesͰσϓϩΠͷґଘؔ܎ΛදݱͰ͖Δ
25. Clusterʹඞཁͳresource΋gitopsͰ ➤ Ingress Controller΍ Datadog AgentͳͲ΋؅ཧ Ͱ͖Δ ➤ Helm Chart Repositoriesͷ ػೳ͕͋ΔͷͰArgo CDͰ Helm ChartΛιʔεʹ؅ཧ Մೳ ➤ ArgoCD ApplicationΛ࡞੒ ͢Ε͹ߏங׬ྃ
26. ެࣜπʔϧͷkubectlͰ͙͢ʹ೺ѲͰ͖Δ ➤ ྫ͑͹ͲͷImage͕σϓϩΠ͞ΕͯΔ͔֬ೝ͢Δͷʹ͜Ε͚ͩͰOK ➤ `kubectl get deploy -o wide`
27. ՝୊3 GRPCͷෛՙ෼ࢄ
28. ౰࣌ͷgRPCͷෛՙ෼ࢄ https://tech.recruit-mp.co.jp/infrastructure/post-17098/
29. ECSͷͱ͖ͷߏ੒
30. σϓϩΠ࣌ʹϦΫΤετ͕མͪΔ໰୊ Blue gRPC Green gRPC Blue gRPC Green gRPC blue.grpc.internal green.grpc.internal Cloud Map lb.grpc.internal ➤ grpcαʔόʔΛRolling Update͢Δ ͱSTOPࡁΈͷίϯςφͷIP͕ฦͬ ͖ͯͯϦΫΤετ͕མͪͯ͠·͏ ➤ JenkinsͷpipelineͱshΛ૊Έ߹Θ ͤͯBlue/GreenσϓϩΠ ➤ ECS Task Definitionͷ؀ڥม਺Λ sedͰBlue->Greenʹॻ͖׵͑ͯ envoyΛRolloing Update ➤ Sleep 30͔ͯ͠ΒBlueΛ࡟আ ➤ ϫʔΫΞϥ΢ϯυײຬࡌͳͷͰͳ Μͱ͔͔ͨͬͨ͠
31. App Meshͷ࠾༻ https://tech.recruit-mp.co.jp/infrastructure/post-20765/
32. App Meshͷ࠾༻ ➤ gRPC routingΛαϙʔτ͍ͯͨ͠ ➤ কདྷతʹαʔϏεϝογϡͱͯ͠׆༻Ͱ͖Δ ➤ ͲͪΒʹ͠Ζenvoyͷ؅ཧ͸୭͔͕΍Δඞཁ͸͋Δ ➤ envoyͷretryPolicyɺkubernetesͷPod LifecycleΛ׆༻ͯ͠ϦΫ ΤετΛམͱͣ͞ʹσϓϩΠͰ͖Δ
33. App Meshͷߏ੒؅ཧ ➤ AWS App Mesh Controller for k8s͕༻ҙ͞Ε͍ͯΔ ➤ APIͷίϯςφͷYAMLΛ͍ ͡Δ͜ͱͳ͘envoyΛInject ͯ͘͠ΕΔ ➤ k8sͷCRDͱͯ͠؅ཧͰ͖ ΔͷͰgitopsͰ͖Δ ➤ ։ൃ؀ڥΛ૿΍͢ͷ΋؆୯
34. σϓϩΠ࣌ʹϦΫΤετ͕མͪΔ໰୊ ➤ App Mesh best practicesͱ ͍͏υΩϡϝϯτ͕༻ҙ͞ Ε͍ͯͯͦͷ௨Γʹ RetryPolicyΛઃఆ
35. σϓϩΠ࣌ʹϦΫΤετ͕མͪΔ໰୊ ➤ PodͷpreStopͰsleepΛೖΕΔ͜ͱͰ Service͔Β੾Γ཭͞ΕΔͷΛ଴ͭ ➤ AWSαϙʔτͷํʹΞυόΠεΛ΋ Β͍ղܾ͠·ͨ͠ɻେײँ ➤ AWSαʔϏεͷ͜ͱ͔͠ฉ͍ͪΌ͍ ͚ͳ͍ͱࢥ͚ͬͯͨͲͦΜͳ͜ͱ͸ͳ ͔ͬͨ ➤ Kubernetes΍Envoyͷ͜ͱ΋αϙʔτ ͯ͘͠Εͨ ➤ Argo CDʹΑΔSync(Deploymentͷ Rolling Update)Ͱγϯϓϧʹσϓϩ ΠͰ͖ΔΑ͏ʹͳͬͨ
36. ༨ஊ: ਓྨ͕଴๬͍ͯͨ͠ALBͷgRPCαϙʔτʂʂ ΋ͬͱૣ͘ཉ͔ͬͨ͠
37. ୈೋ෦ EKSҠߦޙͷ ߏ੒ʹ͍ͭͯ
38. EKSҠߦޙͷߏ੒ ECS EKS
39. EKSҠߦޙͷߏ੒ ALB Ingress Controllerͱ Nginx Ingrss Controllerͷ ૊Έ߹Θͤ App MeshʹΑΔ EnvoyͷInject Cloud MapͰ Service Discovery
40. SPOT OCEANΛ࢖ͬͨ Ϋϥελʔ؅ཧ
41. ৄࡉ͸ͪ͜Βͷϒϩά https://tech.recruit-mp.co.jp/infrastructure/post-19364/
42. Spot Ocean ੲ͸Spotinstͱ͍͏αʔϏεͩͬͨ ࠓ͸NetAppʹങऩ͞ΕͯSpotͱ͍͏໊લʹ
43. Pod-Driven Scaling ➤ Pod͕UnscheduleʹͳΒͳ͍Α͏ʹ NodeΛΦʔτεέʔϦϯάͯ͘͠ ΕΔ ➤ PodʹׂΓ౰ͯΔrequests͚ͩΛߟ ͑Ε͹OK ➤ Node͸ҙࣝ͠ͳͯ͘Α͍ͷͰ FargateͷΑ͏ͳ࢖͍উख ➤ Spot Instance͔Β࠷దͳΠϯελϯ εΛબΜͰ͘ΕΔ ➤ ΦϯσϚϯυͰಈ͍ͯཉ͍͠podʹ ͸annotation෇༩͢Δ͚ͩͰOK
44. Terraform΍Custom ControllerͰߏ੒؅ཧͰ͖Δ ➤ Terraformͷprovider͕༻ҙ ͞Ε͍ͯΔ ➤ Spot OceanͷCustom Controller͕Helm ChartͰ ༻ҙ͞Ε͍ͯΔͷͰ؆୯Π ϯετʔϧ
45. Headroom ➤ Headroom͸༨৒ͷϊʔυ Λ֬อ͓͍ͯͯ͘͠ΕΔػ ೳ ➤ ϦΫΤετͷεύΠΫ౳Ͱ PodͷεέʔϧΛૉૣ͘͢Δ
46. Cluster Roll ➤ Cluster Roll͸Nodeͷ҆શ ͳRolling UpdateΛͯ͘͠Ε Δ ➤ NodeͷೖΕସ͍͑ͨ͠ࣄ৘ ͕ൃੜͨ͠৔߹ͳͲʹศར
47. JENKINSͷJOBΛ CRONJOBʹҠߦ
48. ৄࡉ͸ͪ͜Β https://tech.recruit-mp.co.jp/infrastructure/post-20631/ https://speakerdeck.com/yutachaos/the-story-of-moving-jenkins-job-to-cronjob
49. όον͸JenkinsͰ΍͍ͬͯͨ ➤ αʔόʔΞϓϦέʔγϣϯ ͱಉ͡ίϯςφImageΛ ࢖ͬͯλεΫΛىಈ ➤ JenkinsͷఆظτϦΨʔͰ࣮ ߦ ➤ ௨஌΋ϦτϥΠ΋ศརɺͳ ΜͰ΋Ͱ͖Δɻ͔͠͠ɾɾ
50. Jenkinsศར͗͢໰୊ ➤ ศརա͗ͯWebը໘্Ͱ ͡ΌΜ͡ΌΜδϣϒ͕࡞Β Ε͍ͯ͘ ➤ ҰମԿ͕͍ͭىಈͯ͠Δ͔ Α͘෼͔Βͳ͍ ➤ ාͯ͘updateͰ͖ͣʹԘ௮ ͚ʹɾɾ
51. k8sͷCronJobʹ͢Δ͜ͱͰYAML؅ཧ ➤ CronJob͸ఆظతʹJobΛ࣮ߦ ͢Δk8sͷϦιʔε ➤ EKSҠߦͷӡ༻͸·ͣδϣϒҠ ߦ͔Βߦ͕ͬͨ͏·͍ͬͨ͘ ➤ YAMLͰ؅ཧͰ͖ΔͷͰgitops Ͱ͖Δ ➤ kubectlͰݟͯ΋͍͍͠ɺgitϨ ϙδτϦΛgrep͢Δ͚ͩͰԿ͕ ͍ͭͲ͜ͰJob͕ಈ͍ͯΔ͔͢ ͙ʹ෼͔Δ
52. όον࣮ߦͷͨͼʹNodeͷ૿ݮͯ͠͠·͏ ➤ ఆظతʹόον࣮ߦͷͨͼ ʹPodͷRequestsʹԠͯ͡ Node͕૿ݮ ➤ όον࣮ߦதʹNode͕མͪ ΔͱࠔΔ ➤ όονͷ࣮ߦʹΑͬͯαʔ ϏεʹӨڹ͸༩͑ͨ͘ͳ͍
53. EKS FargateͰCronJobΛ࣮ߦ͢Δ ➤ όον͸ίϯςφ͕ىಈ͠ ͯऴྃ͢Ε͹Α͍ͷͰ FargateʹͽͬͨΓ ➤ kubernetes͔Β͸Fargate͸ 1ͭͷnodeͷΑ͏ʹݟ͑Δ ➤ FargateͰىಈͨ͠όον͕ طଘͷΫϥελʔͷϦιʔ εΛ࢖͏͜ͱ͸ͳ͍
54. Sidecarίϯςφͷఀࢭ͕೉͍͠ ➤ Datadog Agent΍FluentdͳͲΛ Sidecarʹ͓͔͘͠ͳ͍ ➤ JobͷϝΠϯͷίϯςφ͕ఀࢭͯ͠΋ sidecarίϯςφ͸ఀࢭ͠ͳ͍ͷͰJob ͕ऴྃ͠ͳ͍ʂʂ ➤ ऴྃ࣌ʹsidecarίϯςφΛఀࢭͤ͞ ΔͳͲͷ޻෉͕ඞཁ ➤ ECSͰ͍͏essential:trueͷػೳ͕ͳ͍ ➤ Sidecar Containers͕Kubernetes v1.19ʹೖΔ༧ఆ͕ͩͬͨະఆʹͳͬ ͯ͠·ͬͨʁ
55. ·ͱΊͱࡶஊ ➤ ECS͔ΒEKSϔҠߦͨ͠࿩Λ͠·ͨ͠ ➤ ͋͘·Ͱ΋զʑͷ৔߹ͷ࿩ ➤ gitops΍KubernetesͷΤίγεςϜͰಠࣗ࢓༷Λͳͯ࣋͘͠ଓੑͷ͋ΔγεςϜ΁ ➤ App Mesh͸ຊ൪ར༻΋໰୊ͳ͠ ➤ ECS͸ࠓͳΒFargate Spot΋͋Δ͠ALB͕gRPCαϙʔτͨ͠͠࠶ߟͯ͠Έ͍ͨ ➤ ecspressoΛ࢖͑͹ϥΠϑαΠΫϧͱςϯϓϨʔτͱ͔΋ͬͱ͏·͘Ͱ͖͔ͨ΋ ➤ ECSͰɺALBͱͷͻ΋͚ͮɺECS ServiceɺTask Definitionͱ͔Λ౷Ұతʹѻ͑Δπʔϧ ͕͋Ε͹͍͍ͷ͔΋ ➤ copilot͕ͦ͏ͳͷ͔ʁ ➤ ίϯςφ͸͍͍ͧ ➤ ͨͩ͠ɺRDSɺClodwatch LogsɺS3ͳͲϚωʔδυαʔϏε͕͋ͬͯͦ͜