ECSからEKSへの移行への移行事例の紹介

by Masato Oshima

Slide 1

Slide 1 text

&$4͔Β&,4΁ ͷҠߦࣄྫͷ঺հ !@NQPO +"846(ίϯςφࢧ෦

Slide 2

Slide 2 text

Masato Oshima github.com/mpon @_mpon Software Engineer

Slide 3

Slide 3 text

ίϯςφࢧ෦ͷൃද͸3ճ໨Ͱ͢ 2017/12 2018/12

Slide 4

Slide 4 text

ίϝϯτ΍࣭໰ੋඇ͓ئ͍͠·͢ ➤ ਖ਼௚ɺ΋ͬͱ͍͍ํ๏͋Δ͔΋ͱ͍͏ෆ҆ɾɾ ➤ ͳͷͰποίϛ΋Β͑Δͱخ͍͠Ͱ͢ʂ ➤ ͦΜͳ͜ͱ͠ͳͯ͘΋ࠓͳΒ͜ΕͰͰ͖Δͷʹɻͱ͔ ➤ ͬͪ͜ͷπʔϧ࢖͑͹Α͔ͬͨͷʹɻͱ͔ ➤ ͳΜͰ͜͏͍͏ߏ੒ʹͨ͠ͷʁͱ͔ ➤ ͜͏͍͏έʔεͬͯରԠͰ͖ͯ·͔͢ʁͱ͔ ➤ ͕͜͜Α͘෼͔Βͳ͔ͬͨɻͱ͔

Slide 5

Slide 5 text

AGENDA ➤ ελσΟαϓϦENGLISHʹ͍ͭͯ ➤ ୈҰ෦: ECS͔ΒEKS΁Ҡߦͷഎܠ ➤ ՝୊1: ؀ڥ૿΍͢ͷ͕ਏ͔ͬͨ ➤ ՝୊2: Ͳ͜ʹԿ͕σϓϩΠ͞ΕͯΔ͔೺Ѳ͠ʹ͔ͬͨ͘ ➤ ՝୊3: gRPCͷෛՙ෼ࢄ ➤ ୈೋ෦: EKSҠߦޙͷߏ੒ʹ͍ͭͯ ➤ Spot OceanΛ࢖ͬͨΫϥελʔ؅ཧ ➤ JenkinsͷJobΛCronJobʹҠߦʢFargateͷ࿩΋͋ΔΑʣ ➤ ·ͱΊɿίϯςφ͸͍͍ͧ

Slide 6

Slide 6 text

ελσΟαϓϦENGLISH ➤ ΦϯϥΠϯӳޠֶशαʔϏε ➤ ӳձ࿩Ի੠΍ը૾ɾಈըͳͲΛ഑৴ ➤ ΫΠζܗࣜͰֶशσʔλɺཤྺΛอଘ ➤ iOS, Android, WebͰར༻Մೳ

Slide 7

Slide 7 text

αʔϏεߏ੒ͷ֓ཁ

Slide 8

Slide 8 text

αʔϏεߏ੒ͷ֓ཁ ໿50αʔϏε ɾɾɾ ɾ ɾ ɾ dev1, dev2... ໿20؀ڥ

Slide 9

Slide 9 text

ୈҰ෦ ECS͔ΒEKS΁Ҡߦ͠ ͨഎܠʹ͍ͭͯ

Slide 10

Slide 10 text

՝୊1 ؀ڥ૿΍͢ͷ͕ਏ ͔ͬͨ

Slide 11

Slide 11 text

ECSͷͱ͖ͷߏ੒؅ཧํ๏ ෼ྨ ίϯϙʔωϯτ ઃఆํ๏ Ծ૝Ϋϥελʔ ECS Cluster Terraform Πϯελϯε Autoscaling Group Terraform αʔϏεεέδϡʔϥ ECS Service Terraform αʔϏεσΟεΧόϦ Cloud Map Terraform ίϯςφఆٛ ECS Task Deﬁnition Terraform + ಠࣗπʔϧ ϩʔυόϥϯαʔ ALB Terraform αʔϏεͻ΋͚ͮ ALB Target Group Terraform ϧʔςΟϯά ALB Listenr Rule Terraform

Slide 12

Slide 12 text

ECSͰ؀ڥΛͨ͘͞Μ૿΍͢ʹ͸ʁ ➤ TerraformͰECS ClusterɺECS ServiceɺALBͳͲ৭ʑ࡞Δ ➤ Task Deﬁnitionʢಠࣗπʔϧͷ࢓༷ͷYAMLʣΛେྔʹίϐʔ ➤ ςϯϓϨʔτͷػೳ͕ͳ͘؀ڥ͝ͱʹ؀ڥݻ༗ͷ஋ʹॻ͖׵͑Δ ➤ ؀ڥ૿΍͢ͷʹTerraformͰΠϯελϯε΍ALBΛ૿΍͞ͳ͍ͱ͍͚ͳ͍ ➤ Terraform͸ModuleԽ͸͚ͨ͠Ͳɺ͋ͱ͔ΒมߋೖΕΔͱޓ׵ੑอͭͷ ʹେมͳ͜ͱ͕͋ͬͯؾܰʹmoduleΛมߋͰ͖ͳ͍ ➤ σϓϩΠͰมߋ͕ى͖Δ΋ͷΛTerraformͰ؅ཧͯ͠͠·͍ͬͯͨͷͰ ignore_changesͰແཧ΍Γແࢹ ➤ ϥΠϑαΠΫϧͷҧ͍ΛݟۃΊΒΕ͍ͯͳ͔ͬͨɾɾ

Slide 13

Slide 13 text

EKSͷߏ੒؅ཧํ๏ ෼ྨ ίϯϙʔωϯτ ઃఆํ๏ Ծ૝Ϋϥελʔ Namespace YAML(k8s) Πϯελϯε EC2Πϯελϯε Terraform + Spot Ocean αʔϏεεέδϡʔϥ Deployment YAML(k8s) αʔϏεσΟεΧόϦ Service YAML(k8s) ίϯςφఆٛ Pod YAML(k8s) ϩʔυόϥϯαʔ Ingress YAML(k8s) αʔϏεͻ΋͚ͮ Ingress YAML(k8s) ϧʔςΟϯά Ingress YAML(k8s)

Slide 14

Slide 14 text

EKSͰ؀ڥΛͨ͘͞Μ૿΍͢ʹ͸ʁ ➤ Terraformͷग़൪͸EKS ClusterΛ࠷ॳʹ࡞Δ͚ͩͰ؀ڥ૿΍͢ͱ ͖ʹ͸͍Βͳ͍ ➤ ؀ڥ૿΍͢৔߹͸NamespaceΛ૿΍͢ ➤ KustomizeΛ࢖ͬͯ؀ڥ͝ͱʹมΘΔ෦෼͚ͩΛ௥Ճ ➤ ֎෦͔ΒͷϦΫΤετͷϚοϐϯά͸Ingress ➤ ಺෦௨৴͸ServiceΦϒδΣΫτͰ؆୯αʔϏεσΟεΧόϦɺ CoreDNS͕͋ΔͷͰRoute53ͳ͠Ͱ΋಺෦ͰDNS͕༻ҙ ➤ ΄ͱΜͲKubernetesͷΦϒδΣΫτͰ׬݁͢ΔͷͰએݴతʹ؅ཧ ͠΍͍͢

Slide 15

Slide 15 text

ΫϥελʔΛ࿦ཧతʹ෼ׂͰ͖Δ Namespace dev1 EC2 EC2 EC2 Cluster EC2 EC2 EC2 Namespace dev2 Namespace dev3 ➤ Kubernetesʹ͸Namespace ͱ͍͏ΦϒδΣΫτ͕͋Δ ➤ Cluster͑͞࡞ͬͯ͠·͑͹ ͋ͱ͸NamespaceΛ૿΍͢ ͚ͩͰ؀ڥ͕૿΍ͤΔ ➤ Namespace͝ͱͷDNSΤϯ τϦ΋࡞੒͞ΕΔ ➤ ͜Μͳ୯७ͳYAMLͰOK

Slide 16

Slide 16 text

Kustomize, HelmͳͲσϑΝΫτͳߏ੒؅ཧ͕͋Δ

Slide 17

Slide 17 text

Kustomizeͷoverlays

Slide 18

Slide 18 text

՝୊2 Ͳ͜ʹԿ͕σϓϩΠ͞Ε ͯΔ͔೺Ѳ͠ʹ͔ͬͨ͘

Slide 19

Slide 19 text

ECSͷͱ͖ʹײ͍ͯͨ͡՝୊ ➤ hoge؀ڥͷfugaαʔϏεͬͯࠓσϓϩΠ͞ΕͯΔͷ͸Ͳͷϒϥϯνʁ ➤ ίϯςφͷ؀ڥม਺͸Կ͕ઃఆ͞ΕͯΔʁ ➤ aws ecs list-servicesͱ͔ͰҰൃͰͲͷtag͕σϓϩΠ͞ΕͯΔ͔೺ѲͰ͖ ͳ͍ͷͰࣗ࡞πʔϧ࡞͙ͬͯ྇ ➤ σϓϩΠͷઃఆʢJenkinsﬁleʣʹઃఆ͕ͲΜͲΜ૿͍͑ͯͬͯΧΦεʹ ͳ͖ͬͯͨ ➤ Α͘σϓϩΠ͢ΔAPIͳͲͷαʔόʔΞϓϦέʔγϣϯͱ୯ͳΔnginxͱ ͔ͷσϓϩΠ͸σϓϩΠํ๏͕ผʑʹ༻ҙͯ͠ϝϯς͍ͯͨ͠ ➤ gitϦϙδτϦΛݟΕ͹Կ͕σϓϩΠ͞ΕͯΔ͔෼͔ΔΑ͏ʹ͍ͨ͠ ➤ gitops͍ͨ͠

Slide 20

Slide 20 text

GITOPS͍ͨ͠

Slide 21

Slide 21 text

gitopsπʔϧͱͯ͠Argo CDͷಋೖ https://tech.recruit-mp.co.jp/infrastructure/gitops-cd-by-using-argo-cd-at-eks/

Slide 22

Slide 22 text

Argo CDͷArchitecture - k8sͷReconciliation Loop gitϦϙδτϦΛ ݟΕ͹Կ͕σϓϩΠ͞Ε ͍ͯΔ͔೺ѲͰ͖Δ gitϦϙδτϦͱ ࠩ෼Λఆظతʹ νΣοΫͯ͠ ࠩ෼͕͋Ε͹ σϓϩΠ

Slide 23

Slide 23 text

Dashboard͕͋ΔͷͰDeveloper Friendly GUI͕͋Δͱ։ൃऀʹ΋ ֓೦Λཧղͯ͠΋Β͍΍͍͢

Slide 24

Slide 24 text

Sync Phase and WavesͰσϓϩΠͷґଘؔ܎ΛදݱͰ͖Δ

Slide 25

Slide 25 text

Clusterʹඞཁͳresource΋gitopsͰ ➤ Ingress Controller΍ Datadog AgentͳͲ΋؅ཧ Ͱ͖Δ ➤ Helm Chart Repositoriesͷ ػೳ͕͋ΔͷͰArgo CDͰ Helm ChartΛιʔεʹ؅ཧ Մೳ ➤ ArgoCD ApplicationΛ࡞੒ ͢Ε͹ߏங׬ྃ

Slide 26

Slide 26 text

ެࣜπʔϧͷkubectlͰ͙͢ʹ೺ѲͰ͖Δ ➤ ྫ͑͹ͲͷImage͕σϓϩΠ͞ΕͯΔ͔֬ೝ͢Δͷʹ͜Ε͚ͩͰOK ➤ `kubectl get deploy -o wide`

Slide 27

Slide 27 text

՝୊3 GRPCͷෛՙ෼ࢄ

Slide 28

Slide 28 text

౰࣌ͷgRPCͷෛՙ෼ࢄ https://tech.recruit-mp.co.jp/infrastructure/post-17098/

Slide 29

Slide 29 text

ECSͷͱ͖ͷߏ੒

Slide 30

Slide 30 text

σϓϩΠ࣌ʹϦΫΤετ͕མͪΔ໰୊ Blue gRPC Green gRPC Blue gRPC Green gRPC blue.grpc.internal green.grpc.internal Cloud Map lb.grpc.internal ➤ grpcαʔόʔΛRolling Update͢Δ ͱSTOPࡁΈͷίϯςφͷIP͕ฦͬ ͖ͯͯϦΫΤετ͕མͪͯ͠·͏ ➤ JenkinsͷpipelineͱshΛ૊Έ߹Θ ͤͯBlue/GreenσϓϩΠ ➤ ECS Task Deﬁnitionͷ؀ڥม਺Λ sedͰBlue->Greenʹॻ͖׵͑ͯ envoyΛRolloing Update ➤ Sleep 30͔ͯ͠ΒBlueΛ࡟আ ➤ ϫʔΫΞϥ΢ϯυײຬࡌͳͷͰͳ Μͱ͔͔ͨͬͨ͠

Slide 31

Slide 31 text

App Meshͷ࠾༻ https://tech.recruit-mp.co.jp/infrastructure/post-20765/

Slide 32

Slide 32 text

App Meshͷ࠾༻ ➤ gRPC routingΛαϙʔτ͍ͯͨ͠ ➤ কདྷతʹαʔϏεϝογϡͱͯ͠׆༻Ͱ͖Δ ➤ ͲͪΒʹ͠Ζenvoyͷ؅ཧ͸୭͔͕΍Δඞཁ͸͋Δ ➤ envoyͷretryPolicyɺkubernetesͷPod LifecycleΛ׆༻ͯ͠ϦΫ ΤετΛམͱͣ͞ʹσϓϩΠͰ͖Δ

Slide 33

Slide 33 text

App Meshͷߏ੒؅ཧ ➤ AWS App Mesh Controller for k8s͕༻ҙ͞Ε͍ͯΔ ➤ APIͷίϯςφͷYAMLΛ͍ ͡Δ͜ͱͳ͘envoyΛInject ͯ͘͠ΕΔ ➤ k8sͷCRDͱͯ͠؅ཧͰ͖ ΔͷͰgitopsͰ͖Δ ➤ ։ൃ؀ڥΛ૿΍͢ͷ΋؆୯

Slide 34

Slide 34 text

σϓϩΠ࣌ʹϦΫΤετ͕མͪΔ໰୊ ➤ App Mesh best practicesͱ ͍͏υΩϡϝϯτ͕༻ҙ͞ Ε͍ͯͯͦͷ௨Γʹ RetryPolicyΛઃఆ

Slide 35

Slide 35 text

σϓϩΠ࣌ʹϦΫΤετ͕མͪΔ໰୊ ➤ PodͷpreStopͰsleepΛೖΕΔ͜ͱͰ Service͔Β੾Γ཭͞ΕΔͷΛ଴ͭ ➤ AWSαϙʔτͷํʹΞυόΠεΛ΋ Β͍ղܾ͠·ͨ͠ɻେײँ ➤ AWSαʔϏεͷ͜ͱ͔͠ฉ͍ͪΌ͍ ͚ͳ͍ͱࢥ͚ͬͯͨͲͦΜͳ͜ͱ͸ͳ ͔ͬͨ ➤ Kubernetes΍Envoyͷ͜ͱ΋αϙʔτ ͯ͘͠Εͨ ➤ Argo CDʹΑΔSync(Deploymentͷ Rolling Update)Ͱγϯϓϧʹσϓϩ ΠͰ͖ΔΑ͏ʹͳͬͨ

Slide 36

Slide 36 text

༨ஊ: ਓྨ͕଴๬͍ͯͨ͠ALBͷgRPCαϙʔτʂʂ ΋ͬͱૣ͘ཉ͔ͬͨ͠

Slide 37

Slide 37 text

ୈೋ෦ EKSҠߦޙͷ ߏ੒ʹ͍ͭͯ

Slide 38

Slide 38 text

EKSҠߦޙͷߏ੒ ECS EKS

Slide 39

Slide 39 text

EKSҠߦޙͷߏ੒ ALB Ingress Controllerͱ Nginx Ingrss Controllerͷ ૊Έ߹Θͤ App MeshʹΑΔ EnvoyͷInject Cloud MapͰ Service Discovery

Slide 40

Slide 40 text

SPOT OCEANΛ࢖ͬͨ Ϋϥελʔ؅ཧ

Slide 41

Slide 41 text

ৄࡉ͸ͪ͜Βͷϒϩά https://tech.recruit-mp.co.jp/infrastructure/post-19364/

Slide 42

Slide 42 text

Spot Ocean ੲ͸Spotinstͱ͍͏αʔϏεͩͬͨ ࠓ͸NetAppʹങऩ͞ΕͯSpotͱ͍͏໊લʹ

Slide 43

Slide 43 text

Pod-Driven Scaling ➤ Pod͕UnscheduleʹͳΒͳ͍Α͏ʹ NodeΛΦʔτεέʔϦϯάͯ͘͠ ΕΔ ➤ PodʹׂΓ౰ͯΔrequests͚ͩΛߟ ͑Ε͹OK ➤ Node͸ҙࣝ͠ͳͯ͘Α͍ͷͰ FargateͷΑ͏ͳ࢖͍উख ➤ Spot Instance͔Β࠷దͳΠϯελϯ εΛબΜͰ͘ΕΔ ➤ ΦϯσϚϯυͰಈ͍ͯཉ͍͠podʹ ͸annotation෇༩͢Δ͚ͩͰOK

Slide 44

Slide 44 text

Terraform΍Custom ControllerͰߏ੒؅ཧͰ͖Δ ➤ Terraformͷprovider͕༻ҙ ͞Ε͍ͯΔ ➤ Spot OceanͷCustom Controller͕Helm ChartͰ ༻ҙ͞Ε͍ͯΔͷͰ؆୯Π ϯετʔϧ

Slide 45

Slide 45 text

Headroom ➤ Headroom͸༨৒ͷϊʔυ Λ֬อ͓͍ͯͯ͘͠ΕΔػ ೳ ➤ ϦΫΤετͷεύΠΫ౳Ͱ PodͷεέʔϧΛૉૣ͘͢Δ

Slide 46

Slide 46 text

Cluster Roll ➤ Cluster Roll͸Nodeͷ҆શ ͳRolling UpdateΛͯ͘͠Ε Δ ➤ NodeͷೖΕସ͍͑ͨ͠ࣄ৘ ͕ൃੜͨ͠৔߹ͳͲʹศར

Slide 47

Slide 47 text

JENKINSͷJOBΛ CRONJOBʹҠߦ

Slide 48

Slide 48 text

ৄࡉ͸ͪ͜Β https://tech.recruit-mp.co.jp/infrastructure/post-20631/ https://speakerdeck.com/yutachaos/the-story-of-moving-jenkins-job-to-cronjob

Slide 49

Slide 49 text

όον͸JenkinsͰ΍͍ͬͯͨ ➤ αʔόʔΞϓϦέʔγϣϯ ͱಉ͡ίϯςφImageΛ ࢖ͬͯλεΫΛىಈ ➤ JenkinsͷఆظτϦΨʔͰ࣮ ߦ ➤ ௨஌΋ϦτϥΠ΋ศརɺͳ ΜͰ΋Ͱ͖Δɻ͔͠͠ɾɾ

Slide 50

Slide 50 text

Jenkinsศར͗͢໰୊ ➤ ศརա͗ͯWebը໘্Ͱ ͡ΌΜ͡ΌΜδϣϒ͕࡞Β Ε͍ͯ͘ ➤ ҰମԿ͕͍ͭىಈͯ͠Δ͔ Α͘෼͔Βͳ͍ ➤ ාͯ͘updateͰ͖ͣʹԘ௮ ͚ʹɾɾ

Slide 51

Slide 51 text

k8sͷCronJobʹ͢Δ͜ͱͰYAML؅ཧ ➤ CronJob͸ఆظతʹJobΛ࣮ߦ ͢Δk8sͷϦιʔε ➤ EKSҠߦͷӡ༻͸·ͣδϣϒҠ ߦ͔Βߦ͕ͬͨ͏·͍ͬͨ͘ ➤ YAMLͰ؅ཧͰ͖ΔͷͰgitops Ͱ͖Δ ➤ kubectlͰݟͯ΋͍͍͠ɺgitϨ ϙδτϦΛgrep͢Δ͚ͩͰԿ͕ ͍ͭͲ͜ͰJob͕ಈ͍ͯΔ͔͢ ͙ʹ෼͔Δ

Slide 52

Slide 52 text

όον࣮ߦͷͨͼʹNodeͷ૿ݮͯ͠͠·͏ ➤ ఆظతʹόον࣮ߦͷͨͼ ʹPodͷRequestsʹԠͯ͡ Node͕૿ݮ ➤ όον࣮ߦதʹNode͕མͪ ΔͱࠔΔ ➤ όονͷ࣮ߦʹΑͬͯαʔ ϏεʹӨڹ͸༩͑ͨ͘ͳ͍

Slide 53

Slide 53 text

EKS FargateͰCronJobΛ࣮ߦ͢Δ ➤ όον͸ίϯςφ͕ىಈ͠ ͯऴྃ͢Ε͹Α͍ͷͰ FargateʹͽͬͨΓ ➤ kubernetes͔Β͸Fargate͸ 1ͭͷnodeͷΑ͏ʹݟ͑Δ ➤ FargateͰىಈͨ͠όον͕ طଘͷΫϥελʔͷϦιʔ εΛ࢖͏͜ͱ͸ͳ͍

Slide 54

Slide 54 text

Sidecarίϯςφͷఀࢭ͕೉͍͠ ➤ Datadog Agent΍FluentdͳͲΛ Sidecarʹ͓͔͘͠ͳ͍ ➤ JobͷϝΠϯͷίϯςφ͕ఀࢭͯ͠΋ sidecarίϯςφ͸ఀࢭ͠ͳ͍ͷͰJob ͕ऴྃ͠ͳ͍ʂʂ ➤ ऴྃ࣌ʹsidecarίϯςφΛఀࢭͤ͞ ΔͳͲͷ޻෉͕ඞཁ ➤ ECSͰ͍͏essential:trueͷػೳ͕ͳ͍ ➤ Sidecar Containers͕Kubernetes v1.19ʹೖΔ༧ఆ͕ͩͬͨະఆʹͳͬ ͯ͠·ͬͨʁ

Slide 55

Slide 55 text

·ͱΊͱࡶஊ ➤ ECS͔ΒEKSϔҠߦͨ͠࿩Λ͠·ͨ͠ ➤ ͋͘·Ͱ΋զʑͷ৔߹ͷ࿩ ➤ gitops΍KubernetesͷΤίγεςϜͰಠࣗ࢓༷Λͳͯ࣋͘͠ଓੑͷ͋ΔγεςϜ΁ ➤ App Mesh͸ຊ൪ར༻΋໰୊ͳ͠ ➤ ECS͸ࠓͳΒFargate Spot΋͋Δ͠ALB͕gRPCαϙʔτͨ͠͠࠶ߟͯ͠Έ͍ͨ ➤ ecspressoΛ࢖͑͹ϥΠϑαΠΫϧͱςϯϓϨʔτͱ͔΋ͬͱ͏·͘Ͱ͖͔ͨ΋ ➤ ECSͰɺALBͱͷͻ΋͚ͮɺECS ServiceɺTask Deﬁnitionͱ͔Λ౷Ұతʹѻ͑Δπʔϧ ͕͋Ε͹͍͍ͷ͔΋ ➤ copilot͕ͦ͏ͳͷ͔ʁ ➤ ίϯςφ͸͍͍ͧ ➤ ͨͩ͠ɺRDSɺClodwatch LogsɺS3ͳͲϚωʔδυαʔϏε͕͋ͬͯͦ͜