Next FRESH! Applications with Amazon ECS

by stormcat24

Slide 1

Slide 1 text

Next FRESH! Applications with Amazon ECS 2015/11/20 JAWS-UGίϯςφࢧ෦ #3 #jawsug_ct @stormcat24

Slide 2

Slide 2 text

Who? ‣ Akinori Yamada ( @stormcat24 ) ‣ http://stormcat.hatenablog.com/ ‣ CyberAgent, Inc ‣ Ameba౷ׅຊ෦ FRESHάϧʔϓ ‣ ServerSide / DevOps ‣ ޷͖ͳݴޠ͸Scala

Slide 3

Slide 3 text

Agenda ‣ AmebaFRESH! ‣ Architecture & ECS ‣ Blue Green Deployment ‣ Diet Docker Image ‣ Local Development ‣ ecs-formation ‣ Others

Slide 4

Slide 4 text

AmebaFRESH!

Slide 5

Slide 5 text

‣ ੜ์ૹಈը഑৴ϓϥοτϑΥʔϜ ‣ جຊແྉ ‣ PCϒϥ΢β / iOS / Android ‣ εϚϗφΠζυ͞ΕͨUI ‣ ߴ඼࣭ͳಈը഑৴ ‣ 2015೥12݄Ұൠެ։༧ఆ ‣ ݱࡏΫϩʔζυެ։த ‣ ※AbemaTVͱ͸ผͷαʔϏεͰ͢YO

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

‣ 2015/04͔ΒϓϩδΣΫτ։࢝ ‣ ϓϩδΣΫτϝϯόʔ໿30໊ ‣ Serverside ☓ 6 ‣ Frontend ☓ 6 ‣ iOS ☓ 4 ‣ Android ☓ 4 ‣ Designer ☓ 3

Slide 11

Slide 11 text

‣ Movie & Broadcasting ‣ RTMP Publishing ‣ HTTP Live Streaming (HLS)

Slide 12

Slide 12 text

‣ Frontend ‣ Node.js v4 ‣ TypeScript1.6 ‣ React / Flux ‣ SPA + SSR ‣ iOS ‣ Swift2.0 ‣ Android ‣ Kotlin

Slide 13

Slide 13 text

‣ Serverside & Infrastructure ‣ Amazon Web Services (AWS) ‣ Go1.5 ‣ Docker1.9.0 ‣ Microservices ‣ RESTful API (goji)

Slide 14

Slide 14 text

‣ Tools ‣ Slack + Hubot (ChatOps) ‣ Github Enterprise ‣ CircleCI (Enterprise) ‣ Crashlytics ‣ JIRA Agile ‣ Mackerel ‣ ࡳଋ

Slide 15

Slide 15 text

Architecture & ECS

Slide 16

Slide 16 text

Architecture ‣ ࢥ૝ ‣ جຊํ਑ ‣ Task Deﬁnitionͷߏ੒ ‣ ClusterͱServiceͷѻ͍

Slide 17

Slide 17 text

໨ࢦ͍ͯ͠Δ͜ͱ ‣ ۃྗϝϯςΛೖΕͳ͍ ‣ θϩμ΢ϯλΠϜϦϦʔε → Blue Green Deployment ‣ Πϯϑϥ͸࢖͍ࣺͯͯೖΕସ͑Δ → Immutable Infrastructure ‣ ૉૣ͘සൟͳϦϦʔε ‣ γεςϜ͸ݶΓͳ͘ૄ݁߹ͷํ͕ྑ͍ → Microservices ‣ ίϯςφͩ → Docker

Slide 18

Slide 18 text

Microservices ‣ ಛఆͷ։ൃݴޠʹґଘ͠ͳ͍ʢ๞͖Δ͠ʣ ‣ ௨৴ϓϩτίϧ ‣ RESTFul API (HTTP) ‣ ͦͷ͏ͪgRPC(HTTP2)͍ͨ͠ ‣ αʔϏεͷཻ౓ΛఆΊΔͷ͸΍͸Γ೉͍͠ ‣ γεςϜతͳυϝΠϯྖҬͰ෼͚Δͷ΋Ұͭͷࢦඪ ‣ ޙ͔ΒผαʔϏεͱͯ͠੾Γམͱ͢΋OK

Slide 19

Slide 19 text

Infrastructure ‣ AWS + EC2 Container Service(ECS)Λ࠾༻ ‣ ͪΐ͏Ͳٕज़ݕূதʹECSͷ౦ژϦʔδϣϯ͕དྷͨ ‣ ࠷௿ݶͷίϯςφߏ੒؅ཧͱεέδϡʔϦϯά͕Ͱ͖Ε͹े෼ ‣ ECSҎ֎ʹ΋࢖͍͍ͨ΋ͷ͕͋ͬͨ͠ ‣ Lambda ‣ RDS Aurora

Slide 20

Slide 20 text

جຊํ਑ ‣ MicroservicesຖͰίϯςφΛߏ੒͢ΔTask DeﬁnitionΛͭ͘Δ ‣ 1ECS Clusterʹ͖ͭ1Service ‣ 1Clusterʹ͸1ͭͷAutoScaling Group ‣ ಈը഑৴αʔό͸ಛघͳͷͰྫ֎ ‣ Clusterؒ௨৴͸Internal ELBΛར༻

Slide 21

Slide 21 text

Task Deﬁnitionͷํ਑ ‣ ϩάͷసૹʹtd-agentΛ࢖͏ ‣ ֤ίϯςφϩά͸ϗετʹϚ΢ϯτ ‣ ϗετʹϚ΢ϯτ͞ΕͨϩάΛtd-agentͰసૹ ‣ logging driver͸·ͩಋೖͯ͠ͳ͍ ‣ Internal Service(API)Ͱ͋ͬͯ΋جຊNginxΛ௨͢ ‣ ΞΫηεϩάग़͢ͷָͩ͠

Slide 22

Slide 22 text

Task Deﬁnition(Service API)

Slide 23

Slide 23 text

Task Deﬁnition(WEB+API)

Slide 24

Slide 24 text

Task Deﬁnition(Job)

Slide 25

Slide 25 text

Task Deﬁnition(movie)

Slide 26

Slide 26 text

جຊతʹ͜ΕΒͷ૊Έ߹Θͤ

Slide 27

Slide 27 text

cluster = serviceͷσϝϦοτ ‣ Ϧιʔεޮ཰తʹ͸ϕετͰ͸ͳ͍ ‣ Cluster : Service = 1:N ʹൺ΂Δͱ ‣ Πϯελϯε਺͸૿͑Δ܏޲ʹ͋Δ ‣ ։ൃ؀ڥͰ͸t2.microΛ༗ޮ׆༻͍ͯ͠Δ ‣ nanoΠϯελϯεщ ƅшƅщ)ŜŹŖƃ

Slide 28

Slide 28 text

cluster = service ʹͯ͠Δཧ༝ ‣ ࢹ֮తʹΘ͔Γ΍͍͢ʢϏΪφʔϑϨϯυϦʔʣ ‣ Service୯ҐͰIAM roleΛઃఆͰ͖ͳ͍ ‣ IAM RoleͰݫີͳݖݶ੍ޚΛ͢Δʹ͸଍Γͳ͍ ‣ Service୯ҐͰͷSecurity GroupʹະରԠ ‣ ͨͩ͠ɺELBલఏʹͯ͠͠·͑͹ղܾͰ͖Δ

Slide 29

Slide 29 text

Blue Green Deployment

Slide 30

Slide 30 text

2AutoScalingύλʔϯ ‣ BlueɺGreenܥ౷ͷClusterΛ࡞Δ ‣ ͦΕͧΕ͕AutoScalingGroupʹଐ͢Δ ‣ api1-blue, api1-green Έ͍ͨͳ ‣ AutoScalingGroup୯ҐͰELBΛ੾Γସ͑Δ

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

2AutoScalingͷಛ௃ ‣ ͱͯ΋҆શ ‣ DeployޙͷϩʔϧόοΫ༰қ ‣ DeployલͷStandbyܥ౷ͷ΢ΥʔϜΞοϓඞཁ ‣ 10෼લʹ͸΍͓͖͍ͬͯͨ ‣ ίετͷέΞ͕ॏཁ ‣ DeployޙɺStandbyʹͳͬͨܥ౷Λམͱ͢౳

Slide 33

Slide 33 text

Diet Docker Image

Slide 34

Slide 34 text

Πϝʔδ͸খ͍͞΄Ͳྑ͍ ‣ docker build࣌ؒˣ ‣ CI࣌ؒˣ ‣ Registry͔ΒͷΠϝʔδμ΢ϯϩʔυ࣌ؒˣ ‣ AutoScaleͰ࡞੒͞ΕͨΠϯελϯε͕αʔϏεΠϯ͢Δ ·Ͱͷ࣌ؒ↓

Slide 35

Slide 35 text

docker hub ‣ hub.docker.com ‣ ਺ଟ͘ͷެࣜΠϝʔδ ‣ αΠζΛ࡟͗མͱͨ͠΋ͷ͹͔ΓͰ͸ͳ͍ ‣ 1GB௒͑ΔΠϝʔδ͸σϒ

Slide 36

Slide 36 text

ෆཁͳ΋ͷ͸࡟আ͢Δ ‣ ෆཁͳϑΝΠϧΛݟམͱ͞ͳ͍ ‣ ϏϧυͷͨΊʹੜͨ͡࢈ۀഇغ෺Λ࡟আ ‣ npm cache clear ‣ rm -rf ~/.grade ‣ apt-get clean ‣ Data VolumeΛ࢖͏ʢϙʔλϏϦςΟ͸མͪΔʣ

Slide 37

Slide 37 text

RUNͷճ਺ΛݮΒ͢ ‣ RUNͷ਺͚ͩΠϝʔδͷϨΠϠʔ͕ॏͳΓɺΠ ϝʔδ༰ྔ͸૿͑Δ ‣ && ͰνΣΠϯͯ͠ɺRUNͷճ਺ΛݮΒ͢ ‣ ௕͍docker buildͷ৔߹ɺ్தͰࣦഊ͢ΔͱRUN ͷ಄͔Β΍Γͳ͓͠ͳͷͰফ໣͸͢Δ

Slide 38

Slide 38 text

RUNͷճ਺ΛݮΒ͢ FROM ubuntu:15.10 RUN apt-get update RUN apt-get install -y curl RUN apt-get apt-get clean FROM ubuntu:15.10 RUN apt-get update && \ apt-get install -y curl && \ apt-get apt-get clean

Slide 39

Slide 39 text

ܰྔΠϝʔδΛ࢖͏ ‣ ࡟͗མͱ͞ΕͨܰྔΠϝʔδΛ࢖͏ ‣ ࠷ۙ͸ΦϑΟγϟϧͰslimΠϝʔδ͕͋Δ΋ͷ΋͋Δ ‣ e.g. Node, Go ‣ busybox௒ઈ͍ܰ

Slide 40

Slide 40 text

‣ ࠷ۙ͸ܰྔΠϝʔδ(slim)΋༻ҙ͞ ΕͯΔ ‣ Docker HubΛීஈ͔Β८ճ͓ͯ͠ ͜͏

Slide 41

Slide 41 text

‣ ٻΊΒΕΔϙʔλϏϦςΟˢ ‣ GoͰ͋Ε͹࣮ߦϑΝΠϧΛࡌͤΔ ͚ͩ ‣ busyboxͷதͰϏϧυ͸ΩπΠ ʢ೉қ౓ߴʣ

Slide 42

Slide 42 text

ݮྔʹΑΔࢥΘ͵ฐ֐ ‣ x509: failed to load system roots and no roots provided ‣ ίϯςφ͔ΒHTTPS௨৴͕Ͱ͖ͳ͘ͳΔ ‣ apt-get install -y ca-certiﬁcates Ͱղܾ ‣ ֎෦πʔϧ΁ͷґଘʢΞϓϦ಺͔ΒͷΩοΫʣ

Slide 43

Slide 43 text

ϕʔεΠϝʔδΛͭ͘Δ ‣ apt updateɺapt-get install ͏Μ͵ΜΛऴΘΒͤͨ ΋ͷ ‣ ຖ౓΍ͬͯͨΒCIͷ͕࣌ؒ૿͑Δʢdocker buildͷ ҆ఆʣ ‣ ΞϓϦଆͷDockerﬁleͰ͸ຊ࣭ͷॲཧʹઐ೦ͤ͞Δ

Slide 44

Slide 44 text

Local Development

Slide 45

Slide 45 text

ϩʔΧϧ։ൃͰ࢖͏΋ͷ ‣ docker-machine + VirtualBox ‣ docker-compose

Slide 46

Slide 46 text

docker-machine ‣ VirtualBox, AWS, Azure, Digital Ocean্ʹ DockerϗετΛߏங͢Δ ‣ ͔͋ͨ΋ϩʔΧϧ্ʹDocker؀ڥ͕͋Δ͔ͷΑ͏ ʹDockerͷૢ࡞͕Ͱ͖Δ

Slide 47

Slide 47 text

docker-machine ‣ docker-machine + VirtualBoxΛબ୒ ‣ Vagrant͸ࣺͯͨ ‣ Dockerʹൺ΂Δͱ࢖͍ࣺͯίετˢ ‣ ΞϓϦέʔγϣϯɺϛυϧ΢ΣΞؚΊ͍ͯͭͰ΋ϩʔΧϧͰ֬ ೝͰ͖ΔΑ͏ʹ ‣ VirtualBoxͷϙʔτϑΥϫʔυར༻

Slide 48

Slide 48 text

ϚγϯϦιʔεඞཁ ‣ ϩʔΧϧ΋ϑϧDockerͩͱɺٻΊΒΕΔϚγϯεϖο Ϋ͸ߴ͘ͳΔ ‣ 16GBͳ͍ͱ݁ߏਏ͍ ‣ αʔόαΠυΤϯδχΞʹ͸ඞਢ ‣ ίϯςφ͍ͬͺ্͍͛ͯɺշదʹTwitter͢Δͷॏཁ

Slide 49

Slide 49 text

׳Ε͸ා͍

Slide 50

Slide 50 text

docker-compose ‣ Docker ToolboxͷҰ෦ʢݩʑﬁgʣ ‣ Dockerίϯςφ܈ͷߏ੒ΛYAMLͰ؅ཧ ‣ docker-compose up -d Ͱىಈ

Slide 51

Slide 51 text

σʔλετΞ΋Docker ‣ ϩʔΧϧͰ͸σʔλετΞ΋Dockerίϯςφར༻ ‣ library/mysql, library/redis ‣ VagrantΑΓ΋ߴ଎ͰؾܰʹεΫϥοϓϏϧυ

Slide 52

Slide 52 text

DBϚΠάϨʔγϣϯॏཁ ‣ ؀ڥ͸ἧͬͯ΋σʔλෆඋ͋Δͱҙຯ͕ແ͍ ‣ FRESH! Ͱ͸ goose ‣ https://bitbucket.org/liamstask/goose/ ‣ SQL͚ͩ͡Όͳ͘ɺGoͰϚΠάϨʔγϣϯ΋ॻ͚Δ

Slide 53

Slide 53 text

ecs-formation

Slide 54

Slide 54 text

ecs-formation ‣ https://github.com/stormcat24/ecs-formation ‣ docker-composeͷΑ͏ʹɺYAMLϑΝΠϧͰίϯςφ ͷߏ੒Λ؅ཧ͢Δ ‣ ౰࣌͸ecs-cliͱ͔ແ͔ͬͨͷͰ࡞ͬͨ ‣ aws-sdk-goར༻

Slide 55

Slide 55 text

ecs-formation features ‣ Task Deﬁnitionsͷߋ৽ ‣ Clusterʹ഑ஔ͢ΔServiceͷߋ৽ ‣ Blue-Green Deploymentͷ࣮ߦ

Slide 56

Slide 56 text

Task Deﬁnitions (task/jawsug-api.yml) nginx: image: registry.jawsug.local:5000/jawsug/nginx:latest ports: - 80:80 environment: SERVER_NAME: jawsug.example.com volumes: - /var/log/container/nginx:/var/log/nginx links: - api memory: 512 cpu_units: 512 essential: true

Slide 57

Slide 57 text

Services on cluster (service/jawsug-cluster.yml) api-service: task_deﬁnition: jawsug-api desired_count: 1

Slide 58

Slide 58 text

Blue Green Deployment (bluegreen/jawsug-cluster.yml) blue: cluster: jawsug-cluster-blue service: api-service autoscaling_group: jawsug-cluster-blue green: cluster: jawsug-cluster-green service: api-service autoscaling_group: jawsug-cluster-green primary_elb: jawsug-api-primary standby_elb: jawsug-api-standby

Slide 59

Slide 59 text

ecs-formationͷӡ༻ ‣ ecs-formationઐ༻ͷϦϙδτϦΛͭ͘Δ ‣ Task, Service, BlueGreenͷఆٛͷYAML ‣ masterʹϚʔδ͞ΕΔͱTask Deﬁnitionߋ৽ ‣ service update͸Hubot -> CircleCI͔Β

Slide 60

Slide 60 text

Update Task Definitions push merge PR webhook test update task notification webhook ecs-formation repository Amazon ECS ※࠷৽ͷTask Definitionͷόʔδϣϯʹߋ৽͞ΕΔ

Slide 61

Slide 61 text

Update Service(Deploy) webhook test update service deploy webhook ecs-formation repository Amazon ECS ※࠷৽ͷTask DeﬁnitionͷόʔδϣϯΛར༻͠ɺServiceΛߋ৽͢Δ push deploy branch notiﬁcation

Slide 62

Slide 62 text

Others

Slide 63

Slide 63 text

Others ‣ AMI ‣ Private Registry ‣ CircleCI + Docker ‣ Terraform ‣ Mackerel

Slide 64

Slide 64 text

EC2-Optimized AMI ‣ Current version 2015.09.b ‣ Amazon Linuxϕʔε ‣ Docker + ECS Agent ‣ ศར͚ͩͲࣾ಺Ͱ໘౗ݟͯ͘Εͳͦ͞͏ͩͬͨ

Slide 65

Slide 65 text

Ubuntu ‣ FRESH!͸UbuntuΛ࠾༻ ‣ DockerͷΠϯετʔϧ → ηϧϑαʔϏε ‣ ECS AgentͷಋೖͱαʔϏεԽʢUpstartʣ → ηϧϑαʔϏε ‣ ੵۃతͳDockerͷΞοϓσʔτ → ਓப ‣ apt-get install docker-engine=1.9.0-0~trusty

Slide 66

Slide 66 text

Private Registry ‣ S3ΛόοΫΤϯυʹɺPrivate RegistryΛ࡞ΕΔ ‣ registry:2.2.0 ‣ konradkleine/docker-registry-frontend:v2

Slide 67

Slide 67 text

No content

Slide 68

Slide 68 text

CircleCI + Docker ‣ ΞϓϦέʔγϣϯ͸1ϦϙδτϦʹ1Dockerﬁle ‣ nginx΍td-agentͷΑ͏ͳϛυϧ΢ΣΞܥ͸ผ్ઐ ༻ϦϙδτϦ ‣ CircleCIͰmasterϏϧυ࣌ʹdocker build + push

Slide 69

Slide 69 text

Build Docker Image push merge PR webhook test docker build private registry docker push notiﬁcation webhook

Slide 70

Slide 70 text

Terraform ‣ ΠϯϑϥߏஙͷͨΊͷΦʔέετϨʔγϣϯπʔ ϧ ‣ ଟ࠼ͳProviderΛఏڙ ‣ AWSͰͷΠϯϑϥߏஙʹར༻

Slide 71

Slide 71 text

Terraform؅ཧͯ͠Δ΋ͷ ‣ EC2 ‣ Security Group ‣ Route53 (ΠϯλʔφϧυϝΠϯͷΈʣ ‣ ECS Cluster ‣ AutoScaling Groupͷىಈߏ੒

Slide 72

Slide 72 text

Terraform؅ཧͷߟ͑ํ ‣ ӡ༻ʹΑͬͯঢ়ଶ͕มΘΔ΋ͷ؅ཧʹ͸޲͔ͳ͍ ‣ ELB ‣ AutoScaling Group ‣ εΫϥοϓʴϏϧυʹ͕͔͔࣌ؒΔ΋ͷ ‣ RDS ‣ ElastiCache ‣ EC2͸CloudinitͰߏ੒͢Δ ‣ Provisioning͸͠ͳ͍ɻ࢖͍ࣺͯ

Slide 73

Slide 73 text

ΫϦςΟΧϧͳ΋ͷ͸ආ͚Δ ‣ Route53 ‣ Ͳ͔ͬͷϓϩδΣΫτͰɺϨϏϡʔ͕ܗ֚ԽˠηϧϑϚ ˠେࣄނ͕͋ͬͨΒ͍͠ ‣ roadworkerઐ༻ϦϙδτϦ༻ҙ͠ɺݫॏʹΫϩεϨ Ϗϡʔ͢Δ ‣ IAM

Slide 74

Slide 74 text

Terraformͷӡ༻ ‣ tfϑΝΠϧ͸GHE্Ͱ؅ཧ ‣ tfstate͸S3্ʹอ࣋ ‣ શͯΛҰͭͷtfstateͰ؅ཧ͠ͳ͍ ‣ dev/shared/staging/production/load ͘Β͍ ‣ CircleCI ‣ PRͰ terraform plan ͷࠩ෼νΣοΫ ‣ planͰ͸ݫີͳνΣοΫ͸Ͱ͖ͳ͍ͷͰͦΕͳΓʹ৺؟ඞཁ ‣ hubot -> CircleCIͰ terraform apply

Slide 75

Slide 75 text

Mackerel ‣ ؂ࢹ͸جຊతʹMackerel ‣ ݟ΍͍͢ɺ͖Ε͍ ‣ ࠷ۙDockerͷϝτϦΫε͕औΕΔΑ͏ʹͳͬͨ

Slide 76

Slide 76 text

Mackerel

Slide 77

Slide 77 text

Mackerel

Slide 78

Slide 78 text

Mackerel ˡίϯςφ୯Ґͷ ϝτϦΫε

Slide 79

Slide 79 text

࠷ޙʹॴײ ‣ ECS͸΋ͪΖΜपลπʔϧ΋ἧ͖ͬͯͯɺDocker ຊ൪ӡ༻ͷෑډ͕େ͖͘Լ͕͍ͬͯΔ ‣ ೰ΜͰΔͳΒͱΓ͋͑ͣࢼͯ͠ΈΑʁ

Slide 80

Slide 80 text

Thank you for listening