Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Next FRESH! Applications with Amazon ECS

stormcat24
November 20, 2015

Next FRESH! Applications with Amazon ECS

2015/11/20 JAWS-UGコンテナ支部 #3 #jawsug_ct

stormcat24

November 20, 2015
Tweet

More Decks by stormcat24

Other Decks in Programming

Transcript

  1. Next FRESH! Applications
    with Amazon ECS
    2015/11/20 JAWS-UGίϯςφࢧ෦ #3 #jawsug_ct
    @stormcat24

    View full-size slide

  2. Who?
    ‣ Akinori Yamada ( @stormcat24 )
    ‣ http://stormcat.hatenablog.com/
    ‣ CyberAgent, Inc
    ‣ Ameba౷ׅຊ෦ FRESHάϧʔϓ
    ‣ ServerSide / DevOps
    ‣ ޷͖ͳݴޠ͸Scala

    View full-size slide

  3. Agenda
    ‣ AmebaFRESH!
    ‣ Architecture & ECS
    ‣ Blue Green Deployment
    ‣ Diet Docker Image
    ‣ Local Development
    ‣ ecs-formation
    ‣ Others

    View full-size slide

  4. ‣ ੜ์ૹಈը഑৴ϓϥοτϑΥʔϜ
    ‣ جຊແྉ
    ‣ PCϒϥ΢β / iOS / Android
    ‣ εϚϗφΠζυ͞ΕͨUI
    ‣ ߴ඼࣭ͳಈը഑৴
    ‣ 2015೥12݄Ұൠެ։༧ఆ
    ‣ ݱࡏΫϩʔζυެ։த
    ‣ ※AbemaTVͱ͸ผͷαʔϏεͰ͢YO

    View full-size slide

  5. ‣ 2015/04͔ΒϓϩδΣΫτ։࢝
    ‣ ϓϩδΣΫτϝϯόʔ໿30໊
    ‣ Serverside ☓ 6
    ‣ Frontend ☓ 6
    ‣ iOS ☓ 4
    ‣ Android ☓ 4
    ‣ Designer ☓ 3

    View full-size slide

  6. ‣ Movie & Broadcasting
    ‣ RTMP Publishing
    ‣ HTTP Live Streaming (HLS)

    View full-size slide

  7. ‣ Frontend
    ‣ Node.js v4
    ‣ TypeScript1.6
    ‣ React / Flux
    ‣ SPA + SSR
    ‣ iOS
    ‣ Swift2.0
    ‣ Android
    ‣ Kotlin

    View full-size slide

  8. ‣ Serverside & Infrastructure
    ‣ Amazon Web Services (AWS)
    ‣ Go1.5
    ‣ Docker1.9.0
    ‣ Microservices
    ‣ RESTful API (goji)

    View full-size slide

  9. ‣ Tools
    ‣ Slack + Hubot (ChatOps)
    ‣ Github Enterprise
    ‣ CircleCI (Enterprise)
    ‣ Crashlytics
    ‣ JIRA Agile
    ‣ Mackerel
    ‣ ࡳଋ

    View full-size slide

  10. Architecture & ECS

    View full-size slide

  11. Architecture
    ‣ ࢥ૝
    ‣ جຊํ਑
    ‣ Task Definitionͷߏ੒
    ‣ ClusterͱServiceͷѻ͍

    View full-size slide

  12. ໨ࢦ͍ͯ͠Δ͜ͱ
    ‣ ۃྗϝϯςΛೖΕͳ͍
    ‣ θϩμ΢ϯλΠϜϦϦʔε → Blue Green Deployment
    ‣ Πϯϑϥ͸࢖͍ࣺͯͯೖΕସ͑Δ → Immutable Infrastructure
    ‣ ૉૣ͘සൟͳϦϦʔε
    ‣ γεςϜ͸ݶΓͳ͘ૄ݁߹ͷํ͕ྑ͍ → Microservices
    ‣ ίϯςφͩ → Docker

    View full-size slide

  13. Microservices
    ‣ ಛఆͷ։ൃݴޠʹґଘ͠ͳ͍ʢ๞͖Δ͠ʣ
    ‣ ௨৴ϓϩτίϧ
    ‣ RESTFul API (HTTP)
    ‣ ͦͷ͏ͪgRPC(HTTP2)͍ͨ͠
    ‣ αʔϏεͷཻ౓ΛఆΊΔͷ͸΍͸Γ೉͍͠
    ‣ γεςϜతͳυϝΠϯྖҬͰ෼͚Δͷ΋Ұͭͷࢦඪ
    ‣ ޙ͔ΒผαʔϏεͱͯ͠੾Γམͱ͢΋OK

    View full-size slide

  14. Infrastructure
    ‣ AWS + EC2 Container Service(ECS)Λ࠾༻
    ‣ ͪΐ͏Ͳٕज़ݕূதʹECSͷ౦ژϦʔδϣϯ͕དྷͨ
    ‣ ࠷௿ݶͷίϯςφߏ੒؅ཧͱεέδϡʔϦϯά͕Ͱ͖Ε͹े෼
    ‣ ECSҎ֎ʹ΋࢖͍͍ͨ΋ͷ͕͋ͬͨ͠
    ‣ Lambda
    ‣ RDS Aurora

    View full-size slide

  15. جຊํ਑
    ‣ MicroservicesຖͰίϯςφΛߏ੒͢ΔTask DefinitionΛͭ͘Δ
    ‣ 1ECS Clusterʹ͖ͭ1Service
    ‣ 1Clusterʹ͸1ͭͷAutoScaling Group
    ‣ ಈը഑৴αʔό͸ಛघͳͷͰྫ֎
    ‣ Clusterؒ௨৴͸Internal ELBΛར༻

    View full-size slide

  16. Task Definitionͷํ਑
    ‣ ϩάͷసૹʹtd-agentΛ࢖͏
    ‣ ֤ίϯςφϩά͸ϗετʹϚ΢ϯτ
    ‣ ϗετʹϚ΢ϯτ͞ΕͨϩάΛtd-agentͰసૹ
    ‣ logging driver͸·ͩಋೖͯ͠ͳ͍
    ‣ Internal Service(API)Ͱ͋ͬͯ΋جຊNginxΛ௨͢
    ‣ ΞΫηεϩάग़͢ͷָͩ͠

    View full-size slide

  17. Task Definition(Service API)

    View full-size slide

  18. Task Definition(WEB+API)

    View full-size slide

  19. Task Definition(Job)

    View full-size slide

  20. Task Definition(movie)

    View full-size slide

  21. جຊతʹ͜ΕΒͷ૊Έ߹Θͤ

    View full-size slide

  22. cluster = serviceͷσϝϦοτ
    ‣ Ϧιʔεޮ཰తʹ͸ϕετͰ͸ͳ͍
    ‣ Cluster : Service = 1:N ʹൺ΂Δͱ
    ‣ Πϯελϯε਺͸૿͑Δ܏޲ʹ͋Δ
    ‣ ։ൃ؀ڥͰ͸t2.microΛ༗ޮ׆༻͍ͯ͠Δ
    ‣ nanoΠϯελϯεщ ƅшƅщ)ŜŹŖƃ

    View full-size slide

  23. cluster = service ʹͯ͠Δཧ༝
    ‣ ࢹ֮తʹΘ͔Γ΍͍͢ʢϏΪφʔϑϨϯυϦʔʣ
    ‣ Service୯ҐͰIAM roleΛઃఆͰ͖ͳ͍
    ‣ IAM RoleͰݫີͳݖݶ੍ޚΛ͢Δʹ͸଍Γͳ͍
    ‣ Service୯ҐͰͷSecurity GroupʹະରԠ
    ‣ ͨͩ͠ɺELBલఏʹͯ͠͠·͑͹ղܾͰ͖Δ

    View full-size slide

  24. Blue Green Deployment

    View full-size slide

  25. 2AutoScalingύλʔϯ
    ‣ BlueɺGreenܥ౷ͷClusterΛ࡞Δ
    ‣ ͦΕͧΕ͕AutoScalingGroupʹଐ͢Δ
    ‣ api1-blue, api1-green Έ͍ͨͳ
    ‣ AutoScalingGroup୯ҐͰELBΛ੾Γସ͑Δ

    View full-size slide

  26. 2AutoScalingͷಛ௃
    ‣ ͱͯ΋҆શ
    ‣ DeployޙͷϩʔϧόοΫ༰қ
    ‣ DeployલͷStandbyܥ౷ͷ΢ΥʔϜΞοϓඞཁ
    ‣ 10෼લʹ͸΍͓͖͍ͬͯͨ
    ‣ ίετͷέΞ͕ॏཁ
    ‣ DeployޙɺStandbyʹͳͬͨܥ౷Λམͱ͢౳

    View full-size slide

  27. Diet Docker Image

    View full-size slide

  28. Πϝʔδ͸খ͍͞΄Ͳྑ͍
    ‣ docker build࣌ؒˣ
    ‣ CI࣌ؒˣ
    ‣ Registry͔ΒͷΠϝʔδμ΢ϯϩʔυ࣌ؒˣ
    ‣ AutoScaleͰ࡞੒͞ΕͨΠϯελϯε͕αʔϏεΠϯ͢Δ
    ·Ͱͷ࣌ؒ↓

    View full-size slide

  29. docker hub
    ‣ hub.docker.com
    ‣ ਺ଟ͘ͷެࣜΠϝʔδ
    ‣ αΠζΛ࡟͗མͱͨ͠΋ͷ͹͔ΓͰ͸ͳ͍
    ‣ 1GB௒͑ΔΠϝʔδ͸σϒ

    View full-size slide

  30. ෆཁͳ΋ͷ͸࡟আ͢Δ
    ‣ ෆཁͳϑΝΠϧΛݟམͱ͞ͳ͍
    ‣ ϏϧυͷͨΊʹੜͨ͡࢈ۀഇغ෺Λ࡟আ
    ‣ npm cache clear
    ‣ rm -rf ~/.grade
    ‣ apt-get clean
    ‣ Data VolumeΛ࢖͏ʢϙʔλϏϦςΟ͸མͪΔʣ

    View full-size slide

  31. RUNͷճ਺ΛݮΒ͢
    ‣ RUNͷ਺͚ͩΠϝʔδͷϨΠϠʔ͕ॏͳΓɺΠ
    ϝʔδ༰ྔ͸૿͑Δ
    ‣ && ͰνΣΠϯͯ͠ɺRUNͷճ਺ΛݮΒ͢
    ‣ ௕͍docker buildͷ৔߹ɺ్தͰࣦഊ͢ΔͱRUN
    ͷ಄͔Β΍Γͳ͓͠ͳͷͰফ໣͸͢Δ

    View full-size slide

  32. RUNͷճ਺ΛݮΒ͢
    FROM ubuntu:15.10
    RUN apt-get update
    RUN apt-get install -y curl
    RUN apt-get apt-get clean
    FROM ubuntu:15.10
    RUN apt-get update && \
    apt-get install -y curl && \
    apt-get apt-get clean

    View full-size slide

  33. ܰྔΠϝʔδΛ࢖͏
    ‣ ࡟͗མͱ͞ΕͨܰྔΠϝʔδΛ࢖͏
    ‣ ࠷ۙ͸ΦϑΟγϟϧͰslimΠϝʔδ͕͋Δ΋ͷ΋͋Δ
    ‣ e.g. Node, Go
    ‣ busybox௒ઈ͍ܰ

    View full-size slide

  34. ‣ ࠷ۙ͸ܰྔΠϝʔδ(slim)΋༻ҙ͞
    ΕͯΔ
    ‣ Docker HubΛීஈ͔Β८ճ͓ͯ͠
    ͜͏

    View full-size slide

  35. ‣ ٻΊΒΕΔϙʔλϏϦςΟˢ
    ‣ GoͰ͋Ε͹࣮ߦϑΝΠϧΛࡌͤΔ
    ͚ͩ
    ‣ busyboxͷதͰϏϧυ͸ΩπΠ
    ʢ೉қ౓ߴʣ

    View full-size slide

  36. ݮྔʹΑΔࢥΘ͵ฐ֐
    ‣ x509: failed to load system roots and no roots
    provided
    ‣ ίϯςφ͔ΒHTTPS௨৴͕Ͱ͖ͳ͘ͳΔ
    ‣ apt-get install -y ca-certificates Ͱղܾ
    ‣ ֎෦πʔϧ΁ͷґଘʢΞϓϦ಺͔ΒͷΩοΫʣ

    View full-size slide

  37. ϕʔεΠϝʔδΛͭ͘Δ
    ‣ apt updateɺapt-get install ͏Μ͵ΜΛऴΘΒͤͨ
    ΋ͷ
    ‣ ຖ౓΍ͬͯͨΒCIͷ͕࣌ؒ૿͑Δʢdocker buildͷ
    ҆ఆʣ
    ‣ ΞϓϦଆͷDockerfileͰ͸ຊ࣭ͷॲཧʹઐ೦ͤ͞Δ

    View full-size slide

  38. Local Development

    View full-size slide

  39. ϩʔΧϧ։ൃͰ࢖͏΋ͷ
    ‣ docker-machine + VirtualBox
    ‣ docker-compose

    View full-size slide

  40. docker-machine
    ‣ VirtualBox, AWS, Azure, Digital Ocean্ʹ
    DockerϗετΛߏங͢Δ
    ‣ ͔͋ͨ΋ϩʔΧϧ্ʹDocker؀ڥ͕͋Δ͔ͷΑ͏
    ʹDockerͷૢ࡞͕Ͱ͖Δ

    View full-size slide

  41. docker-machine
    ‣ docker-machine + VirtualBoxΛબ୒
    ‣ Vagrant͸ࣺͯͨ
    ‣ Dockerʹൺ΂Δͱ࢖͍ࣺͯίετˢ
    ‣ ΞϓϦέʔγϣϯɺϛυϧ΢ΣΞؚΊ͍ͯͭͰ΋ϩʔΧϧͰ֬
    ೝͰ͖ΔΑ͏ʹ
    ‣ VirtualBoxͷϙʔτϑΥϫʔυར༻

    View full-size slide

  42. ϚγϯϦιʔεඞཁ
    ‣ ϩʔΧϧ΋ϑϧDockerͩͱɺٻΊΒΕΔϚγϯεϖο
    Ϋ͸ߴ͘ͳΔ
    ‣ 16GBͳ͍ͱ݁ߏਏ͍
    ‣ αʔόαΠυΤϯδχΞʹ͸ඞਢ
    ‣ ίϯςφ͍ͬͺ্͍͛ͯɺշదʹTwitter͢Δͷॏཁ

    View full-size slide

  43. docker-compose
    ‣ Docker ToolboxͷҰ෦ʢݩʑfigʣ
    ‣ Dockerίϯςφ܈ͷߏ੒ΛYAMLͰ؅ཧ
    ‣ docker-compose up -d Ͱىಈ

    View full-size slide

  44. σʔλετΞ΋Docker
    ‣ ϩʔΧϧͰ͸σʔλετΞ΋Dockerίϯςφར༻
    ‣ library/mysql, library/redis
    ‣ VagrantΑΓ΋ߴ଎ͰؾܰʹεΫϥοϓϏϧυ

    View full-size slide

  45. DBϚΠάϨʔγϣϯॏཁ
    ‣ ؀ڥ͸ἧͬͯ΋σʔλෆඋ͋Δͱҙຯ͕ແ͍
    ‣ FRESH! Ͱ͸ goose
    ‣ https://bitbucket.org/liamstask/goose/
    ‣ SQL͚ͩ͡Όͳ͘ɺGoͰϚΠάϨʔγϣϯ΋ॻ͚Δ

    View full-size slide

  46. ecs-formation

    View full-size slide

  47. ecs-formation
    ‣ https://github.com/stormcat24/ecs-formation
    ‣ docker-composeͷΑ͏ʹɺYAMLϑΝΠϧͰίϯςφ
    ͷߏ੒Λ؅ཧ͢Δ
    ‣ ౰࣌͸ecs-cliͱ͔ແ͔ͬͨͷͰ࡞ͬͨ
    ‣ aws-sdk-goར༻

    View full-size slide

  48. ecs-formation features
    ‣ Task Definitionsͷߋ৽
    ‣ Clusterʹ഑ஔ͢ΔServiceͷߋ৽
    ‣ Blue-Green Deploymentͷ࣮ߦ

    View full-size slide

  49. Task Definitions
    (task/jawsug-api.yml)
    nginx:
    image: registry.jawsug.local:5000/jawsug/nginx:latest
    ports:
    - 80:80
    environment:
    SERVER_NAME: jawsug.example.com
    volumes:
    - /var/log/container/nginx:/var/log/nginx
    links:
    - api
    memory: 512
    cpu_units: 512
    essential: true

    View full-size slide

  50. Services on cluster
    (service/jawsug-cluster.yml)
    api-service:
    task_definition: jawsug-api
    desired_count: 1

    View full-size slide

  51. Blue Green Deployment
    (bluegreen/jawsug-cluster.yml)
    blue:
    cluster: jawsug-cluster-blue
    service: api-service
    autoscaling_group: jawsug-cluster-blue
    green:
    cluster: jawsug-cluster-green
    service: api-service
    autoscaling_group: jawsug-cluster-green
    primary_elb: jawsug-api-primary
    standby_elb: jawsug-api-standby

    View full-size slide

  52. ecs-formationͷӡ༻
    ‣ ecs-formationઐ༻ͷϦϙδτϦΛͭ͘Δ
    ‣ Task, Service, BlueGreenͷఆٛͷYAML
    ‣ masterʹϚʔδ͞ΕΔͱTask Definitionߋ৽
    ‣ service update͸Hubot -> CircleCI͔Β

    View full-size slide

  53. Update Task Definitions
    push
    merge PR
    webhook
    test
    update
    task
    notification
    webhook
    ecs-formation
    repository
    Amazon ECS
    ※࠷৽ͷTask Definitionͷόʔδϣϯʹߋ৽͞ΕΔ

    View full-size slide

  54. Update Service(Deploy)
    webhook
    test
    update
    service
    deploy
    webhook
    ecs-formation
    repository
    Amazon ECS
    ※࠷৽ͷTask DefinitionͷόʔδϣϯΛར༻͠ɺServiceΛߋ৽͢Δ
    push
    deploy branch
    notification

    View full-size slide

  55. Others
    ‣ AMI
    ‣ Private Registry
    ‣ CircleCI + Docker
    ‣ Terraform
    ‣ Mackerel

    View full-size slide

  56. EC2-Optimized AMI
    ‣ Current version 2015.09.b
    ‣ Amazon Linuxϕʔε
    ‣ Docker + ECS Agent
    ‣ ศར͚ͩͲࣾ಺Ͱ໘౗ݟͯ͘Εͳͦ͞͏ͩͬͨ

    View full-size slide

  57. Ubuntu
    ‣ FRESH!͸UbuntuΛ࠾༻
    ‣ DockerͷΠϯετʔϧ → ηϧϑαʔϏε
    ‣ ECS AgentͷಋೖͱαʔϏεԽʢUpstartʣ → ηϧϑαʔϏε
    ‣ ੵۃతͳDockerͷΞοϓσʔτ → ਓப
    ‣ apt-get install docker-engine=1.9.0-0~trusty

    View full-size slide

  58. Private Registry
    ‣ S3ΛόοΫΤϯυʹɺPrivate RegistryΛ࡞ΕΔ
    ‣ registry:2.2.0
    ‣ konradkleine/docker-registry-frontend:v2

    View full-size slide

  59. CircleCI + Docker
    ‣ ΞϓϦέʔγϣϯ͸1ϦϙδτϦʹ1Dockerfile
    ‣ nginx΍td-agentͷΑ͏ͳϛυϧ΢ΣΞܥ͸ผ్ઐ
    ༻ϦϙδτϦ
    ‣ CircleCIͰmasterϏϧυ࣌ʹdocker build + push

    View full-size slide

  60. Build Docker Image
    push
    merge PR
    webhook
    test
    docker build
    private registry
    docker push
    notification
    webhook

    View full-size slide

  61. Terraform
    ‣ ΠϯϑϥߏஙͷͨΊͷΦʔέετϨʔγϣϯπʔ
    ϧ
    ‣ ଟ࠼ͳProviderΛఏڙ
    ‣ AWSͰͷΠϯϑϥߏஙʹར༻

    View full-size slide

  62. Terraform؅ཧͯ͠Δ΋ͷ
    ‣ EC2
    ‣ Security Group
    ‣ Route53 (ΠϯλʔφϧυϝΠϯͷΈʣ
    ‣ ECS Cluster
    ‣ AutoScaling Groupͷىಈߏ੒

    View full-size slide

  63. Terraform؅ཧͷߟ͑ํ
    ‣ ӡ༻ʹΑͬͯঢ়ଶ͕มΘΔ΋ͷ؅ཧʹ͸޲͔ͳ͍
    ‣ ELB
    ‣ AutoScaling Group
    ‣ εΫϥοϓʴϏϧυʹ͕͔͔࣌ؒΔ΋ͷ
    ‣ RDS
    ‣ ElastiCache
    ‣ EC2͸CloudinitͰߏ੒͢Δ
    ‣ Provisioning͸͠ͳ͍ɻ࢖͍ࣺͯ

    View full-size slide

  64. ΫϦςΟΧϧͳ΋ͷ͸ආ͚Δ
    ‣ Route53
    ‣ Ͳ͔ͬͷϓϩδΣΫτͰɺϨϏϡʔ͕ܗ֚ԽˠηϧϑϚ
    ˠେࣄނ͕͋ͬͨΒ͍͠
    ‣ roadworkerઐ༻ϦϙδτϦ༻ҙ͠ɺݫॏʹΫϩεϨ
    Ϗϡʔ͢Δ
    ‣ IAM

    View full-size slide

  65. Terraformͷӡ༻
    ‣ tfϑΝΠϧ͸GHE্Ͱ؅ཧ
    ‣ tfstate͸S3্ʹอ࣋
    ‣ શͯΛҰͭͷtfstateͰ؅ཧ͠ͳ͍
    ‣ dev/shared/staging/production/load ͘Β͍
    ‣ CircleCI
    ‣ PRͰ terraform plan ͷࠩ෼νΣοΫ
    ‣ planͰ͸ݫີͳνΣοΫ͸Ͱ͖ͳ͍ͷͰͦΕͳΓʹ৺؟ඞཁ
    ‣ hubot -> CircleCIͰ terraform apply

    View full-size slide

  66. Mackerel
    ‣ ؂ࢹ͸جຊతʹMackerel
    ‣ ݟ΍͍͢ɺ͖Ε͍
    ‣ ࠷ۙDockerͷϝτϦΫε͕औΕΔΑ͏ʹͳͬͨ

    View full-size slide

  67. Mackerel
    ˡίϯςφ୯Ґͷ
    ϝτϦΫε

    View full-size slide

  68. ࠷ޙʹॴײ
    ‣ ECS͸΋ͪΖΜपลπʔϧ΋ἧ͖ͬͯͯɺDocker
    ຊ൪ӡ༻ͷෑډ͕େ͖͘Լ͕͍ͬͯΔ
    ‣ ೰ΜͰΔͳΒͱΓ͋͑ͣࢼͯ͠ΈΑʁ

    View full-size slide

  69. Thank you for listening

    View full-size slide