Save 37% off PRO during our Black Friday Sale! »

Next FRESH! Applications with Amazon ECS

0aac627116c6e2a87b9ae179500801df?s=47 stormcat24
November 20, 2015

Next FRESH! Applications with Amazon ECS

2015/11/20 JAWS-UGコンテナ支部 #3 #jawsug_ct

0aac627116c6e2a87b9ae179500801df?s=128

stormcat24

November 20, 2015
Tweet

Transcript

  1. Next FRESH! Applications with Amazon ECS 2015/11/20 JAWS-UGίϯςφࢧ෦ #3 #jawsug_ct

    @stormcat24
  2. Who? ‣ Akinori Yamada ( @stormcat24 ) ‣ http://stormcat.hatenablog.com/ ‣

    CyberAgent, Inc ‣ Ameba౷ׅຊ෦ FRESHάϧʔϓ ‣ ServerSide / DevOps ‣ ޷͖ͳݴޠ͸Scala
  3. Agenda ‣ AmebaFRESH! ‣ Architecture & ECS ‣ Blue Green

    Deployment ‣ Diet Docker Image ‣ Local Development ‣ ecs-formation ‣ Others
  4. AmebaFRESH!

  5. ‣ ੜ์ૹಈը഑৴ϓϥοτϑΥʔϜ ‣ جຊແྉ ‣ PCϒϥ΢β / iOS / Android

    ‣ εϚϗφΠζυ͞ΕͨUI ‣ ߴ඼࣭ͳಈը഑৴ ‣ 2015೥12݄Ұൠެ։༧ఆ ‣ ݱࡏΫϩʔζυެ։த ‣ ※AbemaTVͱ͸ผͷαʔϏεͰ͢YO
  6. None
  7. None
  8. None
  9. None
  10. ‣ 2015/04͔ΒϓϩδΣΫτ։࢝ ‣ ϓϩδΣΫτϝϯόʔ໿30໊ ‣ Serverside ☓ 6 ‣ Frontend

    ☓ 6 ‣ iOS ☓ 4 ‣ Android ☓ 4 ‣ Designer ☓ 3
  11. ‣ Movie & Broadcasting ‣ RTMP Publishing ‣ HTTP Live

    Streaming (HLS)
  12. ‣ Frontend ‣ Node.js v4 ‣ TypeScript1.6 ‣ React /

    Flux ‣ SPA + SSR ‣ iOS ‣ Swift2.0 ‣ Android ‣ Kotlin
  13. ‣ Serverside & Infrastructure ‣ Amazon Web Services (AWS) ‣

    Go1.5 ‣ Docker1.9.0 ‣ Microservices ‣ RESTful API (goji)
  14. ‣ Tools ‣ Slack + Hubot (ChatOps) ‣ Github Enterprise

    ‣ CircleCI (Enterprise) ‣ Crashlytics ‣ JIRA Agile ‣ Mackerel ‣ ࡳଋ
  15. Architecture & ECS

  16. Architecture ‣ ࢥ૝ ‣ جຊํ਑ ‣ Task Definitionͷߏ੒ ‣ ClusterͱServiceͷѻ͍

  17. ໨ࢦ͍ͯ͠Δ͜ͱ ‣ ۃྗϝϯςΛೖΕͳ͍ ‣ θϩμ΢ϯλΠϜϦϦʔε → Blue Green Deployment ‣

    Πϯϑϥ͸࢖͍ࣺͯͯೖΕସ͑Δ → Immutable Infrastructure ‣ ૉૣ͘සൟͳϦϦʔε ‣ γεςϜ͸ݶΓͳ͘ૄ݁߹ͷํ͕ྑ͍ → Microservices ‣ ίϯςφͩ → Docker
  18. Microservices ‣ ಛఆͷ։ൃݴޠʹґଘ͠ͳ͍ʢ๞͖Δ͠ʣ ‣ ௨৴ϓϩτίϧ ‣ RESTFul API (HTTP) ‣

    ͦͷ͏ͪgRPC(HTTP2)͍ͨ͠ ‣ αʔϏεͷཻ౓ΛఆΊΔͷ͸΍͸Γ೉͍͠ ‣ γεςϜతͳυϝΠϯྖҬͰ෼͚Δͷ΋Ұͭͷࢦඪ ‣ ޙ͔ΒผαʔϏεͱͯ͠੾Γམͱ͢΋OK
  19. Infrastructure ‣ AWS + EC2 Container Service(ECS)Λ࠾༻ ‣ ͪΐ͏Ͳٕज़ݕূதʹECSͷ౦ژϦʔδϣϯ͕དྷͨ ‣

    ࠷௿ݶͷίϯςφߏ੒؅ཧͱεέδϡʔϦϯά͕Ͱ͖Ε͹े෼ ‣ ECSҎ֎ʹ΋࢖͍͍ͨ΋ͷ͕͋ͬͨ͠ ‣ Lambda ‣ RDS Aurora
  20. جຊํ਑ ‣ MicroservicesຖͰίϯςφΛߏ੒͢ΔTask DefinitionΛͭ͘Δ ‣ 1ECS Clusterʹ͖ͭ1Service ‣ 1Clusterʹ͸1ͭͷAutoScaling Group

    ‣ ಈը഑৴αʔό͸ಛघͳͷͰྫ֎ ‣ Clusterؒ௨৴͸Internal ELBΛར༻
  21. Task Definitionͷํ਑ ‣ ϩάͷసૹʹtd-agentΛ࢖͏ ‣ ֤ίϯςφϩά͸ϗετʹϚ΢ϯτ ‣ ϗετʹϚ΢ϯτ͞ΕͨϩάΛtd-agentͰసૹ ‣ logging

    driver͸·ͩಋೖͯ͠ͳ͍ ‣ Internal Service(API)Ͱ͋ͬͯ΋جຊNginxΛ௨͢ ‣ ΞΫηεϩάग़͢ͷָͩ͠
  22. Task Definition(Service API)

  23. Task Definition(WEB+API)

  24. Task Definition(Job)

  25. Task Definition(movie)

  26. جຊతʹ͜ΕΒͷ૊Έ߹Θͤ

  27. cluster = serviceͷσϝϦοτ ‣ Ϧιʔεޮ཰తʹ͸ϕετͰ͸ͳ͍ ‣ Cluster : Service =

    1:N ʹൺ΂Δͱ ‣ Πϯελϯε਺͸૿͑Δ܏޲ʹ͋Δ ‣ ։ൃ؀ڥͰ͸t2.microΛ༗ޮ׆༻͍ͯ͠Δ ‣ nanoΠϯελϯεщ ƅшƅщ)ŜŹŖƃ
  28. cluster = service ʹͯ͠Δཧ༝ ‣ ࢹ֮తʹΘ͔Γ΍͍͢ʢϏΪφʔϑϨϯυϦʔʣ ‣ Service୯ҐͰIAM roleΛઃఆͰ͖ͳ͍ ‣

    IAM RoleͰݫີͳݖݶ੍ޚΛ͢Δʹ͸଍Γͳ͍ ‣ Service୯ҐͰͷSecurity GroupʹະରԠ ‣ ͨͩ͠ɺELBલఏʹͯ͠͠·͑͹ղܾͰ͖Δ
  29. Blue Green Deployment

  30. 2AutoScalingύλʔϯ ‣ BlueɺGreenܥ౷ͷClusterΛ࡞Δ ‣ ͦΕͧΕ͕AutoScalingGroupʹଐ͢Δ ‣ api1-blue, api1-green Έ͍ͨͳ ‣

    AutoScalingGroup୯ҐͰELBΛ੾Γସ͑Δ
  31. None
  32. 2AutoScalingͷಛ௃ ‣ ͱͯ΋҆શ ‣ DeployޙͷϩʔϧόοΫ༰қ ‣ DeployલͷStandbyܥ౷ͷ΢ΥʔϜΞοϓඞཁ ‣ 10෼લʹ͸΍͓͖͍ͬͯͨ ‣

    ίετͷέΞ͕ॏཁ ‣ DeployޙɺStandbyʹͳͬͨܥ౷Λམͱ͢౳
  33. Diet Docker Image

  34. Πϝʔδ͸খ͍͞΄Ͳྑ͍ ‣ docker build࣌ؒˣ ‣ CI࣌ؒˣ ‣ Registry͔ΒͷΠϝʔδμ΢ϯϩʔυ࣌ؒˣ ‣ AutoScaleͰ࡞੒͞ΕͨΠϯελϯε͕αʔϏεΠϯ͢Δ

    ·Ͱͷ࣌ؒ↓
  35. docker hub ‣ hub.docker.com ‣ ਺ଟ͘ͷެࣜΠϝʔδ ‣ αΠζΛ࡟͗མͱͨ͠΋ͷ͹͔ΓͰ͸ͳ͍ ‣ 1GB௒͑ΔΠϝʔδ͸σϒ

  36. ෆཁͳ΋ͷ͸࡟আ͢Δ ‣ ෆཁͳϑΝΠϧΛݟམͱ͞ͳ͍ ‣ ϏϧυͷͨΊʹੜͨ͡࢈ۀഇغ෺Λ࡟আ ‣ npm cache clear ‣

    rm -rf ~/.grade ‣ apt-get clean ‣ Data VolumeΛ࢖͏ʢϙʔλϏϦςΟ͸མͪΔʣ
  37. RUNͷճ਺ΛݮΒ͢ ‣ RUNͷ਺͚ͩΠϝʔδͷϨΠϠʔ͕ॏͳΓɺΠ ϝʔδ༰ྔ͸૿͑Δ ‣ && ͰνΣΠϯͯ͠ɺRUNͷճ਺ΛݮΒ͢ ‣ ௕͍docker buildͷ৔߹ɺ్தͰࣦഊ͢ΔͱRUN

    ͷ಄͔Β΍Γͳ͓͠ͳͷͰফ໣͸͢Δ
  38. RUNͷճ਺ΛݮΒ͢ FROM ubuntu:15.10 RUN apt-get update RUN apt-get install -y

    curl RUN apt-get apt-get clean FROM ubuntu:15.10 RUN apt-get update && \ apt-get install -y curl && \ apt-get apt-get clean
  39. ܰྔΠϝʔδΛ࢖͏ ‣ ࡟͗མͱ͞ΕͨܰྔΠϝʔδΛ࢖͏ ‣ ࠷ۙ͸ΦϑΟγϟϧͰslimΠϝʔδ͕͋Δ΋ͷ΋͋Δ ‣ e.g. Node, Go ‣

    busybox௒ઈ͍ܰ
  40. ‣ ࠷ۙ͸ܰྔΠϝʔδ(slim)΋༻ҙ͞ ΕͯΔ ‣ Docker HubΛීஈ͔Β८ճ͓ͯ͠ ͜͏

  41. ‣ ٻΊΒΕΔϙʔλϏϦςΟˢ ‣ GoͰ͋Ε͹࣮ߦϑΝΠϧΛࡌͤΔ ͚ͩ ‣ busyboxͷதͰϏϧυ͸ΩπΠ ʢ೉қ౓ߴʣ

  42. ݮྔʹΑΔࢥΘ͵ฐ֐ ‣ x509: failed to load system roots and no

    roots provided ‣ ίϯςφ͔ΒHTTPS௨৴͕Ͱ͖ͳ͘ͳΔ ‣ apt-get install -y ca-certificates Ͱղܾ ‣ ֎෦πʔϧ΁ͷґଘʢΞϓϦ಺͔ΒͷΩοΫʣ
  43. ϕʔεΠϝʔδΛͭ͘Δ ‣ apt updateɺapt-get install ͏Μ͵ΜΛऴΘΒͤͨ ΋ͷ ‣ ຖ౓΍ͬͯͨΒCIͷ͕࣌ؒ૿͑Δʢdocker buildͷ

    ҆ఆʣ ‣ ΞϓϦଆͷDockerfileͰ͸ຊ࣭ͷॲཧʹઐ೦ͤ͞Δ
  44. Local Development

  45. ϩʔΧϧ։ൃͰ࢖͏΋ͷ ‣ docker-machine + VirtualBox ‣ docker-compose

  46. docker-machine ‣ VirtualBox, AWS, Azure, Digital Ocean্ʹ DockerϗετΛߏங͢Δ ‣ ͔͋ͨ΋ϩʔΧϧ্ʹDocker؀ڥ͕͋Δ͔ͷΑ͏

    ʹDockerͷૢ࡞͕Ͱ͖Δ
  47. docker-machine ‣ docker-machine + VirtualBoxΛબ୒ ‣ Vagrant͸ࣺͯͨ ‣ Dockerʹൺ΂Δͱ࢖͍ࣺͯίετˢ ‣

    ΞϓϦέʔγϣϯɺϛυϧ΢ΣΞؚΊ͍ͯͭͰ΋ϩʔΧϧͰ֬ ೝͰ͖ΔΑ͏ʹ ‣ VirtualBoxͷϙʔτϑΥϫʔυར༻
  48. ϚγϯϦιʔεඞཁ ‣ ϩʔΧϧ΋ϑϧDockerͩͱɺٻΊΒΕΔϚγϯεϖο Ϋ͸ߴ͘ͳΔ ‣ 16GBͳ͍ͱ݁ߏਏ͍ ‣ αʔόαΠυΤϯδχΞʹ͸ඞਢ ‣ ίϯςφ͍ͬͺ্͍͛ͯɺշదʹTwitter͢Δͷॏཁ

  49. ׳Ε͸ා͍

  50. docker-compose ‣ Docker ToolboxͷҰ෦ʢݩʑfigʣ ‣ Dockerίϯςφ܈ͷߏ੒ΛYAMLͰ؅ཧ ‣ docker-compose up -d

    Ͱىಈ
  51. σʔλετΞ΋Docker ‣ ϩʔΧϧͰ͸σʔλετΞ΋Dockerίϯςφར༻ ‣ library/mysql, library/redis ‣ VagrantΑΓ΋ߴ଎ͰؾܰʹεΫϥοϓϏϧυ

  52. DBϚΠάϨʔγϣϯॏཁ ‣ ؀ڥ͸ἧͬͯ΋σʔλෆඋ͋Δͱҙຯ͕ແ͍ ‣ FRESH! Ͱ͸ goose ‣ https://bitbucket.org/liamstask/goose/ ‣

    SQL͚ͩ͡Όͳ͘ɺGoͰϚΠάϨʔγϣϯ΋ॻ͚Δ
  53. ecs-formation

  54. ecs-formation ‣ https://github.com/stormcat24/ecs-formation ‣ docker-composeͷΑ͏ʹɺYAMLϑΝΠϧͰίϯςφ ͷߏ੒Λ؅ཧ͢Δ ‣ ౰࣌͸ecs-cliͱ͔ແ͔ͬͨͷͰ࡞ͬͨ ‣ aws-sdk-goར༻

  55. ecs-formation features ‣ Task Definitionsͷߋ৽ ‣ Clusterʹ഑ஔ͢ΔServiceͷߋ৽ ‣ Blue-Green Deploymentͷ࣮ߦ

  56. Task Definitions (task/jawsug-api.yml) nginx: image: registry.jawsug.local:5000/jawsug/nginx:latest ports: - 80:80 environment:

    SERVER_NAME: jawsug.example.com volumes: - /var/log/container/nginx:/var/log/nginx links: - api memory: 512 cpu_units: 512 essential: true
  57. Services on cluster (service/jawsug-cluster.yml) api-service: task_definition: jawsug-api desired_count: 1

  58. Blue Green Deployment (bluegreen/jawsug-cluster.yml) blue: cluster: jawsug-cluster-blue service: api-service autoscaling_group:

    jawsug-cluster-blue green: cluster: jawsug-cluster-green service: api-service autoscaling_group: jawsug-cluster-green primary_elb: jawsug-api-primary standby_elb: jawsug-api-standby
  59. ecs-formationͷӡ༻ ‣ ecs-formationઐ༻ͷϦϙδτϦΛͭ͘Δ ‣ Task, Service, BlueGreenͷఆٛͷYAML ‣ masterʹϚʔδ͞ΕΔͱTask Definitionߋ৽

    ‣ service update͸Hubot -> CircleCI͔Β
  60. Update Task Definitions push merge PR webhook test update task

    notification webhook ecs-formation repository Amazon ECS ※࠷৽ͷTask Definitionͷόʔδϣϯʹߋ৽͞ΕΔ
  61. Update Service(Deploy) webhook test update service deploy webhook ecs-formation repository

    Amazon ECS ※࠷৽ͷTask DefinitionͷόʔδϣϯΛར༻͠ɺServiceΛߋ৽͢Δ push deploy branch notification
  62. Others

  63. Others ‣ AMI ‣ Private Registry ‣ CircleCI + Docker

    ‣ Terraform ‣ Mackerel
  64. EC2-Optimized AMI ‣ Current version 2015.09.b ‣ Amazon Linuxϕʔε ‣

    Docker + ECS Agent ‣ ศར͚ͩͲࣾ಺Ͱ໘౗ݟͯ͘Εͳͦ͞͏ͩͬͨ
  65. Ubuntu ‣ FRESH!͸UbuntuΛ࠾༻ ‣ DockerͷΠϯετʔϧ → ηϧϑαʔϏε ‣ ECS AgentͷಋೖͱαʔϏεԽʢUpstartʣ

    → ηϧϑαʔϏε ‣ ੵۃతͳDockerͷΞοϓσʔτ → ਓப ‣ apt-get install docker-engine=1.9.0-0~trusty
  66. Private Registry ‣ S3ΛόοΫΤϯυʹɺPrivate RegistryΛ࡞ΕΔ ‣ registry:2.2.0 ‣ konradkleine/docker-registry-frontend:v2

  67. None
  68. CircleCI + Docker ‣ ΞϓϦέʔγϣϯ͸1ϦϙδτϦʹ1Dockerfile ‣ nginx΍td-agentͷΑ͏ͳϛυϧ΢ΣΞܥ͸ผ్ઐ ༻ϦϙδτϦ ‣ CircleCIͰmasterϏϧυ࣌ʹdocker

    build + push
  69. Build Docker Image push merge PR webhook test docker build

    private registry docker push notification webhook
  70. Terraform ‣ ΠϯϑϥߏஙͷͨΊͷΦʔέετϨʔγϣϯπʔ ϧ ‣ ଟ࠼ͳProviderΛఏڙ ‣ AWSͰͷΠϯϑϥߏஙʹར༻

  71. Terraform؅ཧͯ͠Δ΋ͷ ‣ EC2 ‣ Security Group ‣ Route53 (ΠϯλʔφϧυϝΠϯͷΈʣ ‣

    ECS Cluster ‣ AutoScaling Groupͷىಈߏ੒
  72. Terraform؅ཧͷߟ͑ํ ‣ ӡ༻ʹΑͬͯঢ়ଶ͕มΘΔ΋ͷ؅ཧʹ͸޲͔ͳ͍ ‣ ELB ‣ AutoScaling Group ‣ εΫϥοϓʴϏϧυʹ͕͔͔࣌ؒΔ΋ͷ

    ‣ RDS ‣ ElastiCache ‣ EC2͸CloudinitͰߏ੒͢Δ ‣ Provisioning͸͠ͳ͍ɻ࢖͍ࣺͯ
  73. ΫϦςΟΧϧͳ΋ͷ͸ආ͚Δ ‣ Route53 ‣ Ͳ͔ͬͷϓϩδΣΫτͰɺϨϏϡʔ͕ܗ֚ԽˠηϧϑϚ ˠେࣄނ͕͋ͬͨΒ͍͠ ‣ roadworkerઐ༻ϦϙδτϦ༻ҙ͠ɺݫॏʹΫϩεϨ Ϗϡʔ͢Δ ‣

    IAM
  74. Terraformͷӡ༻ ‣ tfϑΝΠϧ͸GHE্Ͱ؅ཧ ‣ tfstate͸S3্ʹอ࣋ ‣ શͯΛҰͭͷtfstateͰ؅ཧ͠ͳ͍ ‣ dev/shared/staging/production/load ͘Β͍

    ‣ CircleCI ‣ PRͰ terraform plan ͷࠩ෼νΣοΫ ‣ planͰ͸ݫີͳνΣοΫ͸Ͱ͖ͳ͍ͷͰͦΕͳΓʹ৺؟ඞཁ ‣ hubot -> CircleCIͰ terraform apply
  75. Mackerel ‣ ؂ࢹ͸جຊతʹMackerel ‣ ݟ΍͍͢ɺ͖Ε͍ ‣ ࠷ۙDockerͷϝτϦΫε͕औΕΔΑ͏ʹͳͬͨ

  76. Mackerel

  77. Mackerel

  78. Mackerel ˡίϯςφ୯Ґͷ ϝτϦΫε

  79. ࠷ޙʹॴײ ‣ ECS͸΋ͪΖΜपลπʔϧ΋ἧ͖ͬͯͯɺDocker ຊ൪ӡ༻ͷෑډ͕େ͖͘Լ͕͍ͬͯΔ ‣ ೰ΜͰΔͳΒͱΓ͋͑ͣࢼͯ͠ΈΑʁ

  80. Thank you for listening