サービス開発の現場を語る! #chibadan の発表資料です http://chibadan.connpass.com/event/41625/
8BOUFEMZ͔Β $IFGΛҰͨ͠2016-10-25αʔϏε։ൃͷݱΛޠΔʂ Daisuke Fujita (@dtan4)https://flic.kr/p/rU5bCU
View Slide
Daisuke Fujita / @dtan4Engineer, Infrastructure Team @ Wantedly, Inc.AWS, Terraform / Terraforming, Developer Productivity
8BOUFEMZ͔Β $IFGΛҰͨͭͭ͠%PDLFSपΓΛஔ͖͑ͨ2016-10-25αʔϏε։ൃͷݱΛޠΔʂ Daisuke Fujita (@dtan4)https://flic.kr/p/rU5bCU
Infrastructure at WantedlyAWS + DNSimpleશαʔϏε͕ Docker ίϯςφͰՔಇEC2 Πϯελϯεͷ OS CoreOS ੲ Ubuntu + ChefCapistrano ΛͬͨಠࣗͷσϓϩΠγεςϜ + Heroku CLI ಉ༷ͷΠϯλʔϑΣΠεΛ࣋ͭ CLIhttps://speakerdeck.com/dtan4/number-abejameetup
Web (Rails)Docker image 2ஈߏࠩͷΈͷ fetch + bundle install Ͱ σϓϩΠΛߴԽ intermediate Chef + Packer ΠϝʔδαΠζ 3GB bundle installgit clone wantedly/wantedlyRubyFROM Ubuntu 14.04rake assets:precompilebundle installgit checkout git fetchFROM intermediateweb (Dockerfile)intermediate (Chef)
pull :intermediatepush :productionpull :productionbuild :production$ sap prod deploy(3:00 am)docker push :intermediate
Chef + PackerAWS Ҡߦॳ͔Β2Ҏ্Ҿ͖ܧ͕Ε͖ͯͨൿͷ Chef ϨγϐPacker Ͱ ubuntu:14.04 Πϝʔδ্ʹϓϩϏδϣχϯά => Private Registry ʹ push
Chef + Packer ???ઐ༻ͷ EC2 Πϯελϯε1ͷΈͰՔಇ͍ͯͨ͠ ਓؒͷ Macbook ͰϓϩϏδϣχϯάͰ͖ͳ͘ͳ͍ͬͯͨ Chef ʢͷΠϯετʔϧํ๏ʣͷਐԽʹैͦ͜͠Ͷͨ
Chef + Packer ???1Πϝʔδ࡞Δͷʹ1࣌ؒҎ্ Ͱ͖͕͋ͬͨΠϝʔδ 3GB Ruby ͷόʔδϣϯΞοϓ apt ύοέʔδՃ͕໘ ΠϯϑϥνʔϜҎ֎͕৮Γʹ͍͘σϓϩΠ࣌ͷ docker build ֤Πϯελϯεͷ͕͍
Chef + Packer => ???ΠϝʔδαΠζΛͬͱখ͍ͨ͘͞͠…ଞͷϝϯόʔ৮Γ͍͢ߏʹ͍ͨ͠…σϓϩΠ͍ͨ͘͠…Dockerfile Ͱॻ͖͍ͨ…ʂ
Private Registry => ???Docker Private Registry V1 (Python) ʹຊؾͰٽ͔͞ΕͨDocker Distribution (Go) ͍ͩͿϚγʹͳ͚ͬͨͲ Registry ίϯςφ͕ࢮ͵ڪා૬มΘΒͣϚωʔδυαʔϏεʹҠߦ͍ͨ͠…ʂ ECR ౦ژʹདྷͨ͠
2016/10
2016/10Wantedly, Inc. ͔Β Chef ΛҰ
2016/10 Chef + Packer -> Dockerfile + Docker Compose Ubuntu 14.04 -> Ubuntu 16.04, Alpine Linux 3.4 Private Docker Registry -> Amazon ECRWantedly, Inc. ͔Β Chef ΛҰ
(3:00 am)docker push :intermediatepull :intermediatepush :productionpull :productionbuild :production$ sap prod deploy
DockerfileChef Ϩγϐ -> Dockerfile 1ຕʹू
DockerfileαʔϏε͝ͱʹ1ຕͷ DockerfileFROM ubuntu 16.04 / alpine:3.4Dockerfile ϕετϓϥΫςΟε Λ࣮ફͯ͠μΠΤοτ
DockerfileRuby, Node.js ͷΠϯετʔϧ ಉ͡ Dockerfile ʹॻ͘ެࣜͷ -alpine Dockerfile ͔Βഈआ Alpine ͷόʔδϣϯ ࣗͨͪͰ੍ޚ αʔϏεʹ߹Θͤͨݴޠόʔδϣϯ
DockerfileλΠϜελϯϓΛૠೖͯ͠ ҎԼຖճڧ੍తʹϏϧυιʔείʔυΛ git clone Access Token Λͬͨೝূbundle install -> assets:precompileARG BUILD_DATE--build-args BUILD_DATE=$(date +%Y%m%d%H%M%S)
Docker ComposeશΠϝʔδ Docker Compose ܦ༝Ͱ build & pushbuild args ΛҰׅཧ Ruby ͷόʔδϣϯΞοϓ ͔͜͜Βawesome:extends: basebuild:context: awesomeargs:- RUBY_MAJOR=2.3- RUBY_VERSION=2.3.0- RUBYGEMS_VERSION=2.6.6- BUNDLER_VERSION=1.13.1- NODE_VERSION=5.11.1- NPM_VERSION=3.3.12awesome-production:extends: awesomebuild:args:- RAILS_ENV=productionimage: ${AWS_ECR_DOMAIN}/awesome:intermediate
Alpine Linuxbusybox + α ͷܰྔσΟετϦ (~ 5MB)ύοέʔδϚωʔδϟ apk͍͍ͨͯͳ͘ಈ͘ɻ ͳΜ͔Μ apk ʹϥΠϒϥϦҰ௨ΓἧͬͯΔɻ
Alpine Linux ʹҠߦ͢Δҙ֎෦ίϚϯυݺͼग़͠ҙ `...` ͱ͔ os.Exec(...) ͱ͔ ݺͼग़͢όΠφϦ͕ Alpine Linux ্ͩͱ ಈ͔ͳ͍Մೳੑ͕͋Δ (ex. wkhtmltopdf) ඪ४ίϚϯυ (ex. timeout) busybox ͳͷͰ Ubuntu ͷʹൺͯύϥϝʔλগͳ͔ͬͨΓ
Alpine Linux ʹҠߦ͢Δҙ όΠφϦΛ static link ͰϏϧυ͢Δ alpine-pkg-glibc, coreutils ΛೖΕΔ ֎෦ίϚϯυݺͼग़͠Ίͯ Pure ͳखஈʹͩΊͳΒૉʹ Ubuntu CentOS ͱ͍ͬͨ ϝδϟʔͳσΟετϦΛ͏ దʹαʔϏεΛఏڙͰ͖Δͷ͕͋͘·Ͱେલఏ
intermediate-build-serverintermediate image Λఆظతʹ build ͢ΔαʔόCoreOS on EC2 + IAM Roleຖਂʹશ intermediate image Λ docker-compose build -> docker-compose push
intermediate-build-server#!/bin/bashset -eumetadata=$(curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/intermediate-build-server/)export AWS_ACCESS_KEY_ID=$(echo $metadata | jq -r .AccessKeyId)export AWS_SECRET_ACCESS_KEY=$(echo $metadata | jq -r .SecretAccessKey)export AWS_SECURITY_TOKEN=$(echo $metadata | jq -r .Token)document=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document)account_id=$(echo $document | jq -r .accountId)region=$(echo $document | jq -r .region)export AWS_ECR_DOMAIN=$account_id.dkr.ecr.$region.amazonaws.comexport BUILD_DATE=$(date +%Y%m%d%H%M%S)cd intermediate-dockerfilesif [[ $# -eq 1 ]]; then/opt/bin/docker-compose build $1elsefor app in $(cat docker-compose.yml | grep -E '^ [a-z-]+:' | grep -E -- '-(production|qa)' | sed -E 's/^ ([a-z-]*):/\1/'); doecho "[`date`] Building $app..."/opt/bin/docker-compose build $appdonefi
intermediate-build-server ?ຊ CI as a Service ্ͰΓ͔ͨͬͨ image pull -> build -> pushDocker 1.10 ͔Β Docker image ͷΈ͕มΘͬͯɺ pull ͖ͯͨ͠Πϝʔδͷʮ్த·ͰΛʯΩϟογϡͱͯ͠ ͏͜ͱ͕Ͱ͖ͳ͘ͳͬͨhttps://docs.docker.com/engine/userguide/storagedriver/imagesandcontainers/#/content-addressable-storage
intermediate-build-serverDocker 1.13 Ͱղফ༧ఆ
ECRAWS ECS ଐͷ ϑϧϚωʔδυ Docker RegistryTerraform ͰϦϙδτϦΛཧϩάΠϯ͢ΔεΫϦϓτΛ 1͓͖࣌ؒʹఆظ࣮ߦ
ECR1ϨϙδτϦ࠷େ1000Πϝʔδ80 - 90Πϝʔδ / week ϖʔεͰ૿Ճ Lambda Ͱλά͍ͯͳ͍ΠϝʔδΛఆظతʹ͓আ
ECR dtan4/ecr-gc
Ҡߦ݁Ռ
Ҡߦ݁Ռ ΠϝʔδαΠζ 3.307 GB -> 2.24 GB -> 2.43 GB Ϗϧυ࣌ؒ 48 m -> 16 m 5 sAlpine Linux Ubuntu
TODO Serverspec ॻ͍͚ͨͲ CI ͕ͳ͍ shallow clone ͰΠϝʔδΛ͞ΒʹμΠΤοτ ͱ͔ͱ͔…
Recap