Wantedly から Chef を一掃した話 / #chibadan

Transcript

1. 8BOUFEMZ
   ͔Β
 $IFG
   ΛҰ૟ͨ͠࿩ 2016-10-25 αʔϏε։ൃͷݱ৔ΛޠΔʂ
 Daisuke Fujita (@dtan4) https://flic.kr/p/rU5bCU

2. Daisuke Fujita / @dtan4 Engineer, Infrastructure Team
 @ Wantedly, Inc.

   AWS, Terraform / Terraforming, Developer Productivity

3. 8BOUFEMZ
   ͔Β
 $IFG
   ΛҰ૟ͨ͠࿩ͭͭ
   %PDLFS
   पΓΛஔ͖׵͑ͨ࿩ 2016-10-25 αʔϏε։ൃͷݱ৔ΛޠΔʂ
 Daisuke Fujita (@dtan4) https://flic.kr/p/rU5bCU

4. Infrastructure at Wantedly AWS + DNSimple શαʔϏε͕ Docker ίϯςφͰՔಇ EC2

   Πϯελϯεͷ OS ͸ CoreOS
 ੲ͸ Ubuntu + Chef Capistrano Λ࢖ͬͨಠࣗͷσϓϩΠγεςϜ 
 + Heroku CLI ಉ༷ͷΠϯλʔϑΣΠεΛ࣋ͭ CLI https://speakerdeck.com/dtan4/number-abejameetup

5. Web (Rails) Docker image ͸2ஈߏ੒ ࠩ෼ͷΈͷ fetch + bundle install

   Ͱ
 σϓϩΠΛߴ଎Խ intermediate ͸ Chef + Packer ΠϝʔδαΠζ 3GB ௒ bundle install git clone wantedly/wantedly Ruby FROM Ubuntu 14.04 rake assets:precompile bundle install git checkout <branch> git fetch FROM intermediate web (Dockerfile) intermediate (Chef)

6. Web (Rails) Docker image ͸2ஈߏ੒ ࠩ෼ͷΈͷ fetch + bundle install

   Ͱ
 σϓϩΠΛߴ଎Խ intermediate ͸ Chef + Packer ΠϝʔδαΠζ 3GB ௒ bundle install git clone wantedly/wantedly Ruby FROM Ubuntu 14.04 rake assets:precompile bundle install git checkout <branch> git fetch FROM intermediate web (Dockerfile) intermediate (Chef)

7. pull
 :intermediate push
 :production pull
 :production build
 :production $ sap

   prod deploy (3:00 am) docker push
 :intermediate

8. Chef + Packer AWS Ҡߦ౰ॳ͔Β2೥Ҏ্Ҿ͖ܧ͕Ε͖ͯͨൿ఻ͷ Chef Ϩγϐ Packer Ͱ ubuntu:14.04

   Πϝʔδ্ʹϓϩϏδϣχϯά
 => Private Registry ʹ push

9. Chef + Packer ??? ઐ༻ͷ EC2 Πϯελϯε1୆ͷΈͰՔಇ͍ͯͨ͠
 ΋͸΍ਓؒͷ Macbook ͰϓϩϏδϣχϯάͰ͖ͳ͘ͳ͍ͬͯͨ


   Chef ʢͷΠϯετʔϧํ๏ʣͷਐԽʹ௥ैͦ͜͠Ͷͨ

10. Chef + Packer ??? 1Πϝʔδ࡞Δͷʹ1࣌ؒҎ্ Ͱ͖͕͋ͬͨΠϝʔδ͸ 3GB ௒ Ruby ͷόʔδϣϯΞοϓ΍

    apt ύοέʔδ௥Ճ͕໘౗
 ΠϯϑϥνʔϜҎ֎͕৮Γʹ͍͘ σϓϩΠ࣌ͷ docker build ΍֤Πϯελϯε΁ͷ഑෍͕஗͍

11. Chef + Packer => ??? ΠϝʔδαΠζΛ΋ͬͱখ͍ͨ͘͞͠… ଞͷϝϯόʔ΋৮Γ΍͍͢ߏ੒ʹ͍ͨ͠… σϓϩΠ଎͍ͨ͘͠… Dockerfile Ͱॻ͖͍ͨ…ʂ

12. Private Registry => ??? Docker Private Registry V1 (Python) ʹ͸ຊؾͰٽ͔͞Εͨ

    Docker Distribution (Go) ͸͍ͩͿϚγʹͳ͚ͬͨͲ
 Registry ίϯςφ͕ࢮ͵ڪා͸૬มΘΒͣ ϚωʔδυαʔϏεʹҠߦ͍ͨ͠…ʂ
 ECR ౦ژʹདྷͨ͠

13. 2016/10

14. 2016/10 Wantedly, Inc. ͔Β Chef ΛҰ૟

15. 2016/10 Chef + Packer -> Dockerfile + Docker Compose Ubuntu

    14.04 -> Ubuntu 16.04, Alpine Linux 3.4 Private Docker Registry -> Amazon ECR Wantedly, Inc. ͔Β Chef ΛҰ૟

16. (3:00 am) docker push
 :intermediate pull
 :intermediate push
 :production pull


    :production build
 :production $ sap prod deploy
17. None

18. Dockerfile Chef Ϩγϐ -> Dockerfile 1ຕʹू໿

19. Dockerfile αʔϏε͝ͱʹ1ຕͷ Dockerfile FROM ubuntu 16.04 / alpine:3.4 Dockerfile ϕετϓϥΫςΟε


    Λ࣮ફͯ͠μΠΤοτ

20. Dockerfile Ruby, Node.js ͷΠϯετʔϧ΋
 ಉ͡ Dockerfile ʹॻ͘ ެࣜͷ -alpine Dockerfile

    ͔Βഈआ Alpine ͷόʔδϣϯ΋
 ࣗ෼ͨͪͰ੍ޚ αʔϏεʹ߹Θͤͨݴޠόʔδϣϯ

21. Dockerfile λΠϜελϯϓΛૠೖͯ͠
 ҎԼ͸ຖճڧ੍తʹϏϧυ ιʔείʔυΛ git clone
 Access Token Λ࢖ͬͨೝূ bundle

    install -> assets:precompile ARG BUILD_DATE --build-args BUILD_DATE=$(date +%Y%m%d%H%M%S)

22. Docker Compose શΠϝʔδ Docker Compose
 ܦ༝Ͱ build & push build

    args ΛҰׅ؅ཧ
 Ruby ͷόʔδϣϯΞοϓ΋
 ͔͜͜Β awesome: extends: base build: context: awesome args: - RUBY_MAJOR=2.3 - RUBY_VERSION=2.3.0 - RUBYGEMS_VERSION=2.6.6 - BUNDLER_VERSION=1.13.1 - NODE_VERSION=5.11.1 - NPM_VERSION=3.3.12 awesome-production: extends: awesome build: args: - RAILS_ENV=production image: ${AWS_ECR_DOMAIN}/awesome:intermediate

23. Alpine Linux busybox + α ͷ௒ܰྔσΟετϦ (~ 5MB) ύοέʔδϚωʔδϟ͸ apk

    ͍͍ͨͯ͸໰୊ͳ͘ಈ͘ɻ
 ͳΜ΍͔Μ΍ apk ʹϥΠϒϥϦ͸Ұ௨ΓἧͬͯΔɻ

24. Alpine Linux ʹҠߦ͢Δ஫ҙ఺ ֎෦ίϚϯυݺͼग़͠͸஫ҙ
 `...` ͱ͔ os.Exec(...) ͱ͔
 ݺͼग़͢όΠφϦ͕ Alpine

    Linux ্ͩͱ
 ಈ͔ͳ͍Մೳੑ͕͋Δ (ex. wkhtmltopdf) 
 ඪ४ίϚϯυ (ex. timeout) ΋ busybox ͳͷͰ
 Ubuntu ͷʹൺ΂ͯύϥϝʔλগͳ͔ͬͨΓ
25. None

26. Alpine Linux ʹҠߦ͢Δ஫ҙ఺ όΠφϦΛ static link ͰϏϧυ͢Δ alpine-pkg-glibc, coreutils ΛೖΕΔ

    ֎෦ίϚϯυݺͼग़͠΍Ίͯ Pure ͳखஈʹ ͩΊͳΒૉ௚ʹ Ubuntu ΍ CentOS ͱ͍ͬͨ
 ϝδϟʔͳσΟετϦΛ࢖͏
 ద੾ʹαʔϏεΛఏڙͰ͖Δͷ͕͋͘·Ͱେલఏ

27. intermediate-build-server intermediate image Λఆظతʹ build ͢Δαʔό CoreOS on EC2 +

    IAM Role ຖ೔ਂ໷ʹશ intermediate image Λ 
 docker-compose build -> docker-compose push

28. intermediate-build-server #!/bin/bash set -eu metadata=$(curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/intermediate-build-server/) export AWS_ACCESS_KEY_ID=$(echo $metadata

    | jq -r .AccessKeyId) export AWS_SECRET_ACCESS_KEY=$(echo $metadata | jq -r .SecretAccessKey) export AWS_SECURITY_TOKEN=$(echo $metadata | jq -r .Token) document=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document) account_id=$(echo $document | jq -r .accountId) region=$(echo $document | jq -r .region) export AWS_ECR_DOMAIN=$account_id.dkr.ecr.$region.amazonaws.com export BUILD_DATE=$(date +%Y%m%d%H%M%S) cd intermediate-dockerfiles if [[ $# -eq 1 ]]; then /opt/bin/docker-compose build $1 else for app in $(cat docker-compose.yml | grep -E '^ [a-z-]+:' | grep -E -- '-(production|qa)' | sed -E 's/^ ([a-z-]*):/\1/'); do echo "[`date`] Building $app..." /opt/bin/docker-compose build $app done fi

29. intermediate-build-server ? ຊ౰͸ CI as a Service ্Ͱ΍Γ͔ͨͬͨ
 image pull

    -> build -> push Docker 1.10 ͔Β͸ Docker image ͷ࢓૊Έ͕มΘͬͯɺ
 pull ͖ͯͨ͠Πϝʔδͷʮ్த·ͰΛʯΩϟογϡͱͯ͠
 ࢖͏͜ͱ͕Ͱ͖ͳ͘ͳͬͨ https://docs.docker.com/engine/userguide/storagedriver/ imagesandcontainers/#/content-addressable-storage
30. None

31. intermediate-build-server Docker 1.13 Ͱղফ༧ఆ

32. ECR AWS ECS ෇ଐͷ
 ϑϧϚωʔδυ Docker Registry Terraform ͰϦϙδτϦΛ؅ཧ ϩάΠϯ͢ΔεΫϦϓτΛ


    1͓͖࣌ؒʹఆظ࣮ߦ

33. ECR 1ϨϙδτϦ࠷େ1000Πϝʔδ 80 - 90Πϝʔδ / week ϖʔεͰ૿Ճ Lambda Ͱλά෇͍ͯͳ͍ΠϝʔδΛఆظతʹ͓૟আ

34. ECR dtan4/ecr-gc

35. Ҡߦ݁Ռ

36. Ҡߦ݁Ռ ΠϝʔδαΠζ 3.307 GB -> 2.24 GB -> 2.43 GB

    Ϗϧυ࣌ؒ 48 m -> 16 m 5 s Alpine Linux Ubuntu

37. TODO Serverspec ॻ͍͚ͨͲ CI ͕ͳ͍ shallow clone ͰΠϝʔδΛ͞ΒʹμΠΤοτ ͱ͔ͱ͔…

38. Recap

39. None