Wantedly から Chef を一掃した話 / #chibadan

サービス開発の現場を語る！ #chibadan の発表資料です
http://chibadan.connpass.com/event/41625/

Transcript

1. 8BOUFEMZ͔Β

   $IFGΛҰ૟ͨ͠࿩
   2016-10-25
   αʔϏε։ൃͷݱ৔ΛޠΔʂ

   Daisuke Fujita (@dtan4)
   https://flic.kr/p/rU5bCU
   

   View Slide

2. Daisuke Fujita / @dtan4
   Engineer, Infrastructure Team

   @ Wantedly, Inc.
   AWS, Terraform / Terraforming, Developer Productivity
   

   View Slide

3. 8BOUFEMZ͔Β

   $IFGΛҰ૟ͨ͠࿩ͭͭ
   %PDLFSपΓΛஔ͖׵͑ͨ࿩
   2016-10-25
   αʔϏε։ൃͷݱ৔ΛޠΔʂ

   Daisuke Fujita (@dtan4)
   https://flic.kr/p/rU5bCU
   

   View Slide

4. Infrastructure at Wantedly
   AWS + DNSimple
   
   શαʔϏε͕ Docker ίϯςφͰՔಇ
   
   EC2 Πϯελϯεͷ OS ͸ CoreOS

   ੲ͸ Ubuntu + Chef
   Capistrano Λ࢖ͬͨಠࣗͷσϓϩΠγεςϜ 

   + Heroku CLI ಉ༷ͷΠϯλʔϑΣΠεΛ࣋ͭ CLI
   https://speakerdeck.com/dtan4/number-abejameetup
   

   View Slide

5. Web (Rails)
   Docker image ͸2ஈߏ੒
   
   ࠩ෼ͷΈͷ fetch + bundle install Ͱ

   σϓϩΠΛߴ଎Խ
   
   intermediate ͸ Chef + Packer
   
   ΠϝʔδαΠζ 3GB ௒
   bundle install
   git clone wantedly/wantedly
   Ruby
   FROM Ubuntu 14.04
   rake assets:precompile
   bundle install
   git checkout
   git fetch
   FROM intermediate
   web (Dockerfile)
   intermediate (Chef)
   

   View Slide

6. Web (Rails)
   Docker image ͸2ஈߏ੒
   
   ࠩ෼ͷΈͷ fetch + bundle install Ͱ

   σϓϩΠΛߴ଎Խ
   
   intermediate ͸ Chef + Packer
   
   ΠϝʔδαΠζ 3GB ௒
   bundle install
   git clone wantedly/wantedly
   Ruby
   FROM Ubuntu 14.04
   rake assets:precompile
   bundle install
   git checkout
   git fetch
   FROM intermediate
   web (Dockerfile)
   intermediate (Chef)
   

   View Slide

7. pull

   :intermediate
   push

   :production
   pull

   :production
   build

   :production
   $ sap prod deploy
   (3:00 am)
   docker push

   :intermediate
   

   View Slide

8. Chef + Packer
   AWS Ҡߦ౰ॳ͔Β2೥Ҏ্Ҿ͖ܧ͕Ε͖ͯͨൿ఻ͷ Chef Ϩγϐ
   
   Packer Ͱ ubuntu:14.04 Πϝʔδ্ʹϓϩϏδϣχϯά

   => Private Registry ʹ push
   

   View Slide

9. Chef + Packer ???
   ઐ༻ͷ EC2 Πϯελϯε1୆ͷΈͰՔಇ͍ͯͨ͠

   ΋͸΍ਓؒͷ Macbook ͰϓϩϏδϣχϯάͰ͖ͳ͘ͳ͍ͬͯͨ

   Chef ʢͷΠϯετʔϧํ๏ʣͷਐԽʹ௥ैͦ͜͠Ͷͨ
   

   View Slide

10. Chef + Packer ???
    1Πϝʔδ࡞Δͷʹ1࣌ؒҎ্ Ͱ͖͕͋ͬͨΠϝʔδ͸ 3GB ௒
    
    Ruby ͷόʔδϣϯΞοϓ΍ apt ύοέʔδ௥Ճ͕໘౗

    ΠϯϑϥνʔϜҎ֎͕৮Γʹ͍͘
    
    σϓϩΠ࣌ͷ docker build ΍֤Πϯελϯε΁ͷ഑෍͕஗͍
    

    View Slide

11. Chef + Packer => ???
    ΠϝʔδαΠζΛ΋ͬͱখ͍ͨ͘͞͠…
    
    ଞͷϝϯόʔ΋৮Γ΍͍͢ߏ੒ʹ͍ͨ͠…
    
    σϓϩΠ଎͍ͨ͘͠…
    
    Dockerfile Ͱॻ͖͍ͨ…ʂ
    

    View Slide

12. Private Registry => ???
    Docker Private Registry V1 (Python) ʹ͸ຊؾͰٽ͔͞Εͨ
    
    Docker Distribution (Go) ͸͍ͩͿϚγʹͳ͚ͬͨͲ

    Registry ίϯςφ͕ࢮ͵ڪා͸૬มΘΒͣ
    
    ϚωʔδυαʔϏεʹҠߦ͍ͨ͠…ʂ

    ECR ౦ژʹདྷͨ͠
    

    View Slide

13. 2016/10
    

    View Slide

14. 2016/10
    Wantedly, Inc. ͔Β Chef ΛҰ૟
    

    View Slide

15. 2016/10
    Chef + Packer -> Dockerfile + Docker Compose
    
    Ubuntu 14.04 -> Ubuntu 16.04, Alpine Linux 3.4
    Private Docker Registry -> Amazon ECR
    Wantedly, Inc. ͔Β Chef ΛҰ૟
    

    View Slide

16. (3:00 am)
    docker push

    :intermediate
    pull

    :intermediate
    push

    :production
    pull

    :production
    build

    :production
    $ sap prod deploy
    

    View Slide

17. View Slide

18. Dockerfile
    Chef Ϩγϐ -> Dockerfile 1ຕʹू໿
    
    

    View Slide

19. Dockerfile
    αʔϏε͝ͱʹ1ຕͷ Dockerfile
    
    FROM ubuntu 16.04 / alpine:3.4
    Dockerfile ϕετϓϥΫςΟε

    Λ࣮ફͯ͠μΠΤοτ
    

    View Slide

20. Dockerfile
    Ruby, Node.js ͷΠϯετʔϧ΋

    ಉ͡ Dockerfile ʹॻ͘
    
    ެࣜͷ -alpine Dockerfile ͔Βഈआ
    
    Alpine ͷόʔδϣϯ΋

    ࣗ෼ͨͪͰ੍ޚ
    
    αʔϏεʹ߹Θͤͨݴޠόʔδϣϯ
    

    View Slide

21. Dockerfile
    λΠϜελϯϓΛૠೖͯ͠

    ҎԼ͸ຖճڧ੍తʹϏϧυ
    
    ιʔείʔυΛ git clone

    Access Token Λ࢖ͬͨೝূ
    
    bundle install -> assets:precompile
    ARG BUILD_DATE
    --build-args BUILD_DATE=$(date +%Y%m%d%H%M%S)
    

    View Slide

22. Docker Compose
    શΠϝʔδ Docker Compose

    ܦ༝Ͱ build & push
    
    build args ΛҰׅ؅ཧ

    Ruby ͷόʔδϣϯΞοϓ΋

    ͔͜͜Β
    awesome:
    extends: base
    build:
    context: awesome
    args:
    - RUBY_MAJOR=2.3
    - RUBY_VERSION=2.3.0
    - RUBYGEMS_VERSION=2.6.6
    - BUNDLER_VERSION=1.13.1
    - NODE_VERSION=5.11.1
    - NPM_VERSION=3.3.12
    awesome-production:
    extends: awesome
    build:
    args:
    - RAILS_ENV=production
    image: ${AWS_ECR_DOMAIN}/awesome:intermediate
    

    View Slide

23. Alpine Linux
    busybox + α ͷ௒ܰྔσΟετϦ (~ 5MB)
    
    ύοέʔδϚωʔδϟ͸ apk
    
    ͍͍ͨͯ͸໰୊ͳ͘ಈ͘ɻ

    ͳΜ΍͔Μ΍ apk ʹϥΠϒϥϦ͸Ұ௨ΓἧͬͯΔɻ
    

    View Slide

24. Alpine Linux ʹҠߦ͢Δ஫ҙ఺
    ֎෦ίϚϯυݺͼग़͠͸஫ҙ

    `...` ͱ͔ os.Exec(...) ͱ͔

    ݺͼग़͢όΠφϦ͕ Alpine Linux ্ͩͱ

    ಈ͔ͳ͍Մೳੑ͕͋Δ (ex. wkhtmltopdf)
    
    

    ඪ४ίϚϯυ (ex. timeout) ΋ busybox ͳͷͰ

    Ubuntu ͷʹൺ΂ͯύϥϝʔλগͳ͔ͬͨΓ
    

    View Slide

25. View Slide

26. Alpine Linux ʹҠߦ͢Δ஫ҙ఺
    όΠφϦΛ static link ͰϏϧυ͢Δ
    
    alpine-pkg-glibc, coreutils ΛೖΕΔ
    
    ֎෦ίϚϯυݺͼग़͠΍Ίͯ Pure ͳखஈʹ
    
    ͩΊͳΒૉ௚ʹ Ubuntu ΍ CentOS ͱ͍ͬͨ

    ϝδϟʔͳσΟετϦΛ࢖͏

    ద੾ʹαʔϏεΛఏڙͰ͖Δͷ͕͋͘·Ͱେલఏ
    

    View Slide

27. intermediate-build-server
    intermediate image Λఆظతʹ build ͢Δαʔό
    
    CoreOS on EC2 + IAM Role
    
    ຖ೔ਂ໷ʹશ intermediate image Λ 

    docker-compose build -> docker-compose push
    

    View Slide

28. intermediate-build-server
    #!/bin/bash
    set -eu
    metadata=$(curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/intermediate-build-server/)
    export AWS_ACCESS_KEY_ID=$(echo $metadata | jq -r .AccessKeyId)
    export AWS_SECRET_ACCESS_KEY=$(echo $metadata | jq -r .SecretAccessKey)
    export AWS_SECURITY_TOKEN=$(echo $metadata | jq -r .Token)
    document=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document)
    account_id=$(echo $document | jq -r .accountId)
    region=$(echo $document | jq -r .region)
    export AWS_ECR_DOMAIN=$account_id.dkr.ecr.$region.amazonaws.com
    export BUILD_DATE=$(date +%Y%m%d%H%M%S)
    cd intermediate-dockerfiles
    if [[ $# -eq 1 ]]; then
    /opt/bin/docker-compose build $1
    else
    for app in $(cat docker-compose.yml | grep -E '^ [a-z-]+:' | grep -E -- '-(production|qa)' | sed -E 's/^ ([a-z-]*):/\1/'); do
    echo "[`date`] Building $app..."
    /opt/bin/docker-compose build $app
    done
    fi
    

    View Slide

29. intermediate-build-server ?
    ຊ౰͸ CI as a Service ্Ͱ΍Γ͔ͨͬͨ

    image pull -> build -> push
    
    Docker 1.10 ͔Β͸ Docker image ͷ࢓૊Έ͕มΘͬͯɺ

    pull ͖ͯͨ͠Πϝʔδͷʮ్த·ͰΛʯΩϟογϡͱͯ͠

    ࢖͏͜ͱ͕Ͱ͖ͳ͘ͳͬͨ
    
    https://docs.docker.com/engine/userguide/storagedriver/
    imagesandcontainers/#/content-addressable-storage
    

    View Slide

30. View Slide

31. intermediate-build-server
    Docker 1.13 Ͱղফ༧ఆ
    

    View Slide

32. ECR
    AWS ECS ෇ଐͷ

    ϑϧϚωʔδυ Docker Registry
    
    Terraform ͰϦϙδτϦΛ؅ཧ
    
    ϩάΠϯ͢ΔεΫϦϓτΛ

    1͓͖࣌ؒʹఆظ࣮ߦ
    

    View Slide

33. ECR
    1ϨϙδτϦ࠷େ1000Πϝʔδ
    
    80 - 90Πϝʔδ / week ϖʔεͰ૿Ճ
    
    Lambda Ͱλά෇͍ͯͳ͍ΠϝʔδΛఆظతʹ͓૟আ
    

    View Slide

34. ECR dtan4/ecr-gc
    

    View Slide

35. Ҡߦ݁Ռ
    

    View Slide

36. Ҡߦ݁Ռ
    ΠϝʔδαΠζ 3.307 GB -> 2.24 GB -> 2.43 GB
    
    Ϗϧυ࣌ؒ 48 m -> 16 m 5 s
    Alpine Linux Ubuntu
    

    View Slide

37. TODO
    Serverspec ॻ͍͚ͨͲ CI ͕ͳ͍
    
    shallow clone ͰΠϝʔδΛ͞ΒʹμΠΤοτ
    
    ͱ͔ͱ͔…
    

    View Slide

38. Recap
    

    View Slide

39. View Slide