Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Wantedly から Chef を一掃した話 / #chibadan

Wantedly から Chef を一掃した話 / #chibadan

サービス開発の現場を語る! #chibadan の発表資料です
http://chibadan.connpass.com/event/41625/

Daisuke Fujita

October 25, 2016
Tweet

More Decks by Daisuke Fujita

Other Decks in Programming

Transcript

  1. 8BOUFEMZ͔Β

    $IFGΛҰ૟ͨ͠࿩
    2016-10-25
    αʔϏε։ൃͷݱ৔ΛޠΔʂ

    Daisuke Fujita (@dtan4)
    https://flic.kr/p/rU5bCU

    View Slide

  2. Daisuke Fujita / @dtan4
    Engineer, Infrastructure Team

    @ Wantedly, Inc.
    AWS, Terraform / Terraforming, Developer Productivity

    View Slide

  3. 8BOUFEMZ͔Β

    $IFGΛҰ૟ͨ͠࿩ͭͭ
    %PDLFSपΓΛஔ͖׵͑ͨ࿩
    2016-10-25
    αʔϏε։ൃͷݱ৔ΛޠΔʂ

    Daisuke Fujita (@dtan4)
    https://flic.kr/p/rU5bCU

    View Slide

  4. Infrastructure at Wantedly
    AWS + DNSimple

    શαʔϏε͕ Docker ίϯςφͰՔಇ

    EC2 Πϯελϯεͷ OS ͸ CoreOS

    ੲ͸ Ubuntu + Chef
    Capistrano Λ࢖ͬͨಠࣗͷσϓϩΠγεςϜ 

    + Heroku CLI ಉ༷ͷΠϯλʔϑΣΠεΛ࣋ͭ CLI
    https://speakerdeck.com/dtan4/number-abejameetup

    View Slide

  5. Web (Rails)
    Docker image ͸2ஈߏ੒

    ࠩ෼ͷΈͷ fetch + bundle install Ͱ

    σϓϩΠΛߴ଎Խ

    intermediate ͸ Chef + Packer

    ΠϝʔδαΠζ 3GB ௒
    bundle install
    git clone wantedly/wantedly
    Ruby
    FROM Ubuntu 14.04
    rake assets:precompile
    bundle install
    git checkout
    git fetch
    FROM intermediate
    web (Dockerfile)
    intermediate (Chef)

    View Slide

  6. Web (Rails)
    Docker image ͸2ஈߏ੒

    ࠩ෼ͷΈͷ fetch + bundle install Ͱ

    σϓϩΠΛߴ଎Խ

    intermediate ͸ Chef + Packer

    ΠϝʔδαΠζ 3GB ௒
    bundle install
    git clone wantedly/wantedly
    Ruby
    FROM Ubuntu 14.04
    rake assets:precompile
    bundle install
    git checkout
    git fetch
    FROM intermediate
    web (Dockerfile)
    intermediate (Chef)

    View Slide

  7. pull

    :intermediate
    push

    :production
    pull

    :production
    build

    :production
    $ sap prod deploy
    (3:00 am)
    docker push

    :intermediate

    View Slide

  8. Chef + Packer
    AWS Ҡߦ౰ॳ͔Β2೥Ҏ্Ҿ͖ܧ͕Ε͖ͯͨൿ఻ͷ Chef Ϩγϐ

    Packer Ͱ ubuntu:14.04 Πϝʔδ্ʹϓϩϏδϣχϯά

    => Private Registry ʹ push

    View Slide

  9. Chef + Packer ???
    ઐ༻ͷ EC2 Πϯελϯε1୆ͷΈͰՔಇ͍ͯͨ͠

    ΋͸΍ਓؒͷ Macbook ͰϓϩϏδϣχϯάͰ͖ͳ͘ͳ͍ͬͯͨ

    Chef ʢͷΠϯετʔϧํ๏ʣͷਐԽʹ௥ैͦ͜͠Ͷͨ

    View Slide

  10. Chef + Packer ???
    1Πϝʔδ࡞Δͷʹ1࣌ؒҎ্ Ͱ͖͕͋ͬͨΠϝʔδ͸ 3GB ௒

    Ruby ͷόʔδϣϯΞοϓ΍ apt ύοέʔδ௥Ճ͕໘౗

    ΠϯϑϥνʔϜҎ֎͕৮Γʹ͍͘

    σϓϩΠ࣌ͷ docker build ΍֤Πϯελϯε΁ͷ഑෍͕஗͍

    View Slide

  11. Chef + Packer => ???
    ΠϝʔδαΠζΛ΋ͬͱখ͍ͨ͘͞͠…

    ଞͷϝϯόʔ΋৮Γ΍͍͢ߏ੒ʹ͍ͨ͠…

    σϓϩΠ଎͍ͨ͘͠…

    Dockerfile Ͱॻ͖͍ͨ…ʂ

    View Slide

  12. Private Registry => ???
    Docker Private Registry V1 (Python) ʹ͸ຊؾͰٽ͔͞Εͨ

    Docker Distribution (Go) ͸͍ͩͿϚγʹͳ͚ͬͨͲ

    Registry ίϯςφ͕ࢮ͵ڪා͸૬มΘΒͣ

    ϚωʔδυαʔϏεʹҠߦ͍ͨ͠…ʂ

    ECR ౦ژʹདྷͨ͠

    View Slide

  13. 2016/10

    View Slide

  14. 2016/10
    Wantedly, Inc. ͔Β Chef ΛҰ૟

    View Slide

  15. 2016/10
    Chef + Packer -> Dockerfile + Docker Compose

    Ubuntu 14.04 -> Ubuntu 16.04, Alpine Linux 3.4
    Private Docker Registry -> Amazon ECR
    Wantedly, Inc. ͔Β Chef ΛҰ૟

    View Slide

  16. (3:00 am)
    docker push

    :intermediate
    pull

    :intermediate
    push

    :production
    pull

    :production
    build

    :production
    $ sap prod deploy

    View Slide

  17. View Slide

  18. Dockerfile
    Chef Ϩγϐ -> Dockerfile 1ຕʹू໿

    View Slide

  19. Dockerfile
    αʔϏε͝ͱʹ1ຕͷ Dockerfile

    FROM ubuntu 16.04 / alpine:3.4
    Dockerfile ϕετϓϥΫςΟε

    Λ࣮ફͯ͠μΠΤοτ

    View Slide

  20. Dockerfile
    Ruby, Node.js ͷΠϯετʔϧ΋

    ಉ͡ Dockerfile ʹॻ͘

    ެࣜͷ -alpine Dockerfile ͔Βഈआ

    Alpine ͷόʔδϣϯ΋

    ࣗ෼ͨͪͰ੍ޚ

    αʔϏεʹ߹Θͤͨݴޠόʔδϣϯ

    View Slide

  21. Dockerfile
    λΠϜελϯϓΛૠೖͯ͠

    ҎԼ͸ຖճڧ੍తʹϏϧυ

    ιʔείʔυΛ git clone

    Access Token Λ࢖ͬͨೝূ

    bundle install -> assets:precompile
    ARG BUILD_DATE
    --build-args BUILD_DATE=$(date +%Y%m%d%H%M%S)

    View Slide

  22. Docker Compose
    શΠϝʔδ Docker Compose

    ܦ༝Ͱ build & push

    build args ΛҰׅ؅ཧ

    Ruby ͷόʔδϣϯΞοϓ΋

    ͔͜͜Β
    awesome:
    extends: base
    build:
    context: awesome
    args:
    - RUBY_MAJOR=2.3
    - RUBY_VERSION=2.3.0
    - RUBYGEMS_VERSION=2.6.6
    - BUNDLER_VERSION=1.13.1
    - NODE_VERSION=5.11.1
    - NPM_VERSION=3.3.12
    awesome-production:
    extends: awesome
    build:
    args:
    - RAILS_ENV=production
    image: ${AWS_ECR_DOMAIN}/awesome:intermediate

    View Slide

  23. Alpine Linux
    busybox + α ͷ௒ܰྔσΟετϦ (~ 5MB)

    ύοέʔδϚωʔδϟ͸ apk

    ͍͍ͨͯ͸໰୊ͳ͘ಈ͘ɻ

    ͳΜ΍͔Μ΍ apk ʹϥΠϒϥϦ͸Ұ௨ΓἧͬͯΔɻ

    View Slide

  24. Alpine Linux ʹҠߦ͢Δ஫ҙ఺
    ֎෦ίϚϯυݺͼग़͠͸஫ҙ

    `...` ͱ͔ os.Exec(...) ͱ͔

    ݺͼग़͢όΠφϦ͕ Alpine Linux ্ͩͱ

    ಈ͔ͳ͍Մೳੑ͕͋Δ (ex. wkhtmltopdf)


    ඪ४ίϚϯυ (ex. timeout) ΋ busybox ͳͷͰ

    Ubuntu ͷʹൺ΂ͯύϥϝʔλগͳ͔ͬͨΓ

    View Slide

  25. View Slide

  26. Alpine Linux ʹҠߦ͢Δ஫ҙ఺
    όΠφϦΛ static link ͰϏϧυ͢Δ

    alpine-pkg-glibc, coreutils ΛೖΕΔ

    ֎෦ίϚϯυݺͼग़͠΍Ίͯ Pure ͳखஈʹ

    ͩΊͳΒૉ௚ʹ Ubuntu ΍ CentOS ͱ͍ͬͨ

    ϝδϟʔͳσΟετϦΛ࢖͏

    ద੾ʹαʔϏεΛఏڙͰ͖Δͷ͕͋͘·Ͱେલఏ

    View Slide

  27. intermediate-build-server
    intermediate image Λఆظతʹ build ͢Δαʔό

    CoreOS on EC2 + IAM Role

    ຖ೔ਂ໷ʹશ intermediate image Λ 

    docker-compose build -> docker-compose push

    View Slide

  28. intermediate-build-server
    #!/bin/bash
    set -eu
    metadata=$(curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/intermediate-build-server/)
    export AWS_ACCESS_KEY_ID=$(echo $metadata | jq -r .AccessKeyId)
    export AWS_SECRET_ACCESS_KEY=$(echo $metadata | jq -r .SecretAccessKey)
    export AWS_SECURITY_TOKEN=$(echo $metadata | jq -r .Token)
    document=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document)
    account_id=$(echo $document | jq -r .accountId)
    region=$(echo $document | jq -r .region)
    export AWS_ECR_DOMAIN=$account_id.dkr.ecr.$region.amazonaws.com
    export BUILD_DATE=$(date +%Y%m%d%H%M%S)
    cd intermediate-dockerfiles
    if [[ $# -eq 1 ]]; then
    /opt/bin/docker-compose build $1
    else
    for app in $(cat docker-compose.yml | grep -E '^ [a-z-]+:' | grep -E -- '-(production|qa)' | sed -E 's/^ ([a-z-]*):/\1/'); do
    echo "[`date`] Building $app..."
    /opt/bin/docker-compose build $app
    done
    fi

    View Slide

  29. intermediate-build-server ?
    ຊ౰͸ CI as a Service ্Ͱ΍Γ͔ͨͬͨ

    image pull -> build -> push

    Docker 1.10 ͔Β͸ Docker image ͷ࢓૊Έ͕มΘͬͯɺ

    pull ͖ͯͨ͠Πϝʔδͷʮ్த·ͰΛʯΩϟογϡͱͯ͠

    ࢖͏͜ͱ͕Ͱ͖ͳ͘ͳͬͨ

    https://docs.docker.com/engine/userguide/storagedriver/
    imagesandcontainers/#/content-addressable-storage

    View Slide

  30. View Slide

  31. intermediate-build-server
    Docker 1.13 Ͱղফ༧ఆ

    View Slide

  32. ECR
    AWS ECS ෇ଐͷ

    ϑϧϚωʔδυ Docker Registry

    Terraform ͰϦϙδτϦΛ؅ཧ

    ϩάΠϯ͢ΔεΫϦϓτΛ

    1͓͖࣌ؒʹఆظ࣮ߦ

    View Slide

  33. ECR
    1ϨϙδτϦ࠷େ1000Πϝʔδ

    80 - 90Πϝʔδ / week ϖʔεͰ૿Ճ

    Lambda Ͱλά෇͍ͯͳ͍ΠϝʔδΛఆظతʹ͓૟আ

    View Slide

  34. ECR dtan4/ecr-gc

    View Slide

  35. Ҡߦ݁Ռ

    View Slide

  36. Ҡߦ݁Ռ
    ΠϝʔδαΠζ 3.307 GB -> 2.24 GB -> 2.43 GB

    Ϗϧυ࣌ؒ 48 m -> 16 m 5 s
    Alpine Linux Ubuntu

    View Slide

  37. TODO
    Serverspec ॻ͍͚ͨͲ CI ͕ͳ͍

    shallow clone ͰΠϝʔδΛ͞ΒʹμΠΤοτ

    ͱ͔ͱ͔…

    View Slide

  38. Recap

    View Slide

  39. View Slide