Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Continuous Delivery in COOKPAD

Naoto Takai
November 21, 2013

Continuous Delivery in COOKPAD

at RubyWorld Conference 2013, on Nov. 21st, 2013.

Naoto Takai

November 21, 2013
Tweet

More Decks by Naoto Takai

Other Decks in Programming

Transcript

  1. Continuous Delivery in COOKPAD
    RubyWorld Conference 2013

    View Slide

  2. Naoto Takai
    Manager, Tech Department

    View Slide

  3. 20M UU/month
    1.5M recipes
    (October 31 2013)

    View Slide

  4. Ruby 2.0
    Rails 3.2
    (October 31 2013)

    View Slide

  5. ✓ 1,082 models
    ✓ 318 controllers
    ✓ 3,304 view templates
    ✓ 2,127 lines of routes.rb
    ✓ 3,946 assets
    (October 31 2013)

    View Slide

  6. 14,432 examples
    (October 31 2013)

    View Slide

  7. about 10 minutes
    (October 31 2013)

    View Slide

  8. How often do you deploy?

    View Slide

  9. 11+ deploys/day
    (October 31 2013)

    View Slide

  10. Build-Measure-Learn
    Build
    Measure
    Learn
    product
    data
    idea

    View Slide

  11. How we do that?

    View Slide

  12. Continuous Delivery
    “Continuous Delivery is a software
    development discipline where you build
    software in such a way that the software can
    be released to production at any time.”
    http://martinfowler.com/bliki/ContinuousDelivery.html

    View Slide

  13. Continuous Delivery (cont.)
    “You achieve continuous delivery by
    continuously integrating the software done
    by the development team, building
    executables, and running automated tests
    on those executables to detect problems.”
    http://martinfowler.com/bliki/ContinuousDelivery.html

    View Slide

  14. ✓ Development
    ✓ Source code review
    ✓ Continuous integration
    ✓ Production test
    ✓ Production
    Source Code Review
    Continuous Integration
    Production Test
    Developement
    Production
    GitHub Git Repository
    merge
    pull req
    pull
    tag
    deploy
    deploy
    Deployment Pipeline

    View Slide

  15. Development Environment
    Developer Machine
    Redis
    Shared Development MySQL
    EC2
    Ruby on Rails
    memcached
    Remote Spec Workers
    remote_spec worker
    remote_spec worker

    GitHub Enterprise
    LAN
    Tokyo Tyrant
    Development App Server
    push
    rspec
    deploy
    HipChat

    View Slide

  16. How many people use git?

    View Slide

  17.  GitHub Flow
    ✓ Anything in the master branch is deployable
    ✓ To work on something new, create a descriptively
    named branch off of master (ie: new-oauth2-scopes)
    ✓ When you need feedback or help, or you think the
    branch is ready for merging, open a pull request
    ✓ After someone else has reviewed and signed off on the
    feature, you can merge it into master
    http://scottchacon.com/2011/08/31/github- ow.html

    View Slide

  18.  GitHub Flow
    feature branch
    feature branch
    master branch
    pull request & merge

    View Slide

  19. Feature Toggle
    “The basic idea is to have a con guration le
    that de nes a bunch of toggles for various
    features you have pending. The running
    application then uses these toggles in order
    to decide whether or not to show the new
    feature.”
    http://martinfowler.com/bliki/FeatureToggle.html

    View Slide

  20. Feature Toggle
    Target?
    Original feature
    New feature
    User request
    Yes
    No

    View Slide

  21. View Slide

  22. View Slide

  23. View Slide

  24. View Slide

  25. View Slide

  26. View Slide

  27. Number of Pull Requests
    0
    100
    200
    300
    400
    500
    600
    700
    800
    2012-04
    2012-05
    2012-06
    2012-07
    2012-08
    2012-09
    2012-10
    2012-11
    2012-12
    2013-01
    2013-02
    2013-03
    2013-04
    2013-05
    2013-06
    2013-07
    2013-08
    2013-09
    2013-10
    0
    10
    20
    30
    40
    50
    60
    Developers
    Pull Requests

    View Slide

  28. Source Code Flow
    LAN
    EC2
    CI Server
    Git Repository Production Test
    HipChat
    tag
    pull
    notify
    deploy
    Development DB
    schema
    Remote Spec
    distribute
    Git Repository
    clone pusher
    service hook pull
    push
    Developer
    merge

    View Slide

  29. post '/update' do
    payload = JSON.parse(params[:payload])
    project = payload['repository']['name']
    if system("pgrep -f \"sh -c git --git-dir=#{REPOSITORY_ROOT}/#{project}\"")
    next "Found another process for #{project}\n"
    end
    cmd = "git --git-dir=#{REPOSITORY_ROOT}/#{project} remote prune origin " +
    "&& git --git-dir=#{REPOSITORY_ROOT}/#{project} fetch -q " +
    "&& git --git-dir=#{REPOSITORY_ROOT}/#{project} push -q --mirror git-repo"
    pid = Process.spawn(cmd)
    Process.detach(pid)
    "Spawn following command: #{cmd}\n"
    end
    Clone Pusher

    View Slide

  30. View Slide

  31. HipChat Noti cation

    View Slide

  32. View Slide

  33. CI Servers
    CI Master
    CI Slave
    CI Slave
    CI Slave
    CI Slave
    CI Slave
    Remote Spec Worker
    Remote Spec Worker
    Remote Spec Worker
    Remote Spec Worker
    Remote Spec Worker
    Remote Spec Worker
    Remote Spec Worker
    Remote Spec Worker
    CI Slave

    View Slide

  34. RSpec Examples
    Build Time

    View Slide

  35. HipChat Noti cation

    View Slide

  36. Source Code Flow
    LAN
    EC2
    CI Server
    Git Repository Production Test
    HipChat
    tag
    pull
    notify
    deploy
    Development DB
    schema
    Remote Spec
    distribute
    Git Repository
    clone pusher
    service hook pull
    push
    Developer
    merge

    View Slide

  37. Production Test
    ✓ Same server con guration as production
    ✓ Server speci cations
    ✓ Application server (unicorn)
    ✓ Production database
    ✓ Manual testing

    View Slide

  38. Deployment
    ✓ Latest CI passed revision
    ✓ Weekday 9:30 am - 5:00 pm (3:00pm on Friday)

    View Slide

  39. $ bundle exec cap production deploy

    View Slide

  40. Application Server
    Deploy Server
    deploy
    CI Server
    HipChat
    Blog & Wiki
    Git Repository
    Developer Machine
    SSH
    notify
    post
    pull
    assets
    LAN
    EC2
    Deployment Server

    View Slide

  41. HipChat Noti cation
    Cry Bot
    ✓ “A deploy starts.”
    https://bitbucket.org/winebarrel/zakkuri

    View Slide

  42. View Slide

  43. View Slide

  44. Rollback and Deploy Lock
    ✓ bundle exec cap production deploy:rollback
    ✓ bundle exec cap production deploy:lock

    View Slide

  45. 11+ deploys/day

    View Slide

  46. How we do that?

    View Slide

  47. The answer is “nothing special”:
    ✓ Build a deployment pipeline
    ✓ Automate a process as possible

    View Slide

  48. Continuous delivery makes:
    ✓ the deployment safer.
    ✓ nding the bug easier.
    ✓ faster hypothesis testing.

    View Slide