Continuous Delivery in COOKPAD

at RubyWorld Conference 2013, on Nov. 21st, 2013.

Transcript

1. Continuous Delivery in COOKPAD
   RubyWorld Conference 2013
   

   View Slide

2. Naoto Takai
   Manager, Tech Department
   

   View Slide

3. 20M UU/month
   1.5M recipes
   (October 31 2013)
   

   View Slide

4. Ruby 2.0
   Rails 3.2
   (October 31 2013)
   

   View Slide

5. ✓ 1,082 models
   ✓ 318 controllers
   ✓ 3,304 view templates
   ✓ 2,127 lines of routes.rb
   ✓ 3,946 assets
   (October 31 2013)
   

   View Slide

6. 14,432 examples
   (October 31 2013)
   

   View Slide

7. about 10 minutes
   (October 31 2013)
   

   View Slide

8. How often do you deploy?
   

   View Slide

9. 11+ deploys/day
   (October 31 2013)
   

   View Slide

10. Build-Measure-Learn
    Build
    Measure
    Learn
    product
    data
    idea
    

    View Slide

11. How we do that?
    

    View Slide

12. Continuous Delivery
    “Continuous Delivery is a software
    development discipline where you build
    software in such a way that the software can
    be released to production at any time.”
    http://martinfowler.com/bliki/ContinuousDelivery.html
    

    View Slide

13. Continuous Delivery (cont.)
    “You achieve continuous delivery by
    continuously integrating the software done
    by the development team, building
    executables, and running automated tests
    on those executables to detect problems.”
    http://martinfowler.com/bliki/ContinuousDelivery.html
    

    View Slide

14. ✓ Development
    ✓ Source code review
    ✓ Continuous integration
    ✓ Production test
    ✓ Production
    Source Code Review
    Continuous Integration
    Production Test
    Developement
    Production
    GitHub Git Repository
    merge
    pull req
    pull
    tag
    deploy
    deploy
    Deployment Pipeline
    

    View Slide

15. Development Environment
    Developer Machine
    Redis
    Shared Development MySQL
    EC2
    Ruby on Rails
    memcached
    Remote Spec Workers
    remote_spec worker
    remote_spec worker
    
    GitHub Enterprise
    LAN
    Tokyo Tyrant
    Development App Server
    push
    rspec
    deploy
    HipChat
    

    View Slide

16. How many people use git?
    

    View Slide

17.  GitHub Flow
    ✓ Anything in the master branch is deployable
    ✓ To work on something new, create a descriptively
    named branch off of master (ie: new-oauth2-scopes)
    ✓ When you need feedback or help, or you think the
    branch is ready for merging, open a pull request
    ✓ After someone else has reviewed and signed off on the
    feature, you can merge it into master
    http://scottchacon.com/2011/08/31/github- ow.html
    

    View Slide

18.  GitHub Flow
    feature branch
    feature branch
    master branch
    pull request & merge
    

    View Slide

19. Feature Toggle
    “The basic idea is to have a con guration le
    that de nes a bunch of toggles for various
    features you have pending. The running
    application then uses these toggles in order
    to decide whether or not to show the new
    feature.”
    http://martinfowler.com/bliki/FeatureToggle.html
    

    View Slide

20. Feature Toggle
    Target?
    Original feature
    New feature
    User request
    Yes
    No
    

    View Slide

21. View Slide

22. View Slide

23. View Slide

24. View Slide

25. View Slide

26. View Slide

27. Number of Pull Requests
    0
    100
    200
    300
    400
    500
    600
    700
    800
    2012-04
    2012-05
    2012-06
    2012-07
    2012-08
    2012-09
    2012-10
    2012-11
    2012-12
    2013-01
    2013-02
    2013-03
    2013-04
    2013-05
    2013-06
    2013-07
    2013-08
    2013-09
    2013-10
    0
    10
    20
    30
    40
    50
    60
    Developers
    Pull Requests
    

    View Slide

28. Source Code Flow
    LAN
    EC2
    CI Server
    Git Repository Production Test
    HipChat
    tag
    pull
    notify
    deploy
    Development DB
    schema
    Remote Spec
    distribute
    Git Repository
    clone pusher
    service hook pull
    push
    Developer
    merge
    

    View Slide

29. post '/update' do
    payload = JSON.parse(params[:payload])
    project = payload['repository']['name']
    if system("pgrep -f \"sh -c git --git-dir=#{REPOSITORY_ROOT}/#{project}\"")
    next "Found another process for #{project}\n"
    end
    cmd = "git --git-dir=#{REPOSITORY_ROOT}/#{project} remote prune origin " +
    "&& git --git-dir=#{REPOSITORY_ROOT}/#{project} fetch -q " +
    "&& git --git-dir=#{REPOSITORY_ROOT}/#{project} push -q --mirror git-repo"
    pid = Process.spawn(cmd)
    Process.detach(pid)
    "Spawn following command: #{cmd}\n"
    end
    Clone Pusher
    

    View Slide

30. View Slide

31. HipChat Noti cation
    

    View Slide

32. View Slide

33. CI Servers
    CI Master
    CI Slave
    CI Slave
    CI Slave
    CI Slave
    CI Slave
    Remote Spec Worker
    Remote Spec Worker
    Remote Spec Worker
    Remote Spec Worker
    Remote Spec Worker
    Remote Spec Worker
    Remote Spec Worker
    Remote Spec Worker
    CI Slave
    

    View Slide

34. RSpec Examples
    Build Time
    

    View Slide

35. HipChat Noti cation
    

    View Slide

36. Source Code Flow
    LAN
    EC2
    CI Server
    Git Repository Production Test
    HipChat
    tag
    pull
    notify
    deploy
    Development DB
    schema
    Remote Spec
    distribute
    Git Repository
    clone pusher
    service hook pull
    push
    Developer
    merge
    

    View Slide

37. Production Test
    ✓ Same server con guration as production
    ✓ Server speci cations
    ✓ Application server (unicorn)
    ✓ Production database
    ✓ Manual testing
    

    View Slide

38. Deployment
    ✓ Latest CI passed revision
    ✓ Weekday 9:30 am - 5:00 pm (3:00pm on Friday)
    

    View Slide

39. $ bundle exec cap production deploy
    

    View Slide

40. Application Server
    Deploy Server
    deploy
    CI Server
    HipChat
    Blog & Wiki
    Git Repository
    Developer Machine
    SSH
    notify
    post
    pull
    assets
    LAN
    EC2
    Deployment Server
    

    View Slide

41. HipChat Noti cation
    Cry Bot
    ✓ “A deploy starts.”
    https://bitbucket.org/winebarrel/zakkuri
    

    View Slide

42. View Slide

43. View Slide

44. Rollback and Deploy Lock
    ✓ bundle exec cap production deploy:rollback
    ✓ bundle exec cap production deploy:lock
    

    View Slide

45. 11+ deploys/day
    

    View Slide

46. How we do that?
    

    View Slide

47. The answer is “nothing special”:
    ✓ Build a deployment pipeline
    ✓ Automate a process as possible
    

    View Slide

48. Continuous delivery makes:
    ✓ the deployment safer.
    ✓ nding the bug easier.
    ✓ faster hypothesis testing.
    

    View Slide