Scalable Deployments - How we deploy Rails app to 150+ hosts in a minute

Scalable Deployments - How we deploy Rails app to 150+ hosts in a minute

RubyKaigi edition

626ca235e8dab778c5bad6fc10e94ad8?s=128

Sorah Fukumori

September 19, 2014
Tweet

Transcript

  1. 2.

    AD ✮鼅 㕼  傈 ˋؔٝٓ؎ٝ ו׍׵ַ傈 劤䨌 㕼 

        !-*/&吳䒭⠓爡幪靼ؼؕٔؒؔؿ؍أ ! ˑֶ겗הז׷8FC؟٦ؽأ׾寸׭׵׸׋ؘُٖٖ٦ءّٝך⚥ד ꣲ歲תד넝鸞⻉׾㔳׷ثُ٦صؚٝغزٕ˒ ! ⚺⪵-*/&吳䒭⠓爡  ㉏겗⡲䧭涯ꆃ⹛暟㕦 $PPLQBE http://isucon.net/
  2. 3.

    OK

  3. 7.

    LINKS Cookpad's deployment and auto scaling Continuous Delivery in Cookpad

    https://speakerdeck.com/mirakui/cookpads-deployment-and-auto-scaling https://speakerdeck.com/takai/continuous-delivery-in-cookpad
  4. 10.

    quoted from New Oxford American Dictionary 3rd edition © 2010,

    2012 by Oxford University Press photo: https://www.flickr.com/photos/thenationalguard/4401592829 deploy |diˈploi| verb [ with obj. ] ! move (troops) into position for military
  5. 11.

    Software deployment is all of the activities that make a

    software system available for use. ! The general deployment process consists of several interrelated activities with possible transitions between them. These activities can occur at the producer side or at the consumer side or both. Because every software system is unique, the precise processes or procedures http://en.wikipedia.org/wiki/Software_deployment
  6. 13.

    We perform deployment of the following Rails app: +----------------------+-------+-------+---------+---------+-----+-------+ |

    Name | Lines | LOC | Classes | Methods | M/C | LOC/M | +----------------------+-------+-------+---------+---------+-----+-------+ | Controllers | 41964 | 33824 | 436 | 3397 | 7 | 7 | | Helpers | 13296 | 10950 | 10 | 1289 | 128 | 6 | | Models | 87626 | 69239 | 1530 | 7604 | 4 | 7 | | Mailers | 300 | 240 | 11 | 26 | 2 | 7 | | Javascripts | 38740 | 33240 | 34 | 4789 | 140 | 4 | | Libraries | 56189 | 46375 | 532 | 4371 | 8 | 8 | | Async_view specs | 247 | 212 | 0 | 0 | 0 | 0 | | Controller specs | 55098 | 45557 | 7 | 117 | 16 | 387 | | Feature specs | 36807 | 30226 | 0 | 165 | 0 | 181 | | Helper specs | 3598 | 2956 | 0 | 7 | 0 | 420 | | Lib specs | 21636 | 18095 | 27 | 124 | 4 | 143 | | Mailer specs | 306 | 251 | 0 | 0 | 0 | 0 | | Policy specs | 1594 | 1302 | 0 | 0 | 0 | 0 | | Request specs | 28698 | 24526 | 0 | 11 | 0 | 2227 | | Routing specs | 648 | 523 | 0 | 0 | 0 | 0 | | View specs | 619 | 508 | 0 | 2 | 0 | 252 | | Worker specs | 862 | 715 | 0 | 1 | 0 | 713 | +----------------------+-------+-------+---------+---------+-----+-------+ | Total | 388228 | 318739 | 2587 | 21903 | 8 | 12 | +----------------------+-------+-------+---------+---------+-----+-------+
  7. 14.

    +----------------------+-------+-------+---------+ | Name | Lines | LOC | Classes |

    +----------------------+-------+-------+---------+ | Controllers | 41964 | 33824 | 436 | | Helpers | 13296 | 10950 | 10 | | Models | 87626 | 69239 | 1530 | | Mailers | 300 | 240 | 11 | | Javascripts | 38740 | 33240 | 34 | | Libraries | 56189 | 46375 | 532 | | Controller specs | 55098 | 45557 | 7 | | Feature specs | 36807 | 30226 | 0 | | Helper specs | 3598 | 2956 | 0 | | Lib specs | 21636 | 18095 | 27 |
  8. 15.

    | Controller specs | 55098 | 45557 | 7 |

    | Feature specs | 36807 | 30226 | 0 | | Helper specs | 3598 | 2956 | 0 | | Lib specs | 21636 | 18095 | 27 | | Mailer specs | 306 | 251 | 0 | | Policy specs | 1594 | 1302 | 0 | | Request specs | 28698 | 24526 | 0 | | Routing specs | 648 | 523 | 0 | | View specs | 619 | 508 | 0 | | Worker specs | 862 | 715 | 0 | +----------------------+-------+-------+---------+ | Total | 388228 | 318739 | 2587 +----------------------+-------+-------+---------+
  9. 17.

    RULES ON DEPLOYMENT Deploy revisions which CI build passes Only

    during working time After deployment, monitor errors for an hour Rollback if error rate increase, or any trouble
  10. 18.

    Our deployment was: CI git repo developer # $ "

    % pass tag check deploy & merge ' build
  11. 21.
  12. 23.
  13. 25.

    $ tree config config ├── cutty_deploy.rb ├── deploy │ ├──

    ***.rb │ ├── production.rb │ ├── production_test.rb │ ├── production_***.rb │ ├── production_***_test.rb │ ├── (snip) │ ├── rails41.rb │ ├── ruby210.rb │ ├── staging.rb │ ├── staging_***.rb │ └── ***_test.rb ├── deploy_support │ ├── bundler_capistrano.rb │ ├── chat_notification.rb │ ├── deploy_utils.rb │ └── rsync_with_remote_cache.rb :
  14. 29.
  15. 33.

    How long time spent for deployment? CI git repo developer

    # " % ' 10 min 1..5 min 10 min ? $ &
  16. 34.

    How long time spent for deployment? CI git repo developer

    # $ " % & ' 10 min 1..5 min 10..20 min+
  17. 35.

    My team “dev-infra” aims to: Improve developers’ productivity ! Keep

    development fast Maintain & improve test environment etc
  18. 37.

    IMPROVEMENT PLANS Upgrade to Capistrano 3? ! It has better

    SSH handling, but still depends on SSH. SSH is slow.
  19. 40.

    MAMIYA use Serf for orchestration use Amazon S3 for file

    distribution (by default) compatible directory structure with Capistrano
  20. 42.

    GOSSIP PROTOCOL? A gossip protocol is a style of computer-to-

    computer communication protocol inspired by the form of gossip seen in social networks. http://en.wikipedia.org/wiki/Gossip_protocol
  21. 43.

    GOSSIP PROTOCOL: interval: 200ms, total nodes: 8, fanout: 2 =

    event node node node node node node node node e
  22. 44.

    node node node node node GOSSIP PROTOCOL: node node e

    Receives Event. node 0ms 200ms 400ms 600ms 800ms +
  23. 45.

    node node node node node GOSSIP PROTOCOL: node node e

    Receives Event. Choose nodes to gossip node 0ms 200ms 400ms 600ms 800ms +
  24. 46.

    node node node node node GOSSIP PROTOCOL: node node e

    Receives Event. Choose nodes to gossip node 0ms 200ms 400ms 600ms 800ms +
  25. 47.

    node node node node node GOSSIP PROTOCOL: node node e

    Receives Event. Choose nodes to gossip Fan out. node 0ms 200ms 400ms 600ms 800ms +
  26. 48.

    node node node node node GOSSIP PROTOCOL: node node e

    e e Receives Event. Choose nodes to gossip Fan out. node 0ms 200ms 400ms 600ms 800ms +
  27. 49.

    node node node node node GOSSIP PROTOCOL: node node e

    e e Receives Event. Choose nodes to gossip Fan out. node 0ms 200ms 400ms 600ms 800ms +
  28. 50.

    node node node node node GOSSIP PROTOCOL: node node e

    e e Receives Event. Choose nodes to gossip Fan out. node 0ms 200ms 400ms 600ms 800ms +
  29. 51.

    node node node node node GOSSIP PROTOCOL: node node e

    e e Receives Event. Choose nodes to gossip Fan out. node 0ms 200ms 400ms 600ms 800ms +
  30. 52.

    node node node node node GOSSIP PROTOCOL: node node e

    e e e e e e Receives Event. Choose nodes to gossip Fan out. node 0ms 200ms 400ms 600ms 800ms +
  31. 53.

    node node node node node GOSSIP PROTOCOL: node node e

    e e e e e e Receives Event. Choose nodes to gossip Fan out. node 0ms 200ms 400ms 600ms 800ms +
  32. 54.

    node node node node node GOSSIP PROTOCOL: node node e

    e e e e e e Receives Event. Choose nodes to gossip Fan out. node 0ms 200ms 400ms 600ms 800ms +
  33. 55.

    node node node node node GOSSIP PROTOCOL: node node e

    e e e e e e Receives Event. Choose nodes to gossip Fan out. node 0ms 200ms 400ms 600ms 800ms +
  34. 56.

    node node node node node GOSSIP PROTOCOL: node node e

    e e e e e e Receives Event. Choose nodes to gossip Fan out. node e e e e e e e e 0ms 200ms 400ms 600ms 800ms +
  35. 57.

    node node node node node GOSSIP PROTOCOL: node node e

    e e e e e e Receives Event. Choose nodes to gossip Fan out. node e e e e e e e e Drop received event if it’s known 0ms 200ms 400ms 600ms 800ms +
  36. 58.

    node node node node node GOSSIP PROTOCOL: node node e

    e e e e e e Receives Event. Choose nodes to gossip Fan out. node e Drop received event if it’s known 0ms 200ms 400ms 600ms 800ms +
  37. 59.

    node node node node node GOSSIP PROTOCOL: node node e

    e e e e e e Receives Event. Choose nodes to gossip Fan out. node e Drop received event if it’s known 0ms 200ms 400ms 600ms 800ms +
  38. 60.

    node node node node node GOSSIP PROTOCOL: node node e

    e e e e e e Receives Event. Choose nodes to gossip Fan out. node e Drop received event if it’s known 0ms 200ms 400ms 600ms 800ms +
  39. 61.

    node node node node node GOSSIP PROTOCOL: node node e

    e e e e e e Receives Event. Choose nodes to gossip Fan out. node e Drop received event if it’s known 0ms 200ms 400ms 600ms 800ms +
  40. 62.

    node node node node node GOSSIP PROTOCOL: node node e

    e e e e e e Receives Event. Choose nodes to gossip Fan out. node e Drop received event if it’s known e e 0ms 200ms 400ms 600ms 800ms +
  41. 63.

    node node node node node GOSSIP PROTOCOL: node node e

    e e e e e e Receives Event. Choose nodes to gossip Fan out. node e Drop received event if it’s known e e 0ms 200ms 400ms 600ms 800ms +
  42. 64.

    node node node node node GOSSIP PROTOCOL: node node e

    e e e e e e Receives Event. Choose nodes to gossip Fan out. node e Drop received event if it’s known e e 0ms 200ms 400ms 600ms 800ms +
  43. 65.

    node node node node node GOSSIP PROTOCOL: node node e

    e e e e e e Receives Event. Choose nodes to gossip Fan out. node e Drop received event if it’s known e e 0ms 200ms 400ms 600ms 800ms +
  44. 66.

    node node node node node GOSSIP PROTOCOL: node node e

    e e e e e e Receives Event. Choose nodes to gossip Fan out. node e Drop received event if it’s known 0ms 200ms 400ms 600ms 800ms +
  45. 67.

    node node node node node GOSSIP PROTOCOL: node node e

    e e e e e e Receives Event. Choose nodes to gossip Fan out. node e Drop received event if it’s known 0ms 200ms 400ms 600ms 800ms +
  46. 68.

    node node node node node GOSSIP PROTOCOL: node node e

    e e e e e e Receives Event. Choose nodes to gossip Fan out. node e Drop received event if it’s known 0ms 200ms 400ms 600ms 800ms + Now all nodes has the event.
  47. 73.

    Terminologies & Concept Master node has HTTP API to control

    cluster sends requests to agents (via serf) watches agents’ status
  48. 76.

    Terminologies & Concept Package is a tarball of files to

    deploy can be pushed to storage contains Deploy Script
  49. 79.

    Steps Fetch package from storage Prepare fetched package (bundle install,

    etc) Switch to prepared package (reload, graceful)
  50. 80.

    1. CI builds package when passed 2. CI pushes the

    package to storage 3. — Deployment starts — 4. Master sends “prepare” request to Agents 5. Agents fetch package, then prepare Mamiya’s Deploy flow
  51. 81.

    6. Master confirms all agents have prepared 7. Master sends

    “switch” request to Agents 8. Agent switches symlinks, then reload app process Mamiya’s Deploy flow
  52. 82.
  53. 86.

    mamiya’s deploy flow (prepare earlier) storage CI developer , build+push

    check deploy pass prepare app merge " & # . - % reload app
  54. 88.
  55. 89.

    Future Plans Better documentation (soon) Auto-deploy when joining cluster Web

    UI Better error tracking, handling Incremental Packages “master is always deployed”