Scalable Deployments - How we deploy Rails app to 150+ hosts in a minute

Scalable Deployments
How we deploy
Rails app to
150+ hosts in a minute

View Slide

AD
✮鼅㕼
傈
ˋؔٝٓ؎ٝ ו׍׵ַ傈
劤䨌㕼
!-*/&吳䒭⠓爡幪靼ؼؕٔؒؔؿ؍أ
!
ˑֶ겗הז׷8FC؟٦ؽأ׾寸׭׵׸׋ؘُٖٖ٦ءّٝך⚥ד
ꣲ歲תד넝鸞⻉׾㔳׷ثُ٦صؚٝغزٕ˒
!
⚺⪵-*/&吳䒭⠓爡㉏겗⡲䧭涯ꆃ⹛暟㕦 $PPLQBE

http://isucon.net/

View Slide

OK

View Slide

Scalable Deployments
How we deploy
Rails app to
150+ hosts in a minute

View Slide

TODAY I TALK ABOUT
How Cookpad performs deployments

View Slide

TOPICS NOT INCLUDED
Rails
Continuous Delivery
App servers’ auto scaling

View Slide

LINKS
Cookpad's deployment and auto scaling
Continuous Delivery in Cookpad
https://speakerdeck.com/mirakui/cookpads-deployment-and-auto-scaling
https://speakerdeck.com/takai/continuous-delivery-in-cookpad

View Slide

Cookpad Inc.
Dev-Infra group (Ꟛ涪㛇湍)
Ruby committer
׉׵כ TPSBIFS

Shota Fukumori
@sora_h
! sorah

View Slide

DEPLOYMENTS
SO,

View Slide

quoted from New Oxford American Dictionary 3rd edition © 2010, 2012 by Oxford University Press
photo: https://www.flickr.com/photos/thenationalguard/4401592829
deploy |diˈploi|
verb [ with obj. ]
!
move (troops) into position for military

View Slide

Software deployment is all of the activities
that make a software system available for use.
!
The general deployment process consists of
several interrelated activities with possible
transitions between them. These activities can
occur at the producer side or at the consumer
side or both. Because every software system is
unique, the precise processes or procedures
http://en.wikipedia.org/wiki/Software_deployment

View Slide

DEPLOYMENTS

View Slide

We perform deployment of the following Rails app:
+----------------------+-------+-------+---------+---------+-----+-------+
| Name | Lines | LOC | Classes | Methods | M/C | LOC/M |
+----------------------+-------+-------+---------+---------+-----+-------+
| Controllers | 41964 | 33824 | 436 | 3397 | 7 | 7 |
| Helpers | 13296 | 10950 | 10 | 1289 | 128 | 6 |
| Models | 87626 | 69239 | 1530 | 7604 | 4 | 7 |
| Mailers | 300 | 240 | 11 | 26 | 2 | 7 |
| Javascripts | 38740 | 33240 | 34 | 4789 | 140 | 4 |
| Libraries | 56189 | 46375 | 532 | 4371 | 8 | 8 |
| Async_view specs | 247 | 212 | 0 | 0 | 0 | 0 |
| Controller specs | 55098 | 45557 | 7 | 117 | 16 | 387 |
| Feature specs | 36807 | 30226 | 0 | 165 | 0 | 181 |
| Helper specs | 3598 | 2956 | 0 | 7 | 0 | 420 |
| Lib specs | 21636 | 18095 | 27 | 124 | 4 | 143 |
| Mailer specs | 306 | 251 | 0 | 0 | 0 | 0 |
| Policy specs | 1594 | 1302 | 0 | 0 | 0 | 0 |
| Request specs | 28698 | 24526 | 0 | 11 | 0 | 2227 |
| Routing specs | 648 | 523 | 0 | 0 | 0 | 0 |
| View specs | 619 | 508 | 0 | 2 | 0 | 252 |
| Worker specs | 862 | 715 | 0 | 1 | 0 | 713 |
+----------------------+-------+-------+---------+---------+-----+-------+
| Total | 388228 | 318739 | 2587 | 21903 | 8 | 12 |
+----------------------+-------+-------+---------+---------+-----+-------+

View Slide

+----------------------+-------+-------+---------+
| Name | Lines | LOC | Classes |
+----------------------+-------+-------+---------+
| Controllers | 41964 | 33824 | 436 |
| Helpers | 13296 | 10950 | 10 |
| Models | 87626 | 69239 | 1530 |
| Mailers | 300 | 240 | 11 |
| Javascripts | 38740 | 33240 | 34 |
| Libraries | 56189 | 46375 | 532 |
| Controller specs | 55098 | 45557 | 7 |
| Feature specs | 36807 | 30226 | 0 |
| Helper specs | 3598 | 2956 | 0 |
| Lib specs | 21636 | 18095 | 27 |

View Slide

| Controller specs | 55098 | 45557 | 7 |
| Feature specs | 36807 | 30226 | 0 |
| Helper specs | 3598 | 2956 | 0 |
| Lib specs | 21636 | 18095 | 27 |
| Mailer specs | 306 | 251 | 0 |
| Policy specs | 1594 | 1302 | 0 |
| Request specs | 28698 | 24526 | 0 |
| Routing specs | 648 | 523 | 0 |
| View specs | 619 | 508 | 0 |
| Worker specs | 862 | 715 | 0 |
+----------------------+-------+-------+---------+
| Total | 388228 | 318739 | 2587
+----------------------+-------+-------+---------+

View Slide

140 servers
10 times / day
!
"
(peak)

View Slide

RULES ON DEPLOYMENT
Deploy revisions which CI build passes
Only during working time
After deployment, monitor errors for an hour
Rollback if error rate increase, or any trouble

View Slide

Our deployment was:
CI
git repo
developer
#
$
"
%
pass
tag
check deploy
&
merge
'
build

View Slide

Our deployment was:
App
App
App
:
Deploy Server
ssh+rsync
capistrano 2
(
)

View Slide

Deploy via Chat
# $
(
check
"
deploy!

View Slide

How long time spent for deployment?
CI
git repo
developer
#
$
"
%
&
'
10 min
1..5 min 10 min

View Slide

15…20 min

View Slide

PROBLEMS

View Slide

PROBLEMS
Capistrano 2
with complicated super historical
deploy script
!
Seemed time to re-new

View Slide

$ tree config
config
├── cutty_deploy.rb
├── deploy
│ ├── ***.rb
│ ├── production.rb
│ ├── production_test.rb
│ ├── production_***.rb
│ ├── production_***_test.rb
│ ├── (snip)
│ ├── rails41.rb
│ ├── ruby210.rb
│ ├── staging.rb
│ ├── staging_***.rb
│ └── ***_test.rb
├── deploy_support
│ ├── bundler_capistrano.rb
│ ├── chat_notification.rb
│ ├── deploy_utils.rb
│ └── rsync_with_remote_cache.rb
:

View Slide

$ wc -l config/cutty_deploy.rb \
config/deploy/* \
config/deploy_support/*
!
2595 total

View Slide

PROBLEMS
SSH is slow
High CPU usage on deployment
Sometime Fails

View Slide

PROBLEM:
Sometime Fails

View Slide

CI
git repo
developer
#
$
"
%
&
'
10 min
1..5 min 10 min

View Slide

CI
git repo
developer "

View Slide

CI
git repo
developer
x
3 min
"
retry

View Slide

CI
git repo
developer
x
3 min
"
retry
3 min …
x

View Slide

CI
git repo
developer
#
"
%
'
10 min
1..5 min 10 min ?
$
&

View Slide

CI
git repo
developer
#
$
"
%
&
'
10 min
1..5 min 10..20 min+

View Slide

My team “dev-infra” aims to:
Improve developers’ productivity
!
Keep development fast
Maintain & improve test environment
etc

View Slide

WE HAVE TO
IMPROVE
of course,

View Slide

IMPROVEMENT PLANS
Upgrade to Capistrano 3?
!
It has better SSH handling,
but still depends on SSH.
SSH is slow.

View Slide

CREATE, NEW TOOL!
Create new tool that uses another way
for deployments!

View Slide

INTRODUCING
* sorah/mamiya
(pronounce like mar-me-ya)

View Slide

MAMIYA
use Serf for orchestration
use Amazon S3 for file distribution (by default)
compatible directory structure with Capistrano

View Slide

SERF?
* hashicorp/serf
Orchestration tool
!
Decentralized, fault-tolerant, highly available
Uses Gossip protocol (SWIM)

View Slide

GOSSIP PROTOCOL?
A gossip protocol is a style of computer-to-
computer communication protocol inspired
by the form of gossip seen in social
networks.
http://en.wikipedia.org/wiki/Gossip_protocol

View Slide

GOSSIP PROTOCOL:
interval: 200ms, total nodes: 8, fanout: 2
= event
node
node
node
node
node
node
node
node
e

View Slide

node
node
node
node
node
GOSSIP PROTOCOL:
node
node
e
Receives Event.
node
0ms 200ms 400ms 600ms 800ms
+

View Slide

node
node
node
node
node
GOSSIP PROTOCOL:
node
node
e
Receives Event.
Choose nodes to gossip
node
0ms 200ms 400ms 600ms 800ms
+

View Slide

node
node
node
node
node
GOSSIP PROTOCOL:
node
node
e
Receives Event.
Fan out.
node
0ms 200ms 400ms 600ms 800ms
+

View Slide

node
node
node
node
node
GOSSIP PROTOCOL:
node
node
e
e
e
Receives Event.
Fan out.
node
0ms 200ms 400ms 600ms 800ms
+

View Slide

node
node
node
node
node
GOSSIP PROTOCOL:
node
node
e
e
e
e
e
e
e
Receives Event.
Fan out.
node
0ms 200ms 400ms 600ms 800ms
+

View Slide

node
node
node
node
node
GOSSIP PROTOCOL:
node
node
e
e
e
e
e
e
e
Receives Event.
Fan out.
node
e
e
e
e
e
e
e
e
0ms 200ms 400ms 600ms 800ms
+

View Slide

node
node
node
node
node
GOSSIP PROTOCOL:
node
node
e
e
e
e
e
e
e
Receives Event.
Fan out.
node
e
e
e
e
e
e
e
e
Drop received event if it’s known
0ms 200ms 400ms 600ms 800ms
+

View Slide

node
node
node
node
node
GOSSIP PROTOCOL:
node
node
e
e
e
e
e
e
e
Receives Event.
Fan out.
node
e
0ms 200ms 400ms 600ms 800ms
+

View Slide

node
node
node
node
node
GOSSIP PROTOCOL:
node
node
e
e
e
e
e
e
e
Receives Event.
Fan out.
node
e
e
e
0ms 200ms 400ms 600ms 800ms
+

View Slide

node
node
node
node
node
GOSSIP PROTOCOL:
node
node
e
e
e
e
e
e
e
Receives Event.
Fan out.
node
e
0ms 200ms 400ms 600ms 800ms
+

View Slide

node
node
node
node
node
GOSSIP PROTOCOL:
node
node
e
e
e
e
e
e
e
Receives Event.
Fan out.
node
e
0ms 200ms 400ms 600ms 800ms
+
Now all nodes has the event.

View Slide

SERF
Consumes UDP bandwidth proportional to
cluster size, but better than SSH.
* hashicorp/serf

View Slide

VILLEIN
Side-eﬀect gem for Mamiya
Simple gem to control `serf` from Ruby
* sorah/villein

View Slide

HOW IT WORKS

View Slide

Terminologies & Concept
Master node
Agent node
Package
Storage
Steps

View Slide

Master node
has HTTP API to control cluster
sends requests to agents (via serf)
watches agents’ status

View Slide

Agent node
accepts requests from master node
runs deploy tasks

View Slide

Deploy Script
writes how to “build”, “prepare”,
and “release”

View Slide

Package
is a tarball of files to deploy
can be pushed to storage
contains Deploy Script

View Slide

Storage
can store Packages
used from Agent nodes

View Slide

Step
is part of deployment
can be run separately
called remotely

View Slide

Steps
Fetch package from storage
Prepare fetched package (bundle install, etc)
Switch to prepared package (reload, graceful)

View Slide

1. CI builds package when passed
2. CI pushes the package to storage
3. — Deployment starts —
4. Master sends “prepare” request to Agents
5. Agents fetch package, then prepare
Mamiya’s Deploy ﬂow

View Slide

6. Master confirms all agents have prepared
7. Master sends “switch” request to Agents
8. Agent switches symlinks, then reload app process
Mamiya’s Deploy ﬂow

View Slide

mamiya’s deploy ﬂow
storage
CI
developer
,
build+push
check deploy
pass
prepare
app
merge
"
&
#
-
%
reload
.

View Slide

Result
Removed dependency to slow SSH
!
but, more…?

View Slide

Step
is part of deployment
can be run separately
called remotely

View Slide

ANOTHER GOAL
Do preparation
before developer say “DEPLOY!”

View Slide

mamiya’s deploy ﬂow (prepare earlier)
storage
CI
developer
,
build+push
check deploy
pass
prepare
app
merge
"
&
#
. -
%
reload app

View Slide

Result (cap→mamiya)
Before: 8.4 minutes
After: 45 seconds
!
for 110 servers, 11.2x faster!

View Slide

DEMO

View Slide

Future Plans
Better documentation (soon)
Auto-deploy when joining cluster
Web UI
Better error tracking, handling
Incremental Packages
“master is always deployed”

View Slide

THANKS
* sorah/mamiya
questions? @sora_h

View Slide

Scalable Deployments - How we deploy Rails app to 150+ hosts in a minute

Scalable Deployments - How we deploy Rails app to 150+ hosts in a minute

Sorah Fukumori

More Decks by Sorah Fukumori

Other Decks in Programming

Featured

Transcript