Twilio Cloud
Continuous Delivery under High Availability constraints
BULAT SHAKIRZYANOV
INFRASTRUCTURE ENGINEER
Slide 2
Slide 2 text
Twilio
Slide 3
Slide 3 text
Developer
End User
Carriers Inbound Calls
Outbound Calls
Mobile/Browser VoIP
Voice
SMS
Phone
Numbers
Send To/From Phone
Numbers
Short Codes
Dynamically Purchase
Phone Numbers
Web service APIs to automate Voice and
SMS communications
Overview
Slide 4
Slide 4 text
Control the telecom network with
any web language
Access the power of our platform
with 5 easy verbs
Add VoIP to web or mobile apps
with 5 lines of code
Toolbox
Slide 5
Slide 5 text
Business Process
Automation
Click-to-Call
Interactive Voice
Response
Social Networking
Contact Center
Marketing Campaigns
Some examples from our customers
Call-Tracking/Lead-Gen
Slide 6
Slide 6 text
Globally available 24x7
Slide 7
Slide 7 text
Engineering principles
• Contain failures
• Fail fast
• Retry on failure
• Be idempotent
• Be stateless
• Relax consistency
Slide 8
Slide 8 text
Contain failures
Slide 9
Slide 9 text
master
host
web
server
queue
Contain failures
Slide 10
Slide 10 text
master
host
web
server
queue
Contain failures
POST /data
Slide 11
Slide 11 text
slave
host
web
server
load
balancer
Contain failures
slave
slave
Slide 12
Slide 12 text
slave
host
web
server
load
balancer
Contain failures
GET /data
slave
slave
Slide 13
Slide 13 text
Fail fast
Slide 14
Slide 14 text
Fail fast
ASSERT(expected == actual);
Slide 15
Slide 15 text
Fail fast
class Foo
def initialize(stream)
@stream = stream
end
def print
@stream.puts(inspect)
end
end
Slide 16
Slide 16 text
f = Foo.new(nil)
f.print
Fail fast
Slide 17
Slide 17 text
f = Foo.new(nil)
f.print
Fail fast
$ ruby fail_fast.rb
foo.rb:7:in `print': undefined method `puts' for nil:NilClass (NoMethodError)
from fail_fast.rb:2:in `'
Slide 18
Slide 18 text
f = Foo.new(nil)
f.print
Fail fast
$ ruby fail_fast.rb
foo.rb:7:in `print': undefined method `puts' for nil:NilClass (NoMethodError)
from fail_fast.rb:2:in `'
Problem caused
Slide 19
Slide 19 text
class Foo
def initialize(stream)
@stream = stream
end
def print
@stream.puts(inspect)
end
end
Fail fast
Problem surfaced
Slide 20
Slide 20 text
class Foo
def initialize(io)
@stream = io
Assert.stream(@stream)
end
def print
@stream.puts(inspect)
end
end
Fail fast
Surface the cause
Slide 21
Slide 21 text
Fail fast
module Assert
extend self
def stream(io)
unless io.kind_of?(IO)
raise ArgumentError,
"#{io.inspect} is not IO",
caller
end
end
end
Slide 22
Slide 22 text
f = Foo.new(nil)
f.print
Fail fast
$ ruby fail_fast.rb
foo.rb:4:in `initialize': nil is not IO (ArgumentError)
from fail_fast.rb:1:in `new'
from fail_fast.rb:1:in `'
Slide 23
Slide 23 text
f = Foo.new(nil)
f.print
Fail fast
$ ruby fail_fast.rb
foo.rb:4:in `initialize': nil is not IO (ArgumentError)
from fail_fast.rb:1:in `new'
from fail_fast.rb:1:in `'
Problem caused and surfaced
Slide 24
Slide 24 text
Retry on failure
Slide 25
Slide 25 text
Failed, Retrying in 3 seconds...
Failed, Retrying in 6 seconds...
Failed, Retrying in 12 seconds...
Retry on failure
Slide 26
Slide 26 text
Be idempotent
Slide 27
Slide 27 text
Be idempotent
+ 1 + 1 = 1
Slide 28
Slide 28 text
Be stateless
Slide 29
Slide 29 text
web
server
web
server
web
server
session
database
browser
Be stateless
Slide 30
Slide 30 text
web
server
web
server
web
server
session
database
browser
session id
session id
session
data
response
Be stateless
Slide 31
Slide 31 text
web
server
web
server
web
server
browser
Be stateless
session
data
response
Slide 32
Slide 32 text
Relax consistency
Slide 33
Slide 33 text
Relax consistency
database
server
database
client
database
client
database
client
Slide 34
Slide 34 text
Relax consistency
database
server
database
client
database
client
database
client
database
server
database
server
Slide 35
Slide 35 text
Automation
• Build and test
• Configuration management
• Monitoring
• Orchestration
• Scaling
Slide 36
Slide 36 text
No content
Slide 37
Slide 37 text
1000x
Website
Content
CMS
100x
Website
Code
PHP/Ruby
etc.
10x
REST
API
Python/Java
etc.
1x
Big DB
Schema
SQL
Log Scale
Deployment
Frequency(Risk)
4 buckets
Slide 38
Slide 38 text
• Build and deployment system - boot entire Twilio stack with
one key press
• Host configuration - versioned code & config
• Host orchestration - load balancing
• Monitoring and alerting - nagios
• Multi-datacenter deployment & analytics
BoxConfig
Slide 39
Slide 39 text
Configuration
host
Slide 40
Slide 40 text
Configuration
host
base ami
latest build
role
service service service service
role
service service service service