Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Twilio Cloud: Continuos Delivery under High Availability constraints

Twilio Cloud: Continuos Delivery under High Availability constraints

Twilio opens up the black box of Telecom for developers,
making it incredibly easy to build apps that can communicate via
voice or sms. However, due to the nature of Telecom, this imposes
very strict availability constraints and can affect how fast Twilio can
ship code to production as every change can affect uptime. Learn how
Twilio solves this problem by following a set of engineering principles
to build fault tolerant and highly available services in the cloud.

Bulat Shakirzyanov

April 20, 2012
Tweet

More Decks by Bulat Shakirzyanov

Other Decks in Programming

Transcript

  1. Developer End User Carriers Inbound Calls Outbound Calls Mobile/Browser VoIP

    Voice SMS Phone Numbers Send To/From Phone Numbers Short Codes Dynamically Purchase Phone Numbers Web service APIs to automate Voice and SMS communications Overview
  2. Control the telecom network with any web language Access the

    power of our platform with 5 easy verbs Add VoIP to web or mobile apps with 5 lines of code Toolbox
  3. Business Process Automation Click-to-Call Interactive Voice Response Social Networking Contact

    Center Marketing Campaigns Some examples from our customers Call-Tracking/Lead-Gen
  4. Engineering principles • Contain failures • Fail fast • Retry

    on failure • Be idempotent • Be stateless • Relax consistency
  5. Fail fast class Foo def initialize(stream) @stream = stream end

    def print @stream.puts(inspect) end end
  6. f = Foo.new(nil) f.print Fail fast $ ruby fail_fast.rb foo.rb:7:in

    `print': undefined method `puts' for nil:NilClass (NoMethodError) from fail_fast.rb:2:in `<main>'
  7. f = Foo.new(nil) f.print Fail fast $ ruby fail_fast.rb foo.rb:7:in

    `print': undefined method `puts' for nil:NilClass (NoMethodError) from fail_fast.rb:2:in `<main>' Problem caused
  8. class Foo def initialize(stream) @stream = stream end def print

    @stream.puts(inspect) end end Fail fast Problem surfaced
  9. class Foo def initialize(io) @stream = io Assert.stream(@stream) end def

    print @stream.puts(inspect) end end Fail fast Surface the cause
  10. Fail fast module Assert extend self def stream(io) unless io.kind_of?(IO)

    raise ArgumentError, "#{io.inspect} is not IO", caller end end end
  11. f = Foo.new(nil) f.print Fail fast $ ruby fail_fast.rb foo.rb:4:in

    `initialize': nil is not IO (ArgumentError) from fail_fast.rb:1:in `new' from fail_fast.rb:1:in `<main>'
  12. f = Foo.new(nil) f.print Fail fast $ ruby fail_fast.rb foo.rb:4:in

    `initialize': nil is not IO (ArgumentError) from fail_fast.rb:1:in `new' from fail_fast.rb:1:in `<main>' Problem caused and surfaced
  13. Failed, Retrying in 3 seconds... Failed, Retrying in 6 seconds...

    Failed, Retrying in 12 seconds... Retry on failure
  14. web server web server web server session database browser session

    id session id session data response Be stateless
  15. 1000x Website Content CMS 100x Website Code PHP/Ruby etc. 10x

    REST API Python/Java etc. 1x Big DB Schema SQL Log Scale Deployment Frequency(Risk) 4 buckets
  16. • Build and deployment system - boot entire Twilio stack

    with one key press • Host configuration - versioned code & config • Host orchestration - load balancing • Monitoring and alerting - nagios • Multi-datacenter deployment & analytics BoxConfig
  17. Configuration host base ami latest build role service service service

    service role service service service service