Tweet
Tweet

Slide 1

Slide 1 text

4/17/15 @evan2645 EVAN GILMAN Inside the End-To-End Provider Testing Suite

Slide 2

Slide 2 text

4/17/15 INSIDE THE END-TO-END PROVIDER TESTING SUITE

Slide 3

Slide 3 text

4/17/15 INSIDE THE END-TO-END PROVIDER TESTING SUITE

Slide 4

Slide 4 text

4/17/15 INSIDE THE END-TO-END PROVIDER TESTING SUITE

Slide 5

Slide 5 text

4/17/15 Some of our customers INSIDE THE END-TO-END PROVIDER TESTING SUITE

Slide 6

Slide 6 text

4/17/15 RELIABILITY INSIDE THE END-TO-END PROVIDER TESTING SUITE

Slide 7

Slide 7 text

4/17/15 Reliability INSIDE THE END-TO-END PROVIDER TESTING SUITE CALLBACKS

Slide 8

Slide 8 text

4/17/15 Reliability INSIDE THE END-TO-END PROVIDER TESTING SUITE STATUS PAGES … "url" => "http://status.twilio.com/api/v1/services/rest-api/events/ ahNzfnR3aWxpby1zdGF0dXMtaHJkchILEgVFdmVudBiAgIDAwviwCgw", "timestamp" => "Mon, 09 Mar 2015 23:02:33 GMT", "sid" => "ahNzfnR3aWxpby1zdGF0dXMtaHJkchILEgVFdmVudBiAgIDAwviwCgw", "message" => "We have identified messages that are reporting incorrect statuses and sent-dates in the API for some accounts. The process of correcting these fields is now in progress. The API is operating normally at this time.", "informational" => false }, …

Slide 9

Slide 9 text

4/17/15 Reliability INSIDE THE END-TO-END PROVIDER TESTING SUITE ERROR HANDLING • Retries enqueued immediately • Successful callback no-ops enqueued retries • Status page updates evaluated by a human

Slide 10

Slide 10 text

4/17/15 Reliability INSIDE THE END-TO-END PROVIDER TESTING SUITE IT’S A TARP!!1

Slide 11

Slide 11 text

4/17/15 CONSTANT VALIDATION INSIDE THE END-TO-END PROVIDER TESTING SUITE

Slide 12

Slide 12 text

4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE

Slide 13

Slide 13 text

4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE ANDROID SOFTWARE • Timestamped ruok/imok • Out-of-band result transmission • Statsd emissions

Slide 14

Slide 14 text

4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE LOTS OF PATHS TO THE CUSTOMER • Four cellular carriers • Three SMS providers • Two short codes - every two minutes • 15+ long codes - every fifteen minutes

Slide 15

Slide 15 text

4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE FAILURE MODES • Carrier failure • Phones of a single carrier experience delays/failures across all providers • Provider failure • All phones experience delays/failures from a single provider • Provider-Carrier failure • Phones of a single carrier experience delays/failures from a single provider

Slide 16

Slide 16 text

4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE MESSAGE ROUTING

Slide 17

Slide 17 text

4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE THE NUMBERS: MESSAGE VOLUME • ~162 messages/hour

Slide 18

Slide 18 text

4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE THE NUMBERS: FAILURE FREQUENCY • Semi-monthly provider events • Yearly catastrophic provider outages

Slide 19

Slide 19 text

4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE THE NUMBERS: SAVED SOULS REDACTED

Slide 20

Slide 20 text

4/17/15 RELIABILITY INSIDE THE END-TO-END PROVIDER TESTING SUITE

Slide 21

Slide 21 text

4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE THE NUMBERS: BONUS ROUND

Slide 22

Slide 22 text

4/17/15 Thank you. @evan2645 EVAN GILMAN