Inside the End-to-End Provider Testing Suite

All about the PagerDuty alert routing decision testing system, which leverages a lab full of cell phones to measure alert deliverability across various cellular and telephony providers. Data from this system is used to make routing decisions, such as "switch to an alternate SMS-provider because the current is slow for this carrier".

Transcript

1. 4/17/15 @evan2645 EVAN GILMAN Inside the End-To-End Provider Testing Suite
2. 4/17/15 INSIDE THE END-TO-END PROVIDER TESTING SUITE
3. 4/17/15 INSIDE THE END-TO-END PROVIDER TESTING SUITE
4. 4/17/15 INSIDE THE END-TO-END PROVIDER TESTING SUITE
5. 4/17/15 Some of our customers INSIDE THE END-TO-END PROVIDER TESTING SUITE
6. 4/17/15 RELIABILITY INSIDE THE END-TO-END PROVIDER TESTING SUITE
7. 4/17/15 Reliability INSIDE THE END-TO-END PROVIDER TESTING SUITE CALLBACKS
8. 4/17/15 Reliability INSIDE THE END-TO-END PROVIDER TESTING SUITE STATUS PAGES … "url" => "http://status.twilio.com/api/v1/services/rest-api/events/ ahNzfnR3aWxpby1zdGF0dXMtaHJkchILEgVFdmVudBiAgIDAwviwCgw", "timestamp" => "Mon, 09 Mar 2015 23:02:33 GMT", "sid" => "ahNzfnR3aWxpby1zdGF0dXMtaHJkchILEgVFdmVudBiAgIDAwviwCgw", "message" => "We have identified messages that are reporting incorrect statuses and sent-dates in the API for some accounts. The process of correcting these fields is now in progress. The API is operating normally at this time.", "informational" => false }, …
9. 4/17/15 Reliability INSIDE THE END-TO-END PROVIDER TESTING SUITE ERROR HANDLING • Retries enqueued immediately • Successful callback no-ops enqueued retries • Status page updates evaluated by a human
10. 4/17/15 Reliability INSIDE THE END-TO-END PROVIDER TESTING SUITE IT’S A TARP!!1
11. 4/17/15 CONSTANT VALIDATION INSIDE THE END-TO-END PROVIDER TESTING SUITE
12. 4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE
13. 4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE ANDROID SOFTWARE • Timestamped ruok/imok • Out-of-band result transmission • Statsd emissions
14. 4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE LOTS OF PATHS TO THE CUSTOMER • Four cellular carriers • Three SMS providers • Two short codes - every two minutes • 15+ long codes - every fifteen minutes
15. 4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE FAILURE MODES • Carrier failure • Phones of a single carrier experience delays/failures across all providers • Provider failure • All phones experience delays/failures from a single provider • Provider-Carrier failure • Phones of a single carrier experience delays/failures from a single provider
16. 4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE MESSAGE ROUTING
17. 4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE THE NUMBERS: MESSAGE VOLUME • ~162 messages/hour
18. 4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE THE NUMBERS: FAILURE FREQUENCY • Semi-monthly provider events • Yearly catastrophic provider outages
19. 4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE THE NUMBERS: SAVED SOULS REDACTED
20. 4/17/15 RELIABILITY INSIDE THE END-TO-END PROVIDER TESTING SUITE
21. 4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE THE NUMBERS: BONUS ROUND
22. 4/17/15 Thank you. @evan2645 EVAN GILMAN