Save 37% off PRO during our Black Friday Sale! »

Inside the End-to-End Provider Testing Suite

Inside the End-to-End Provider Testing Suite

All about the PagerDuty alert routing decision testing system, which leverages a lab full of cell phones to measure alert deliverability across various cellular and telephony providers. Data from this system is used to make routing decisions, such as "switch to an alternate SMS-provider because the current is slow for this carrier".

C8a8889a30543fdb8cf2841a19d43834?s=128

Evan Gilman

March 10, 2015
Tweet

Transcript

  1. 4/17/15 @evan2645 EVAN GILMAN Inside the End-To-End Provider Testing Suite

  2. 4/17/15 INSIDE THE END-TO-END PROVIDER TESTING SUITE

  3. 4/17/15 INSIDE THE END-TO-END PROVIDER TESTING SUITE

  4. 4/17/15 INSIDE THE END-TO-END PROVIDER TESTING SUITE

  5. 4/17/15 Some of our customers INSIDE THE END-TO-END PROVIDER TESTING

    SUITE
  6. 4/17/15 RELIABILITY INSIDE THE END-TO-END PROVIDER TESTING SUITE

  7. 4/17/15 Reliability INSIDE THE END-TO-END PROVIDER TESTING SUITE CALLBACKS

  8. 4/17/15 Reliability INSIDE THE END-TO-END PROVIDER TESTING SUITE STATUS PAGES

    … "url" => "http://status.twilio.com/api/v1/services/rest-api/events/ ahNzfnR3aWxpby1zdGF0dXMtaHJkchILEgVFdmVudBiAgIDAwviwCgw", "timestamp" => "Mon, 09 Mar 2015 23:02:33 GMT", "sid" => "ahNzfnR3aWxpby1zdGF0dXMtaHJkchILEgVFdmVudBiAgIDAwviwCgw", "message" => "We have identified messages that are reporting incorrect statuses and sent-dates in the API for some accounts. The process of correcting these fields is now in progress. The API is operating normally at this time.", "informational" => false }, …
  9. 4/17/15 Reliability INSIDE THE END-TO-END PROVIDER TESTING SUITE ERROR HANDLING

    • Retries enqueued immediately • Successful callback no-ops enqueued retries • Status page updates evaluated by a human
  10. 4/17/15 Reliability INSIDE THE END-TO-END PROVIDER TESTING SUITE IT’S A

    TARP!!1
  11. 4/17/15 CONSTANT VALIDATION INSIDE THE END-TO-END PROVIDER TESTING SUITE

  12. 4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE

  13. 4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE ANDROID SOFTWARE

    • Timestamped ruok/imok • Out-of-band result transmission • Statsd emissions
  14. 4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE LOTS OF

    PATHS TO THE CUSTOMER • Four cellular carriers • Three SMS providers • Two short codes - every two minutes • 15+ long codes - every fifteen minutes
  15. 4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE FAILURE MODES

    • Carrier failure • Phones of a single carrier experience delays/failures across all providers • Provider failure • All phones experience delays/failures from a single provider • Provider-Carrier failure • Phones of a single carrier experience delays/failures from a single provider
  16. 4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE MESSAGE ROUTING

  17. 4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE THE NUMBERS:

    MESSAGE VOLUME • ~162 messages/hour
  18. 4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE THE NUMBERS:

    FAILURE FREQUENCY • Semi-monthly provider events • Yearly catastrophic provider outages
  19. 4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE THE NUMBERS:

    SAVED SOULS REDACTED
  20. 4/17/15 RELIABILITY INSIDE THE END-TO-END PROVIDER TESTING SUITE

  21. 4/17/15 E2EPT INSIDE THE END-TO-END PROVIDER TESTING SUITE THE NUMBERS:

    BONUS ROUND
  22. 4/17/15 Thank you. @evan2645 EVAN GILMAN