Retry Strategies for Remote Services

Various strategies for retrying failed remote service requests

Transcript

1. Retry Strategies for Remote Services
2. Hey! I'm Chris Trombley @ctroms
5. What are Remote Services?
6. Services that exist in different processes or nodes seperated by a network. • HTTP APIs • Microservices • Application Servers • Remote databases
7. Why Retry?
8. One of the big differences between in-memory calls and remote calls is that remote calls can fail, or hang without a response until some timeout limit is reached. -- Martin Fowler
9. One of the big differences between in-memory calls and remote calls is that remote calls can fail, or hang without a response until some timeout limit is reached. -- Martin Fowler Remote services can experience failures due to their dependency on a network.
10. Failures • Client connection • Processing the request • Returning a response
11. Retryable Errors • Transient failures • Hardware failures • Logical bugs • Resource contention • Rate limits • Quotas • Throttles
12. Why Retry?
13. Reduce fatal errors == Better user experience
14. How Do We Retry?
15. Retry Immediately • Failed client connections • Transient Errors
16. Scheduled Retries • Hourly quotas • Daily quotas • Scheduled downtime
17. Retry With Backoff • Transient failures • Resource contention • Short term quotas • Throttles • Timeouts
18. Backoff Strategies
19. Backoff Strategies • Constant • Linear • Polynomial • Exponential
20. Basic Implementation $attempt = 1; do { $backoff = $this->getBackoff(); sleep($backoff); $response = $this->doRequest(); $attempt++; } while ($response->hasFailed());
21. Constant Strategy
22. For each failed request, increase the wait by the defined constant.
23. Implementation $attempt = 1; $constant = 1; do { sleep($constant); $response = $this->doRequest(); $attempt++; } while ($response->hasFailed());
24. Constant Retry Strategy Attempt Backoff Cumulative Backoff 1 1 1 2 1 2 3 1 3 4 1 4 5 1 5 6 1 6 7 1 7 8 1 8 9 1 9 10 1 10 Mean Delay: 1s
25. Constant Retry Strategy
26. Use Cases • Client connections • Transient failures • Heartbeats
27. Linear Backoff Strategy
28. Increase the wait proportionally by a defined constant.
29. Implementation $delay = 1; $attempt = 1; $constant = 1; do { $delay += $constant; sleep($delay); $response = $this->doRequest(); $attempt++; } while ($response->hasFailed());
30. Linear Retry Strategy Attempt Backoff Cumulative Backoff 1 2 2 2 3 5 3 4 9 4 5 14 5 6 20 6 7 27 7 8 35 8 9 44 9 10 54 10 11 65 Mean Delay: 6.5s
31. Linear Retry Strategy
32. Use cases • 5xx Errors • 429 Errors • Polling Requests
33. Polynomial Backoff Strategy
34. Increase the wait proportionally by Where is a predefined factor.
35. Implementation $base = 1; $attempt = 1; $factor = 3; do { $delay = $base * pow($attempt, $factor); sleep($delay); $response = $this->doRequest(); $attempt++; } while ($response->hasFailed());
36. Polynomial Backoff Strategy Attempt Backoff Cumulative Backoff 1 1 1 2 8 9 3 27 36 4 64 100 5 125 225 6 216 441 7 343 784 8 512 1296 9 729 2025 10 1000 3025 Mean Delay: 302.5s
37. Exponential Backoff Strategy
38. Increase the wait proportionally by Where is the number of attempts.
39. Implementation $base = 1; $attempt = 1; do { $delay = $base * pow(2, $attempt); sleep($delay); $response = $this->doRequest(); $attempt++; } while ($response->hasFailed());
40. Exponential Backoff Strategy Attempt Backoff Cumulative Backoff 1 2 2 2 4 6 3 8 14 4 16 30 5 32 62 6 64 126 7 128 254 8 256 510 9 512 1022 10 1024 2046 Mean Delay: 204.6s
41. Exponential Backoff Strategy
42. Use cases • Most HTTP API requests • High resource contention • Transient errors
43. 2046 seconds
45. Exponential Backoff Strategy Attempt Backoff Cumulative Backoff 1 2 2 2 4 6 3 8 14 4 16 30 5 32 62 6 64 126 7 128 254 8 256 510 9 512 1022 10 1024 (~17 min) 2046 (~34 min) Mean Delay: 204.6s
46. Limits
47. Common Limits for Retry Strategies
48. Common Limits for Retry Strategies • Retry attempts: To prevent a request from being retried indefinitely
49. Common Limits for Retry Strategies • Retry attempts: To prevent a request from being retried indefinitely • Backoff duration: To keep the maximum delay within a reasonable timeframe
50. Limiting Retry Attempts
51. Implementation 1. Set a max limit for retry attempts 2. When limit is reached • Raise an error • Implement an error handling strategy
52. Limiting Retry Attempts $attempt = 1; $maxAttempts = 10; do { if ($attempt > $maxAttempts) { throw new \Exception(); } $backoff = $this->getDelay(); sleep($backoff); $response = $this->doRequest(); $attempt++; } while ($response->hasFailed());
53. Limiting Backoff Duration
54. Implementation 1. Set a maximum duration of the backoff 2. When limit is reached • Raise an error • Continue to retry with the maximum backoff value until the maximum attempts are reached
55. Limiting Backoff Duration $attempt = 1; $maxAttempts = 10; $maxBackoffDuration = 64; do { if ($attempt > $maxAttempts) { throw new \Exception(); } $backoff = min($maxBackoffDuration, $this->getDelay()); sleep($backoff); $response = $this->doRequest(); $attempt++; } while ($response->hasFailed());
56. Exponential Backoff Strategy Attempt Backoff Cumulative Backoff 1 2 2 2 4 6 3 8 14 4 16 30 5 32 62 6 64 126 7 128 254 8 256 510 9 512 1022 10 1024 (~17 min) 2046 (~34 min) Backoff Mean Delay: 204.6s
57. Exponential Backoff Strategy Attempt Backoff Cumulative Backoff Lim. Backoff Lim. Cumulative 1 2 2 2 2 2 4 6 4 6 3 8 14 8 14 4 16 30 16 30 5 32 62 32 62 6 64 126 64 126 7 128 254 64 190 8 256 510 64 254 9 512 1022 64 318 10 1024 2046 (~34 min) 64 382 (~6.5 min) Backoff Mean Delay: 204.6s Lim. Backoff Mean Delay: 38.2s
58. Exponential Backoff Strategy
59. When the limit is reached
60. When the limit is reached • Raise an error and stop retrying
61. When the limit is reached • Raise an error and stop retrying • Implement an error handling strategy
62. When the limit is reached • Raise an error and stop retrying • Implement an error handling strategy • Inform a circuit breaker pattern of change in state
63. But Wait!
64. But Wait! There's Still a Problem
65. The data we've been examining up to this point assumes we have one client connection per period.
66. What we have simultanious attempts from...
67. What we have simultanious attempts from... • 10 clients
68. What we have simultanious attempts from... • 10 clients • 100 clients
69. What we have simultanious attempts from... • 10 clients • 100 clients • 1000 clients
71. Thundering Herd Problem
72. How Do We Solve This?
73. Jitter
74. Not 100 Cups of Coffee
75. Jitter => Randomness
76. Common Types of Jitter • Constant • Equal • Full • Decorrelated
77. Constant Jitter
78. Retains all of the slowdown of the backoff strategy
79. Implementation $base = 1; $attempt = 1; $maxAttempts = 10; $maxBackoffDuration = 64; do { if ($attempt > $maxAttempts) { throw new \Exception(); } $constant = rand(1, 1000) * .001; $backoff = min($maxDuration, $base * pow(2, $attempt)) + $constant; sleep($backoff); $response = $this->doRequest(); $attempt++; } while ($response->hasFailed());
80. Constant Jitter Attempt Backoff Cumulative Backoff 1 2.252 2.252 2 4.111 6.363 3 8.957 15.32 4 16.091 31.411 5 32.848 64.259 6 64.785 129.044 7 64.635 193.679 8 64.24 257.919 9 64.668 322.587 10 64.247 386.834 Mean Delay: 38.6s
81. Constant Jitter
82. Uses • HTTP API's
83. Equal Jitter
84. Retains some of the slowdown of the backoff strategy
85. Implementation $base = 1; $attempt = 1; $maxAttempts = 10; $maxBackoffDuration = 64; do { if ($attempt > $maxAttempts) { throw new \Exception(); } $backoff = min($maxDuration, $base * pow(2, $attempt)); $backoff = $backoff / 2 + rand(0, $backoff / 2); sleep($backoff); $response = $this->doRequest(); $attempt++; } while ($response->hasFailed());
86. Equal Jitter Attempt Backoff Cumulative Backoff 1 1 1 2 2 3 3 4 7 4 14 21 5 26 47 6 35 82 7 53 135 8 41 176 9 58 234 10 62 296 Mean Delay: 29.6s
89. Full Jitter
90. Does not retain the slowdown of the backoff strategy
91. Full Jitter $base = 1; $attempt = 1; $maxAttempts = 10; $maxBackoffDuration = 64; do { if ($attempt > $maxAttempts) { throw new \Exception(); } $backoff = min($maxDuration, $base * pow(2, $attempt)); $backoff = rand(0, $backoff); sleep($backoff); $response = $this->doRequest(); $attempt++; } while ($response->hasFailed());
92. Full Jitter Attempt Backoff Cumulative Backoff 1 1 1 2 2 3 3 8 11 4 13 24 5 11 35 6 18 53 7 21 74 8 15 89 9 35 124 10 23 147 Mean Delay: 14.7s
95. Decorrelated Jitter
96. Typically shorter delays than the other jittered backoff methods
97. Decorrelated Jitter $base = 1; $attempt = 1; $maxAttempts = 10; $maxBackoffDuration = 64; do { if ($attempt > $maxAttempts) { throw new \Exception(); } $backoff = min($maxDuration rand($base, $previousBackoff * 3)); $backoff = rand($base, $backoff); $previousBackoff = $backoff; sleep($backoff); $response = $this->doRequest(); $attempt++; } while ($response->hasFailed());
98. Decorrelated Jitter $base = 1; $attempt = 1; $maxAttempts = 10; $maxBackoffDuration = 64; do { if ($attempt > $maxAttempts) { throw new \Exception(); } $backoff = min($maxDuration rand($base, $previousBackoff * 3)); $backoff = rand($base, $backoff); $previousBackoff = $backoff; sleep($backoff); $response = $this->doRequest(); $attempt++; } while ($response->hasFailed());
99. Decorrelated Jitter Attempt Backoff Cumulative Backoff 1 3 3 2 1 4 3 2 6 4 3 9 5 6 15 6 7 22 7 3 25 8 4 29 9 1 30 10 3 33 Mean Delay: 3.3s
102. Why Not Exponential?
104. Which One Should We Use?
105. It Depends...
107. • What is the cost to connect
108. • What is the cost to connect • Will the service be available quickly
109. • What is the cost to connect • Will the service be available quickly • How quickly do I need a response
110. Other Considerations • Read The Docs • Would retrying create an inconsistant state? • Quotas, Rate limits, Throttles?
111. Read The Docs • Most docs will(should) tell you how to retry failed requests
112. Twitter Streaming Api https://developer.twitter.com/en/docs/tweets/filter-realtime/guides/connecting.html
113. Google Analytics Core Reporting API https://developers.google.com/analytics/devguides/reporting/core/v3/errors#backoff
114. Stripe API https://stripe.com/docs/api/php#errors
115. Other Considerations • Read The Docs • Will retrying create an inconsistant state? • Quotas, Rate limits, Throttles?
116. Will retrying create an inconsistant state? Read Requests • Generally no risk in retries • Set sensible defaults for $base, $maxAttempts and $maxBackoffDuration • Retry as needed
117. Will retrying create an inconsistant state? Destructive Requests • Create, Update, Delete • Determine if the request is Idempotent • Retry as needed
118. Safe and Idempotent REST Methods Method Safe Idempotent GET yes yes POST no no PUT no yes PATCH no no DELETE no yes
119. Idempotent • Set sensible defaults for $base, $maxAttempts and $maxBackoffDuration • Retry as needed
120. Not Idempotent • Raise an error • Implement idempotency keys with requests
121. Other Considerations • Read The Docs • Would retrying create an inconsistant state? • Quotas, Rate limits, Throttles?
122. Quotas and Rate Limits • Determine a strategy for the use case • Set sensible defaults for $base, $maxAttempts and $maxBackoffDuration • Retry as Needed
123. Other Considerations • Read The Docs • Would retrying create an inconsistant state? • Quotas, Rate limits, Throttles?
124. Wrap Up
125. Wrap Up • Retry strategies can be very effective in maintaining availability of a service with service dependencies
126. Wrap Up • Retry strategies can be very effective in maintaining availability of a service with service dependencies • Limits should be implemented to avoid indefinite retries and long delays between requests.
127. Wrap Up • Retry strategies can be very effective in maintaining availability of a service with service dependencies • Limits should be implemented to avoid indefinite retries and long delays between requests. • Adding Jitter to retry strategies can help avoid collisions with other clients.
128. Thanks! @ctroms ctroms@gmail.com