Avoiding Déjà Vu: Building Resilient APIs with Idempotency

Users clicking "Submit" twice, poor network connections, proxies repeating requests unexpectedly - repeated requests can cause all kinds of problems in our applications if we're not careful! Idempotency ensures repeated API calls produce consistent results, saving systems from duplicate charges, mismanaged resources, and confused customers. This talk explains idempotency’s role in building reliable APIs, with practical Laravel-based examples to show you how to implement it effectively.

We’ll share real-world stories of what happens when idempotency goes wrong (or doesn’t exist), and go through how some of the largest companies in the world like Stripe and Shopify implement this technique. Whether you’re a seasoned API developer or new to the concept, you’ll leave with actionable insights and techniques to make your APIs smarter and more user-friendly.

Transcript

1. Avoiding Déjà Vu: Building Resilient APIs with Idempotency Paul Conroy / @conroyp
2. From Dublin, Ireland Started playing with the web 30+ years ago (Notepad, Frontpage & Geocities!) CTO at Square1 conroyp.com / @conroyp Paul Conroy 👴 🌍 🇮🇪
3. https://en.wikipedia.org/wiki/Déjà_vu
4. https://en.wikipedia.org/wiki/Déjà_vu
5. How does Déjà Vu affect our API servers?
6. 💸 Double charges How does Déjà Vu affect our API servers?
7. 💸 Double charges 📦 Duplicate orders How does Déjà Vu affect our API servers?
8. 💸 Double charges 📦 Duplicate orders 🐛 Data integrity issues How does Déjà Vu affect our API servers?
9. 💸 Double charges 📦 Duplicate orders 🐛 Data integrity issues How does Déjà Vu affect our API servers? 🤬 Angry customers
10. 💸 Double charges 📦 Duplicate orders 🐛 Data integrity issues How does Déjà Vu affect our API servers? 🤬 Angry customers 🗣 Higher support costs
12. What is idempotency?
13. What is idempotency? How do I pronounce
14. eye-dum-po-ten-see
15. https://en.wikipedia.org/wiki/Idempotence “Idem” - same “Potence” - power
16. An operation is idempotent if applying it multiple times has the same effect as applying it once.
20. Idempotency in our APIs
21. 🧑💻
22. 🧑💻
23. 🧑💻
24. 🧑💻
25. 🧑💻
26. 🧑💻 🍔🍟
27. 🧑💻 🍔🍟 💶
28. 🧑💻
29. 🧑💻
30. 🧑💻
31. 🧑💻
32. 🤷
33. 🧑💻
34. 🧑💻
35. 🤷
36. 🤷
37. Reasons for retries • Impatient users • Mobile apps retrying failed connections • Load balancer failovers • CDN fallbacks
38. 💸 Double charges 📦 Duplicate orders 🐛 Data integrity issues 🤬 Angry customers 🗣 Higher support costs Consequences of missing idempotency
39. Who uses it?
40. Who uses it?
41. Who uses it?
42. Who uses it?
43. How can we make sure the same request gets processed only once?
44. Naturally idempotent: • GET • HEAD • OPTIONS HTTP Verbs • PUT • DELETE
45. Naturally idempotent: • GET • HEAD • OPTIONS Non-idempotent verbs: • POST • PATCH HTTP Verbs • PUT • DELETE
46. Naturally idempotent: • GET • HEAD • OPTIONS Non-idempotent verbs: • POST • PATCH HTTP Verbs • PUT • DELETE
47. Naturally idempotent: • GET • HEAD • OPTIONS Non-idempotent verbs: • POST • PATCH HTTP Verbs • PUT • DELETE
48. Naturally idempotent: • GET • HEAD • OPTIONS Non-idempotent verbs: • POST • PATCH HTTP Verbs • PUT • DELETE Observable side-effects
49. Naturally idempotent: • GET • HEAD • OPTIONS Non-idempotent verbs: • POST • PATCH HTTP Verbs • PUT • DELETE Observable side-effects
50. 🧑💻1
51. 🧑💻1 2 🧑💻
52. 🧑💻1 2 🧑💻
53. 🧑💻1 2 🧑💻 🍔🍟
54. 🧑💻1 2 🧑💻 🍔🍟
55. 🧑💻1 2 🧑💻 🍔🍟 💸💸
56. 🧑💻1 2 🧑💻 🍔🍟 💸💸 🍔🍟
57. 🧑💻1 2 🧑💻 🍔🍟 💸💸 🍔🍟
58. 🧑💻1
59. 🧑💻1 1 🧑💻
60. 🧑💻1 1 🧑💻
61. 🧑💻1 1 🧑💻
62. 🧑💻1 1 🧑💻
63. 🧑💻1 1 🧑💻
64. 🧑💻1 1 🧑💻 🍔🍟
65. 🧑💻1 1 🧑💻 🍔🍟
66. 🧑💻1 1 🧑💻 🍔🍟 💶
67. How does it work?
68. 🧑💻 Seen the request before?
69. 🧑💻 Seen the request before? Generate response No
70. 🧑💻 Seen the request before? Generate response Save response to cache No
71. 🧑💻 Seen the request before? Generate response Save response to cache Return response No
72. 🧑💻 Seen the request before? Retrieve response from cache Generate response Save response to cache Return response Yes No
73. Have we seen the request before?
74. Use a unique hash per request, storing it as a cache key. But where do we get the hash from? How do we know if we’ve seen a request before?
75. Request Body Build up hash using all parameters contained in the request order
76. Request Body Build up hash using all parameters contained in the request order 🧑💻 🍔🍟
77. Request Body Build up hash using all parameters contained in the request order 🧑💻 🧑💻 🍔🍟 🍔🍟
78. IP Address Include user’s public IP address in hash 🧑💻 🍔🍟 104.17.3.109
79. IP Address Include user’s public IP address in hash 🧑💻 🧑💻 🍔🍟 🍔🍟 🏢 104.17.3.109 104.17.3.109
80. user_id: 3792 User ID Logged-in user’s ID 🧑💻 🧑💻 🍔🍟 🍔🍟 🏢 user_id: 2552
81. user_id: 3792 User ID Logged-in user’s ID 🧑💻 🧑💻 🍔🍟 🍔🍟 🏢 user_id: 2552 user_id: null user_id: null
82. How do we know if we’ve seen a request before? Use a unique hash per request, storing it as a cache key. But where do we get the hash from?
83. How do we know if we’ve seen a request before? • Request body - but duplicate orders! Use a unique hash per request, storing it as a cache key. But where do we get the hash from?
84. How do we know if we’ve seen a request before? • Request body - but duplicate orders! • IP address - but shared public IPs! Use a unique hash per request, storing it as a cache key. But where do we get the hash from?
85. How do we know if we’ve seen a request before? • Request body - but duplicate orders! • IP address - but shared public IPs! • User ID - but guest checkouts! Use a unique hash per request, storing it as a cache key. But where do we get the hash from?
86. How do we know if we’ve seen a request before? • Request body - but duplicate orders! • IP address - but shared public IPs! • User ID - but guest checkouts! Make the client do some work! Use a unique hash per request, storing it as a cache key. But where do we get the hash from?
87. 🧑💻1 1 🧑💻 🍔🍟 💶
88. 🧑💻1 1 🧑💻 🍔🍟 💶 Idempotency-Key: 1 Idempotency-Key: 1
89. 🧑💻Idempotency-Key: 1 • Make the client pass a key with each idempotent request • Use this as the basis for our server-side cache • Typically use UUIDs to avoid collisions • Client SDKs help with key generation
90. https://github.com/stripe/stripe-php/blob/master/lib/HttpClient/CurlClient.php#L281
91. https://github.com/stripe/stripe-php/blob/master/lib/HttpClient/CurlClient.php#L281
92. 🧑💻 Seen the request before? Retrieve response from cache Generate response Save response to cache Return response Yes No Idempotency-Key: 1
93. How does it work?
94. How does it really work?
95. Application 🧑💻
96. Application Middleware 🧑💻
97. Application Middleware 🧑💻
104. Make app generate response
105. Make app generate response Put whole structure into cache
106. Make app generate response Put whole structure into cache Cache duration!
108. Pull response from cache
109. Pull response from cache Has the key been used elsewhere?
110. Pull response from cache Has the key been used elsewhere? Show it’s a replayed request
113. Internal use?
114. Make app generate response Put whole structure into cache Cache duration!
115. Choose a TTL based on retry behaviour, business impact, and storage constraints How long should we cache for? ⏳ Short TTL (Mins - Hours) ⏱ Medium TTL (Hours to Days) 📅 Long TTL (Days to Weeks) 🔒 Infinite TTL (Persistent Storage)
116. Scenario: Mobile app order processing (Food delivery application) User Behaviour: Users place orders on their phones while outside. Retry Pattern: The mobile app automatically retries failed requests up to 3 times within a 5-minute window. Key Selection Strategy: UUIDs generally sufficient. ⏳ Short TTL (Mins - Hours) • Most connectivity issues resolve within minutes • Retries typically happen almost immediately or within a few minutes • After this window, a new order attempt is likely genuinely new
117. Scenario: Batch payment processing (B2B SaaS platform) User Behaviour: Businesses run scheduled payment batch jobs that process hundreds of transactions. Retry Pattern: Failed batches are retried same day or the next morning. Key Selection Strategy: UUIDs still generally sufficient - user/session identifiers may be helpful for multi-day caches. ⏱ Medium TTL (Hours to Days) • Business hours and operational patterns dictate retry windows • System failures may take several hours to resolve • Next-day retries are common in business workflows
118. Scenario: Subscription Management System (B2B SaaS platform handling subscription payments) User Behaviour: Users change subscription tiers, add features, etc. Retry Pattern: Support teams need to reprocess failed changes days later. Key Selection Strategy: Consider additional business context added to the key. 📅 Long TTL (Days to Weeks) • Customer service tickets often take days to resolve • Subscription changes have billing cycle implications • Retry attempts may happen after delays with customer communication
119. Scenario: Regulatory Compliance Reporting User Behaviour: System submits mandatory reports to govt agencies. Retry Pattern: Failed submissions retried indefinitely until successful, but must never be duplicated. Key Selection Strategy: Additional metadata about requester. 🔒 Infinite TTL (Persistent Storage) • Regulatory requirements prohibit both missed and duplicate reports • Legal penalties for non-compliance are severe • The reporting requirement never expires
120. Should errors be cached? Should we allow retries? What about errors? https://docs.stripe.com/api/idempotent_requests
121. Should errors be cached? Should we allow retries? What about errors? https://docs.stripe.com/api/idempotent_requests
122. Should errors be cached? Should we allow retries? What about errors? https://docs.stripe.com/api/idempotent_requests
123. Race conditions 🧑💻 ⏳
124. Race conditions 🧑💻 ⏳
125. Race conditions 🧑💻 ⏳
126. Cache locking Using a lock for the process allows us to ensure only one process handles the request.
 • Try to get a cache lock • If we can, process the request then release the lock • If we can’t, the same request is being processed by another process. Wait until it’s done
127. Cache locking
128. Cache locking Another process mid-flight
129. Cache locking Another process mid-flight Release the lock afterwards!
133. Another process finished it?
134. Another process finished it? Sleep for a moment, check again
135. Another process finished it? Sleep for a moment, check again Something has gone wrong!
136. Another process finished it? Sleep for a moment, check again Something has gone wrong! ⚠🚨⚠
137. https://github.com/square1-io/laravel-idempotency
138. Laravel Implementation
140. Race conditions
141. Race conditions User ID resolver
142. Race conditions User ID resolver Replay or throw exception?
143. https://apisyouwonthate.com/blog/idemptoency-keys/ Error on idempotent retry after success
144. Implementing Idempotency • Decide on the endpoints which need it • Select appropriate key cache TTLs • Document idempotent operations in your API • Allow your users to trust an operation is idempotent
145. Implementing Idempotency • Decide on the endpoints which need it • Select appropriate key cache TTLs • Document idempotent operations in your API • Allow your users to trust an operation is idempotent • Decide on the endpoints which need it • Select appropriate key cache TTLs • Document idempotent operations in your API
147. Registered As Canonical Username User ID Bigbird bigbird 123
148. Registered As Canonical Username User ID Bigbird bigbird 123
149. Registered As Canonical Username User ID Bigbird bigbird 123 BIGBIRD bigbird
150. Registered As Canonical Username User ID Bigbird bigbird 123 BIGBIRD bigbird
151. BiGbIrD Bigbird BIGBIRD BigBird getCanonicalUsername(user) bigbird
152. BiGbIrD Bigbird BIGBIRD BigBird getCanonicalUsername(user) bigbird
153. Registered As Canonical Username User ID Bigbird bigbird 123
154. Registered As Canonical Username User ID Bigbird bigbird 123 ᴮᴵᴳᴮᴵᴿᴰ
155. Registered As Canonical Username User ID Bigbird bigbird 123 ᴮᴵᴳᴮᴵᴿᴰ u'\u1d2e\u1d35\u1d33\u1d2e\u1d35\u1d3f\u1d30'
156. Registered As Canonical Username User ID Bigbird bigbird 123 ᴮᴵᴳᴮᴵᴿᴰ BIGBIRD u'\u1d2e\u1d35\u1d33\u1d2e\u1d35\u1d3f\u1d30'
157. Registered As Canonical Username User ID Bigbird bigbird 123 ᴮᴵᴳᴮᴵᴿᴰ BIGBIRD u'\u1d2e\u1d35\u1d33\u1d2e\u1d35\u1d3f\u1d30' 456
158. 1. Sign up with unicode username
159. 2. Trigger “forgot password” flow
160. 3. Submit “Forgot password” form
161. 3. Submit “Forgot password” form 🚨🚨🚨
162. 4. Take over account!
163. • getCanonicalUsername relied on nodeprep.prepare • Input string not being valid Unicode 3.2 meant nodeprep.prepare was no longer idempotent!
 • Fixed by double-checking username, and subsequently a library update What went wrong?
164. An operation is idempotent if applying it multiple times has the same effect as applying it once.
165. • Relying on server side hashes alone to identify repeat requests is risky • Make the client do the work! • Cache TTL appropriate to your use case • Don’t worry about verbs that are already idempotent • But don’t forget about DELETE! • Replay behaviour - document your choice! Takeaways
166. Boring Is Good!
167. eye-dum-po-ten-see
168. Thank you! 🌍 🌐 📧 Conroyp.com @conroyp paul@conroyp.com