Building APIs for an unreliable world

Networks and computing systems are unreliable. Connections drop or slow to a crawl. Hard drives break and systems turn off without warning. These issues affect public and private APIs alike, but can spell disaster for microservices, as unreliability can lead to inconsistency. In this talk, we will explore strategies for designing APIs that can help mitigate unreliability. We will look at how the details of an implementation can influence resilience. Finally, we will see how design and implementation influence each other for creating an API that is best suited to tackle unreliability.

Transcript

1. Building APIs For An unreliable World @jrbowes Unreliable
2. FIND ME github.com/jbowes twitter.com/jrbowes James Bowes ABOUT ME • Technical Lead @ www.manifold.co • Loves cats (dogs are great, too!) • Mostly a backend developer @jrbowes
3. @jrbowes THE SAMPLE PLATTER Four delicious APIs, each with its own: • Use case • Spectacular failure • Pragmatic solution • Useful lesson
4. @jrbowes Photo by Samuel Zeller on Unsplash
5. I Have Made Misteaks @jrbowes ...So You Won’t Have To!
6. I Have Made Mistakes @jrbowes ...So You Won’t Have To!
7. @jrbowes Photo by Rodolfo Mari on Unsplash WHAT IS ? UNRELIABLE
8. THE NETWORK @jrbowes Photo by Amy Elting on Unsplash WHAT IS ? UNRELIABLE
9. STORAGE WHAT IS ? UNRELIABLE @jrbowes Photo by Vincent Botta on Unsplash
10. SERVERS WHAT IS ? UNRELIABLE @jrbowes Photo by Fancycrave on Unsplash
11. SOFTWARE WHAT IS ? UNRELIABLE @jrbowes Photo by Markus Spiske on Unsplash
12. @jrbowes Photo by Joel Muniz on Unsplash WHAT IS ? UNRELIABLE WE ARE.
13. WHAT IS ? UNRELIABLE ANIMALS @jrbowes Photo by Gary Bendig on Unsplash 114 RACCOON RELATED OUTAGES (https://cybersquirrel1.com/)
14. File Transfers @jrbowes Photo by Samuel Zeller on Unsplash
15. @jrbowes What We Built • An on-premises systems management and software distribution product • Let users upload their own Linux packages File Transfers
16. @jrbowes How It Failed • People are impatient • Users had very large uploads • Disks filled up File Transfers
17. @jrbowes How We Fixed It • Chunked transfers • A new api to query what the server had • Resumable uploads File Transfers
18. Anticipate The User’s Needs @jrbowes
19. Collaborative Editing @jrbowes Photo by rawpixel on Unsplash
20. @jrbowes What We Built • A Realtime Backend-as-a-Service • Built-in identity with OAuth (GitHub, Twitter) • Shared tree-based data structure Collaborative Editing
21. @jrbowes How It Failed • The most recent change always won • ...Even if that change was made against an old state Collaborative Editing
22. @jrbowes Operational Transformation Com m utative Replicated Data Types Convergent Replicated Data Types Optimistic Locking Pessimistic Locking
23. @jrbowes How We Fixed It • Target common use cases • Append on arrays, insert on maps • Optimistic locking (Conditional set) Collaborative Editing
24. @jrbowes What We Would Have Done • Implement all those cool data structures! • … As opt-in features Collaborative Editing
25. Contention? Use Conditional Updates @jrbowes
26. Purchases @jrbowes Photo by Ramiro Mendes on Unsplash
27. @jrbowes What We Built • An API to buy SaaS products (databases, etc) • Every call == Purchases
28. @jrbowes How It Failed • It didn’t! Purchases
29. @jrbowes How It Failed • It didn’t! Purchases • ...until it did • A failure state could charge a customer multiple times
30. @jrbowes How We Fixed It • Repeated calls with the same unique key achieve the same result • Idempotency! • PUT vs POST Purchases
31. At Least Once At Least Once At Least Once At Least Once @jrbowes At Most Once
32. Pick Your Delivery Guarantees @jrbowes
33. Coordinating Responses @jrbowes Photo by Ruben Mishchuk on Unsplash
34. @jrbowes What We Built • Lots of things! • Any system using (micro)services • Gateway/aggregation layer Coordinating Responses
35. @jrbowes How It Failed • Constructing a response required N services • One service fails? Error the whole response Coordinating Responses
36. @jrbowes How We Fixed It • Evaluate the API and data • Denormalize where possible • Make fields optional and omit on error • The microservice implementation is leaky Coordinating Responses
37. Tolerate Service Failure @jrbowes
38. @jrbowes
39. • Anticipate The User’s Needs • Contention? Use Conditional Updates • Pick Your Delivery Guarantees • Tolerate Service Failure @jrbowes
40. FIND ME github.com/jbowes twitter.com/jrbowes Thank you! @jrbowes