Getting API design right

This slide deck discusses some essential guiding principles regarding good API design, i.e., creating a successful API. It starts with a little counter-example that probably looks just too familiar for many of us.

After that, first I discuss the problems of the counter-example, especially how it feels to the users - the developers who use the API - in terms of DX (developer experience - UX for APIs). Then, I look back into some seminal computer science papers to understand what we can learn from them regarding today's API design.

In the next part, I apply the principles just learnt before to the case study from the beginning. The key of that part is to illustrate the train of thought and guiding principles while designing the API. The details of the resulting API can be discussed of course. If the same train of thought would lead you to a bit different API in your individual context, that's perfectly fine ... ;)

Finally, I added a few complementing thoughts to complete the picture. As always, the voice track is missing which contains a big part of the content. Sorry about that! But I hope that the slides as they are still provide some value for you.

Transcript

1. Getting API design right Some foundational considerations and what we can learn from them Uwe Friedrichsen – codecentric AG – 2013-2019
2. Uwe Friedrichsen CTO @ codecentric https://twitter.com/ufried https://www.speakerdeck.com/ufried https://medium.com/@ufried
3. How can I create a good API?
4. The naïve approach ... seen more often than most of us dare to admit
5. No problem, dude Let me quickly create that API for you
6. Let us first expose a few endpoints With annotations that’s a breeze Let’s not forget to add a version to the API Just to be safe regarding future changes
7. Case study (Very simple) eCommerce shop • Implements the core functionalities • Search & show • Add to shopping cart • Checkout • Shipment • Customer self-care • Product catalog maintenance • Payments only touched as black box • No recommendations, etc.
8. * * * * Customer • name • address • payment methods E Order • customer • payment information • shipping information • order items • product • quantity E Product • name • description • image(s) • price • items in stock • packaging information (size, weight, special care) E
9. CustomerService ProductService • Search/show CustomerService • Customer self care S Customer OrderService • Add to shopping cart S ProductService • Search/show • Product catalog maintenance S CheckoutService • Checkout S ShipmentService • Shipment S Order Product
10. # Product search products IN: search criteria OUT: list of product ids read product IN: product id OUT: product details edit product IN: product details, OPTIONAL product id # for update OUT: product id # Customer find customer IN: search criteria OUT: customer id # exact matches only read customer IN: customer id OUT: customer details edit customer IN: customer details, OPTIONAL customer id # for update OUT: customer id
11. # Order find order IN: search criteria OUT: order id # exact matches only read order IN: order id OUT: order details edit order IN: order details, OPTIONAL: order id # for update OUT: order id # Checkout trigger checkout IN: order id, customer id # Shipment trigger shipment IN: order id, customer id # Version API version: 1.0
12. That was easy, wasn’t it Now about the juicy parts: let’s talk technology …
13. We could use HTTP/JSON (and call it “RESTful”) Exposing your data model with a CRUD-like interface, or your internal flows, or both Or maybe we should use GraphQL. It’s 2019 after all! Exposing your data model with a SQL-like interface Or we could use Events with Kafka. We could even add CQRS Exposing your internal flows, or your data model with a CRUD-like interface, or both Disclaimer: These are all great technologies. They are just not the focus of API design, and especially API design should not start with them
14. And what about user documentation?
15. No problem, dude We have Swagger for that …
18. Properties of the API design • API coupled to internal concepts and implementation • Internal data structures exposed on API • Internal flows exposed on API • Internal changes usually entail API changes • API users need to update their code after internal changes • Weak encapsulation • API users need to understand internal concepts • Low-level API (often with little documentation support) • High intellectual load for API users • Cumbersome to use from an API user’s point of view
19. Source: https://www.youtube.com/watch?v=RT_3BSaHce8
20. What is the problem?
21. APIs are forever
22. “Once customers started building their applications and systems using our APIs, changing those APIs becomes impossible, as we would be impacting our customer’s business operations if we would do so. We knew that designing APIs was a very important task as we’d only have one chance to get it right.” -- Werner Vogels (CTO Amazon) Source: https://www.allthingsdistributed.com/2016/03/10-lessons-from-10-years-of-aws.html
23. APIs are forever • You cannot change a (successful) API once you released it • Your customers have more important things to do than wasting their time with your design deficiencies • You can only change your API without any problems if nobody uses it – but then you should rather discontinue it • Versions give you the illusion you could change your API • You can’t force your customers to switch to the new version • Either you need to support all API versions forever • Or you can only apply backwards compatible changes (but then you do not need API versioning anyway)
24. Avoid API versioning Versions give you the deceptive illusion you could change your API any time
25. Do not limit your options early
26. Technology shapes (and limits) the way you reason about APIs
27. Early technology determination • Language shapes the way you reason about the world • Known as Sapir-Whorf hypothesis * • Same is true for technology and API design • With REST everything looks like a resource • With GraphQL everything looks like a data model • With Kafka everything looks like an event • … • Will distract you or might even keep you from finding the best possible API for your users * See, e.g. https://en.wikipedia.org/wiki/Linguistic_relativity
28. The right moment for technology in API design 1. Understand the problem and needs of the user 2. Reason about an adequate API from a user’s perspective 3. Decide about the technology 4. Adapt your API design to the technology chosen
29. Do not limit your options upfront without an important reason
30. DX is the new UX
31. Developer Experience (DX) • Developers are the users of an API • Developers expect a great experience from an API • Aligned with their mental model • Easy to get started, easy to test, easy to use • Great “look & feel”, great documentation • Advanced concepts can be picked up as needed • DX becomes vital in the context of digital transformation • Decides about success or demise for API-centered offerings
32. In a competing marketplace of API-centered offerings those with the best DX will eventually survive
33. Peering at a leader of API-centered business
39. Companies like Stripe set the future baseline for DX, i.e, your developers will expect at least that level of experience
40. This is DX This is not Know where you are with your API
41. Let us revisit the API design with the additional knowledge How good is it in terms of developer experience (DX)?
42. DX criterion Naïve API design Stability (versioning not needed) Alignment (with user’s mental model) Learnability (advanced concepts) Consistency (same concepts for similar tasks) Approachability (getting started) ✘ O ✘ ✘ O
43. Findings • Poor developer experience • Illusory API consistency (same concepts for different tasks)
44. How can we do better?
45. Let’s do a bit of research ...
46. On the criteria to be used in decomposing systems into modules by David L. Parnas [Par 1972]
47. “The effectiveness of a ‘modularization’ is dependent upon the criteria used in dividing the system into modules.” “The second decomposition was made using "information hiding" as a criterion. [...] Every module in the second decomposition is characterized by its knowledge of a design decision which it hides from all others. Its interface or definition was chosen to reveal as little as possible about its inner workings.” “There are a number of design decisions which are questionable and likely to change under many circumstances. [...] By looking at these changes we can see the differences between the two modularizations.” [Par 1972]
48. Separation of concerns One concept/decision per module Information hiding Reveal as little as possible about internal implementation + Better changeability Changes are kept local Independent teams Teams can easier work independently on different modules Easier to comprehend Modules can be understood on their own easier
49. Separation of concerns One concept/decision per module Information hiding Reveal as little as possible about internal implementation + Better changeability Changes are kept local Independent teams Teams can easier work independently on different modules Easier to comprehend Modules can be understood on their own easier Stable API Usage-oriented API
50. Implementation (volatile) API User (expects stability) I A U Usage-driven API • User context met • Strong encapsulation • High stability ✓ I A U Implementation-driven API • User context not met • Weak encapsulation • Little stability ✘ I A U Driver-independent API • User context not met • Unclear encapsulation • Unclear stability ?
51. Information distribution aspects of design methodology by David L. Parnas [Par 1971]
52. “The connections between modules are the assumptions which the modules make about each other. In most systems we find that these connections are much more extensive than the calling sequences and control block formats usually shown in system structure descriptions. We now consider making a change in the completed system. [...] We may make only those changes which do not violate the assumptions made by other modules about the module being changed. In other words, a single module may be changed only as long as the ‘connections’ still ‘fit’. Here, too, we have a strong argument for making the connections contain as little information as possible.” [Par 1971]
53. The tar pit by Frederick P. Brooks, Jr. (taken from the “The mythical man-month”) [Bro 1995]
54. APIs are the visible surface of a (re-)usable asset Thus, everything written in “The tar pit” also applies to API
55. 1x Program The original module, suitable for the context it was created for 3x Programming Product A generalized module, suitable for multiple contexts A module, ready to be used in an ecosystem of interacting modules A (re-)usable module, that provides a general solution for a problem 3x Programming System 9x Programming Systems Product Usage-driven API Usable API
56. 1x Program 9x Programming Systems Product Completeness of functionality • Hardening implementation • Handling of edge cases • Thorough testing • Design Documentation Completeness of accessibility • Precise interface definition • Clear behavioral contract • Thorough integration (or alike) testing • API Documentation
57. Creating a good API requires a lot of effort Only go for an API if you are willing to render the efforts required
58. Research findings • Go for usage-driven APIs • Supports stability • Aligned with user’s mental model • Go for minimal APIs (“information hiding”) • Smaller likelihood of broken user assumptions • Do not underestimate the efforts to create a good API • Clear and easy to grasp concepts • Consistent usage of concepts throughout the API • Good documentation to make assumptions explicit • Easy discovery and access • Extensive testing – ideally providing contract test suites
59. And now?
60. Design APIs from a user’s point of view Keep APIs minimal
61. Case study (Very simple) eCommerce shop • Implements the core functionalities • Search & show • Add to shopping cart • Checkout • Shipment • Customer self-care • Product catalog maintenance • Payments only touched as black box • No recommendations, etc.
62. Design considerations Design from a user’s point of view. Keep it minimal. What are the concepts a user is interested in?
63. * * * * Customer • name • address • payment methods E Order • customer • payment information • shipping information • order items • product • quantity E Product • name • description • image(s) • price • items in stock • packaging information (size, weight, special care) E
64. Search & Show Add to shopping cart Checkout Shipment Customer self-care Product catalog maintenance
65. Search & Show Add to shopping cart Checkout API user Find the desired product(s) Buy the product(s) in a straightforward manner Shipment Customer self-care Product catalog maintenance
66. Search & Show Add to shopping cart Checkout Customer self-care API user Find the desired product(s) Buy the product(s) in a straightforward manner Not actually interested in (“necessary evil”) Shipment Product catalog maintenance
67. Search & Show Add to shopping cart Checkout Shipment Customer self-care Product catalog maintenance API user Find the desired product(s) Buy the product(s) in a straightforward manner Not actually interested in (“necessary evil”) Not relevant for API user
68. Design considerations Design from a user’s point of view. Keep it minimal. What are the concepts a user is interested in? 1. Finding the desired product(s) 2. Buying the product(s) in a straightforward manner 3. Managing personal data - Keeping control of the process
69. Design considerations • Finding the desired product • Multiple approaches conceivable • Start with a minimal standard implementation • Leave API open for extension • Do not include functionality for “potential future use”
70. # Search search products IN: search criteria OUT: number of hits, list of products # including core details OR product details # exact match
71. Additional reasoning • GraphQL could be a suitable implementation technology • Search is a sweet spot of GraphQL • User can control the product details returned • But avoid coupling with internal data model • Only expose an interface data model • No direct connection to internal data model • Manual mapping required • Otherwise you bound your implementation to your API!
72. Design considerations • Buying the product in a straightforward manner • Client shall control user interaction patterns • No interaction order should be determined by the API • Most important functionality à keep it simple and stable • Should not be affected by internal changes
73. # Search search products IN: search criteria OUT: number of hits, list of products # including core details OR product details # exact match # Buying submit order IN: list of order items, OPTIONAL customer id, OPTIONAL payment data, OPTIONAL shipment data OUT: order number
74. Additional reasoning • Decouple API from (hopefully) revenue increasing measures • Must not break order API, no matter what we add internally • Provide product token that encodes pricing details, etc. • Might want to provide functionality to read submitted order • Not part of core API (“keep it minimal”) • Client might need temporary persistence mechanism • E.g., to temporarily persist their shopping carts • Will be implemented if requirements are clearly understood
75. # Search search products IN: search criteria OUT: number of hits, list of products # including core details OR product details # exact match # product includes product token # (encodes pricing information, …) # Buying submit order IN: list of order items, OPTIONAL customer id, OPTIONAL payment data, OPTIONAL shipment data OUT: order number # order item: product token, # quantity
76. Design considerations • Managing personal data • Driven by user demands, not by internal data model • User typically just wants to change specific data parts • Create API functions for different user demands • Makes decoupling from internal data representation easier
77. # Customer self-service read customer address IN: customer id OUT: customer address edit customer address IN: customer address, OPTIONAL customer id OUT: customer id # same for customer name, email, # payment information, etc.
78. Additional reasoning • Customer data API easily extensible • E.g., adding API function for multiple address types (original function will change invoice and shipment address) • E.g., adding API function to set default address per type (original API function will also set new address as new default invoice and shipment address) • … • GraphQL could be a suitable technology • Again – avoid coupling with internal data model!
79. Nice, but is this API design any better? Again, how good is it in terms of developer experience (DX)?
80. DX criterion Naïve API design Stability (versioning not needed) Alignment (with user’s mental model) Learnability (advanced concepts) Consistency (same concepts for similar tasks) Approachability (getting started) ✘ ? ✓ O ✘ Usage-driven design ✓ (✓) ✓ ✘ O Not determined by design only Not determined by API design
81. Findings • Good developer experience • Not only a design task (e.g., good documentation needed) Side note: It is not a black or white topic, even not in this (simple) example
82. Well, anything else?
83. Limitations of the case study • Only partially suitable to illustrate ideas • Compromise between simplicity and suitability • Should (hopefully) still be good enough to grasp ideas
84. Do not underestimate the efforts • Designing an API is hard work • Finding the concepts to externalize • Always thinking from the API user, not the implementation • Keeping it minimal • Significant part of the efforts of Fred Brooks’ “Rule of 9” • Include them in your plans and budget • Event storming and domain-driven design can support • Still, they are not a panacea • It will still be hard work
85. “Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.” -- Antoine de Saint-Exupery
86. Complementing activities (1/2) • Collect feedback from your users • You will need it to get your API right • Test your API extensively • Contributes to runtime stability • Provide consumer-driven contract test suites • Augment with mock implementations • Reduce implicit assumptions • Contributes to DX
87. Complementing activities (2/2) • Make your APIs secure • Will make them more complex • Work hard to keep the additional mental load small • Provide great user-centered documentation • Always ask yourself what a user of your API needs to know • Provide sandboxes, etc. to increase approachability • Only developers who love your API will make you successful
88. Internal APIs Treat internal APIs like external APIs • You could put less effort intro it … • … but you will pay the price for it and it will be higher • Effects of sloppy design and development of internal APIs • Higher coupling between teams • More unwanted assumptions and dependencies • Slows down development and evolution • More errors in development, test and production • One of the reasons why everyone fears integration projects
89. Events, commands, CQRS, etc. • Events and commands also form an API! • All requirements of API design and DX also apply to them • Side note: How do you solve the versioning problem? • Do not use them because they are fashionable • Only use them if they add more value to the solution than alternative solutions • Do not confuse business/domain and technical events • Events discovered with event storming or DDD do not need to be implemented using technical events
90. Wrap-up
91. Wrap-up • APIs are forever – avoid versioning • Do not start with technology • DX is the new UX • Go for usage-centered APIs • Keep APIs minimal • Accept the efforts (or pay them multiple times later) • Events and commands also form APIs
92. References [Bro 1995] Frederick P. Brooks, jr., “The tar pit”, from “The mythical man-month”, anniversary edition 1995 [Eva 2004] Eric Evans, “Domain-Driven Design – Tackling complexity in the heart of software”, Addison-Wesley, 2004 [Par 1971] David L. Parnas, “Information distribution aspects of design methodology”, Carnegie Mellon University, Research Showcase @CMU, 1971 [Par 1972] David L. Parnas, “On the criteria to be used in decomposing systems into modules”, Communications of the ACM, Vol. 15, No. 12, December 1972, pp. 1053-1058 [Ste 1974] W. P. Stevens, G. J. Myers, and L. L. Constantine, “Structured design”, IBM Systems Journal, Vol 13, No. 2, 1974
93. Uwe Friedrichsen CTO @ codecentric https://twitter.com/ufried https://www.speakerdeck.com/ufried https://medium.com/@ufried