Getting API design right

Slide 1

Slide 1 text

Getting API design right Some foundational considerations and what we can learn from them Uwe Friedrichsen – codecentric AG – 2013-2019

Slide 2

Slide 2 text

Uwe Friedrichsen CTO @ codecentric https://twitter.com/ufried https://www.speakerdeck.com/ufried https://medium.com/@ufried

Slide 3

Slide 3 text

How can I create a good API?

Slide 4

Slide 4 text

The naïve approach ... seen more often than most of us dare to admit

Slide 5

Slide 5 text

No problem, dude Let me quickly create that API for you

Slide 6

Slide 6 text

Let us first expose a few endpoints With annotations that’s a breeze Let’s not forget to add a version to the API Just to be safe regarding future changes

Slide 7

Slide 7 text

Case study (Very simple) eCommerce shop • Implements the core functionalities • Search & show • Add to shopping cart • Checkout • Shipment • Customer self-care • Product catalog maintenance • Payments only touched as black box • No recommendations, etc.

Slide 8

Slide 8 text

* * * * Customer • name • address • payment methods E Order • customer • payment information • shipping information • order items • product • quantity E Product • name • description • image(s) • price • items in stock • packaging information (size, weight, special care) E

Slide 9

Slide 9 text

CustomerService ProductService • Search/show CustomerService • Customer self care S Customer OrderService • Add to shopping cart S ProductService • Search/show • Product catalog maintenance S CheckoutService • Checkout S ShipmentService • Shipment S Order Product

Slide 10

Slide 10 text

# Product search products IN: search criteria OUT: list of product ids read product IN: product id OUT: product details edit product IN: product details, OPTIONAL product id # for update OUT: product id # Customer find customer IN: search criteria OUT: customer id # exact matches only read customer IN: customer id OUT: customer details edit customer IN: customer details, OPTIONAL customer id # for update OUT: customer id

Slide 11

Slide 11 text

# Order find order IN: search criteria OUT: order id # exact matches only read order IN: order id OUT: order details edit order IN: order details, OPTIONAL: order id # for update OUT: order id # Checkout trigger checkout IN: order id, customer id # Shipment trigger shipment IN: order id, customer id # Version API version: 1.0

Slide 12

Slide 12 text

That was easy, wasn’t it Now about the juicy parts: let’s talk technology …

Slide 13

Slide 13 text

We could use HTTP/JSON (and call it “RESTful”) Exposing your data model with a CRUD-like interface, or your internal flows, or both Or maybe we should use GraphQL. It’s 2019 after all! Exposing your data model with a SQL-like interface Or we could use Events with Kafka. We could even add CQRS Exposing your internal flows, or your data model with a CRUD-like interface, or both Disclaimer: These are all great technologies. They are just not the focus of API design, and especially API design should not start with them

Slide 14

Slide 14 text

And what about user documentation?

Slide 15

Slide 15 text

No problem, dude We have Swagger for that …

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

Properties of the API design • API coupled to internal concepts and implementation • Internal data structures exposed on API • Internal flows exposed on API • Internal changes usually entail API changes • API users need to update their code after internal changes • Weak encapsulation • API users need to understand internal concepts • Low-level API (often with little documentation support) • High intellectual load for API users • Cumbersome to use from an API user’s point of view

Slide 19

Slide 19 text

Source: https://www.youtube.com/watch?v=RT_3BSaHce8

Slide 20

Slide 20 text

What is the problem?

Slide 21

Slide 21 text

APIs are forever

Slide 22

Slide 22 text

“Once customers started building their applications and systems using our APIs, changing those APIs becomes impossible, as we would be impacting our customer’s business operations if we would do so. We knew that designing APIs was a very important task as we’d only have one chance to get it right.” -- Werner Vogels (CTO Amazon) Source: https://www.allthingsdistributed.com/2016/03/10-lessons-from-10-years-of-aws.html

Slide 23

Slide 23 text

APIs are forever • You cannot change a (successful) API once you released it • Your customers have more important things to do than wasting their time with your design deficiencies • You can only change your API without any problems if nobody uses it – but then you should rather discontinue it • Versions give you the illusion you could change your API • You can’t force your customers to switch to the new version • Either you need to support all API versions forever • Or you can only apply backwards compatible changes (but then you do not need API versioning anyway)

Slide 24

Slide 24 text

Avoid API versioning Versions give you the deceptive illusion you could change your API any time

Slide 25

Slide 25 text

Do not limit your options early

Slide 26

Slide 26 text

Technology shapes (and limits) the way you reason about APIs

Slide 27

Slide 27 text

Early technology determination • Language shapes the way you reason about the world • Known as Sapir-Whorf hypothesis * • Same is true for technology and API design • With REST everything looks like a resource • With GraphQL everything looks like a data model • With Kafka everything looks like an event • … • Will distract you or might even keep you from finding the best possible API for your users * See, e.g. https://en.wikipedia.org/wiki/Linguistic_relativity

Slide 28

Slide 28 text

The right moment for technology in API design 1. Understand the problem and needs of the user 2. Reason about an adequate API from a user’s perspective 3. Decide about the technology 4. Adapt your API design to the technology chosen

Slide 29

Slide 29 text

Do not limit your options upfront without an important reason

Slide 30

Slide 30 text

DX is the new UX

Slide 31

Slide 31 text

Developer Experience (DX) • Developers are the users of an API • Developers expect a great experience from an API • Aligned with their mental model • Easy to get started, easy to test, easy to use • Great “look & feel”, great documentation • Advanced concepts can be picked up as needed • DX becomes vital in the context of digital transformation • Decides about success or demise for API-centered offerings

Slide 32

Slide 32 text

In a competing marketplace of API-centered offerings those with the best DX will eventually survive

Slide 33

Slide 33 text

Peering at a leader of API-centered business

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

Companies like Stripe set the future baseline for DX, i.e, your developers will expect at least that level of experience

Slide 40

Slide 40 text

This is DX This is not Know where you are with your API

Slide 41

Slide 41 text

Let us revisit the API design with the additional knowledge How good is it in terms of developer experience (DX)?

Slide 42

Slide 42 text

DX criterion Naïve API design Stability (versioning not needed) Alignment (with user’s mental model) Learnability (advanced concepts) Consistency (same concepts for similar tasks) Approachability (getting started) ✘ O ✘ ✘ O

Slide 43

Slide 43 text

Findings • Poor developer experience • Illusory API consistency (same concepts for different tasks)

Slide 44

Slide 44 text

How can we do better?

Slide 45

Slide 45 text

Let’s do a bit of research ...

Slide 46

Slide 46 text

On the criteria to be used in decomposing systems into modules by David L. Parnas [Par 1972]

Slide 47

Slide 47 text

“The effectiveness of a ‘modularization’ is dependent upon the criteria used in dividing the system into modules.” “The second decomposition was made using "information hiding" as a criterion. [...] Every module in the second decomposition is characterized by its knowledge of a design decision which it hides from all others. Its interface or definition was chosen to reveal as little as possible about its inner workings.” “There are a number of design decisions which are questionable and likely to change under many circumstances. [...] By looking at these changes we can see the differences between the two modularizations.” [Par 1972]

Slide 48

Slide 48 text

Separation of concerns One concept/decision per module Information hiding Reveal as little as possible about internal implementation + Better changeability Changes are kept local Independent teams Teams can easier work independently on different modules Easier to comprehend Modules can be understood on their own easier

Slide 49

Slide 49 text

Slide 50

Slide 50 text

Implementation (volatile) API User (expects stability) I A U Usage-driven API • User context met • Strong encapsulation • High stability ✓ I A U Implementation-driven API • User context not met • Weak encapsulation • Little stability ✘ I A U Driver-independent API • User context not met • Unclear encapsulation • Unclear stability ?

Slide 51

Slide 51 text

Information distribution aspects of design methodology by David L. Parnas [Par 1971]

Slide 52

Slide 52 text

“The connections between modules are the assumptions which the modules make about each other. In most systems we find that these connections are much more extensive than the calling sequences and control block formats usually shown in system structure descriptions. We now consider making a change in the completed system. [...] We may make only those changes which do not violate the assumptions made by other modules about the module being changed. In other words, a single module may be changed only as long as the ‘connections’ still ‘fit’. Here, too, we have a strong argument for making the connections contain as little information as possible.” [Par 1971]

Slide 53

Slide 53 text

The tar pit by Frederick P. Brooks, Jr. (taken from the “The mythical man-month”) [Bro 1995]

Slide 54

Slide 54 text

APIs are the visible surface of a (re-)usable asset Thus, everything written in “The tar pit” also applies to API

Slide 55

Slide 55 text

1x Program The original module, suitable for the context it was created for 3x Programming Product A generalized module, suitable for multiple contexts A module, ready to be used in an ecosystem of interacting modules A (re-)usable module, that provides a general solution for a problem 3x Programming System 9x Programming Systems Product Usage-driven API Usable API

Slide 56

Slide 56 text

1x Program 9x Programming Systems Product Completeness of functionality • Hardening implementation • Handling of edge cases • Thorough testing • Design Documentation Completeness of accessibility • Precise interface definition • Clear behavioral contract • Thorough integration (or alike) testing • API Documentation

Slide 57

Slide 57 text

Creating a good API requires a lot of effort Only go for an API if you are willing to render the efforts required

Slide 58

Slide 58 text

Research findings • Go for usage-driven APIs • Supports stability • Aligned with user’s mental model • Go for minimal APIs (“information hiding”) • Smaller likelihood of broken user assumptions • Do not underestimate the efforts to create a good API • Clear and easy to grasp concepts • Consistent usage of concepts throughout the API • Good documentation to make assumptions explicit • Easy discovery and access • Extensive testing – ideally providing contract test suites

Slide 59

Slide 59 text

And now?

Slide 60

Slide 60 text

Design APIs from a user’s point of view Keep APIs minimal

Slide 61

Slide 61 text

Slide 62

Slide 62 text

Design considerations Design from a user’s point of view. Keep it minimal. What are the concepts a user is interested in?

Slide 63

Slide 63 text

Slide 64

Slide 64 text

Search & Show Add to shopping cart Checkout Shipment Customer self-care Product catalog maintenance

Slide 65

Slide 65 text

Search & Show Add to shopping cart Checkout API user Find the desired product(s) Buy the product(s) in a straightforward manner Shipment Customer self-care Product catalog maintenance

Slide 66

Slide 66 text

Search & Show Add to shopping cart Checkout Customer self-care API user Find the desired product(s) Buy the product(s) in a straightforward manner Not actually interested in (“necessary evil”) Shipment Product catalog maintenance

Slide 67

Slide 67 text

Search & Show Add to shopping cart Checkout Shipment Customer self-care Product catalog maintenance API user Find the desired product(s) Buy the product(s) in a straightforward manner Not actually interested in (“necessary evil”) Not relevant for API user

Slide 68

Slide 68 text

Design considerations Design from a user’s point of view. Keep it minimal. What are the concepts a user is interested in? 1. Finding the desired product(s) 2. Buying the product(s) in a straightforward manner 3. Managing personal data - Keeping control of the process

Slide 69

Slide 69 text

Design considerations • Finding the desired product • Multiple approaches conceivable • Start with a minimal standard implementation • Leave API open for extension • Do not include functionality for “potential future use”

Slide 70

Slide 70 text

# Search search products IN: search criteria OUT: number of hits, list of products # including core details OR product details # exact match

Slide 71

Slide 71 text

Additional reasoning • GraphQL could be a suitable implementation technology • Search is a sweet spot of GraphQL • User can control the product details returned • But avoid coupling with internal data model • Only expose an interface data model • No direct connection to internal data model • Manual mapping required • Otherwise you bound your implementation to your API!

Slide 72

Slide 72 text

Design considerations • Buying the product in a straightforward manner • Client shall control user interaction patterns • No interaction order should be determined by the API • Most important functionality à keep it simple and stable • Should not be affected by internal changes

Slide 73

Slide 73 text

# Search search products IN: search criteria OUT: number of hits, list of products # including core details OR product details # exact match # Buying submit order IN: list of order items, OPTIONAL customer id, OPTIONAL payment data, OPTIONAL shipment data OUT: order number

Slide 74

Slide 74 text

Additional reasoning • Decouple API from (hopefully) revenue increasing measures • Must not break order API, no matter what we add internally • Provide product token that encodes pricing details, etc. • Might want to provide functionality to read submitted order • Not part of core API (“keep it minimal”) • Client might need temporary persistence mechanism • E.g., to temporarily persist their shopping carts • Will be implemented if requirements are clearly understood

Slide 75

Slide 75 text

# Search search products IN: search criteria OUT: number of hits, list of products # including core details OR product details # exact match # product includes product token # (encodes pricing information, …) # Buying submit order IN: list of order items, OPTIONAL customer id, OPTIONAL payment data, OPTIONAL shipment data OUT: order number # order item: product token, # quantity

Slide 76

Slide 76 text

Design considerations • Managing personal data • Driven by user demands, not by internal data model • User typically just wants to change specific data parts • Create API functions for different user demands • Makes decoupling from internal data representation easier

Slide 77

Slide 77 text

# Customer self-service read customer address IN: customer id OUT: customer address edit customer address IN: customer address, OPTIONAL customer id OUT: customer id # same for customer name, email, # payment information, etc.

Slide 78

Slide 78 text

Additional reasoning • Customer data API easily extensible • E.g., adding API function for multiple address types (original function will change invoice and shipment address) • E.g., adding API function to set default address per type (original API function will also set new address as new default invoice and shipment address) • … • GraphQL could be a suitable technology • Again – avoid coupling with internal data model!

Slide 79

Slide 79 text

Nice, but is this API design any better? Again, how good is it in terms of developer experience (DX)?

Slide 80

Slide 80 text

Slide 81

Slide 81 text

Findings • Good developer experience • Not only a design task (e.g., good documentation needed) Side note: It is not a black or white topic, even not in this (simple) example

Slide 82

Slide 82 text

Well, anything else?

Slide 83

Slide 83 text

Limitations of the case study • Only partially suitable to illustrate ideas • Compromise between simplicity and suitability • Should (hopefully) still be good enough to grasp ideas

Slide 84

Slide 84 text

Do not underestimate the efforts • Designing an API is hard work • Finding the concepts to externalize • Always thinking from the API user, not the implementation • Keeping it minimal • Significant part of the efforts of Fred Brooks’ “Rule of 9” • Include them in your plans and budget • Event storming and domain-driven design can support • Still, they are not a panacea • It will still be hard work

Slide 85

Slide 85 text

“Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.” -- Antoine de Saint-Exupery

Slide 86

Slide 86 text

Complementing activities (1/2) • Collect feedback from your users • You will need it to get your API right • Test your API extensively • Contributes to runtime stability • Provide consumer-driven contract test suites • Augment with mock implementations • Reduce implicit assumptions • Contributes to DX

Slide 87

Slide 87 text

Complementing activities (2/2) • Make your APIs secure • Will make them more complex • Work hard to keep the additional mental load small • Provide great user-centered documentation • Always ask yourself what a user of your API needs to know • Provide sandboxes, etc. to increase approachability • Only developers who love your API will make you successful

Slide 88

Slide 88 text

Internal APIs Treat internal APIs like external APIs • You could put less effort intro it … • … but you will pay the price for it and it will be higher • Effects of sloppy design and development of internal APIs • Higher coupling between teams • More unwanted assumptions and dependencies • Slows down development and evolution • More errors in development, test and production • One of the reasons why everyone fears integration projects

Slide 89

Slide 89 text

Events, commands, CQRS, etc. • Events and commands also form an API! • All requirements of API design and DX also apply to them • Side note: How do you solve the versioning problem? • Do not use them because they are fashionable • Only use them if they add more value to the solution than alternative solutions • Do not confuse business/domain and technical events • Events discovered with event storming or DDD do not need to be implemented using technical events

Slide 90

Slide 90 text

Wrap-up

Slide 91

Slide 91 text

Wrap-up • APIs are forever – avoid versioning • Do not start with technology • DX is the new UX • Go for usage-centered APIs • Keep APIs minimal • Accept the efforts (or pay them multiple times later) • Events and commands also form APIs

Slide 92

Slide 92 text

References [Bro 1995] Frederick P. Brooks, jr., “The tar pit”, from “The mythical man-month”, anniversary edition 1995 [Eva 2004] Eric Evans, “Domain-Driven Design – Tackling complexity in the heart of software”, Addison-Wesley, 2004 [Par 1971] David L. Parnas, “Information distribution aspects of design methodology”, Carnegie Mellon University, Research Showcase @CMU, 1971 [Par 1972] David L. Parnas, “On the criteria to be used in decomposing systems into modules”, Communications of the ACM, Vol. 15, No. 12, December 1972, pp. 1053-1058 [Ste 1974] W. P. Stevens, G. J. Myers, and L. L. Constantine, “Structured design”, IBM Systems Journal, Vol 13, No. 2, 1974

Slide 93

Slide 93 text

Uwe Friedrichsen CTO @ codecentric https://twitter.com/ufried https://www.speakerdeck.com/ufried https://medium.com/@ufried