REST API Design Pitfalls

An entertaining tour of the most common REST API design mistakes (with examples):

Domain Leakage
Sensitive Data Exposure
Performance Mistakes
CQRS
PUT overload
Religious REST
Breaking Changes.
Collected with love from over 150 companies.

Speaker:
With two decades of experience, Victor is a Java Champion who has trained thousands of engineers in over 150 companies. For first-class training services, consultancy, and extensive educational YouTube videos, check out https://victorrentea.ro

As usual, the event will be live-streamed on YouTube, where it will also remain recorded: https://youtube.com/live/0XXwk647tKw

Transcript

1. TOP REST API DESIGN PITFALLS BY VICTOR RENTEA
2. 👋 I'm Victor Rentea 🇷🇴 Java Champion, PhD(CS) 18 years of coding in Java, .kt, .js/ts, .scala, .cs, .py ... 10 years of training at 150+ companies in 20+ countries on: - Architecture, Refactoring, Unit TesNng - Spring, Hibernate, Performance, ReacNve European SoRware CraRers Community (on meetup.com) YouTube.com/vrentea - past events & talks Life += 👰 + 👧 + 🙇 + 🐈 victorrentea.ro
3. A 84 VictorRentea.ro a training by 1. Everything is a Tradeoff. = Don't just Hear, Critically Think! è Understand the pros & cons in your context. 2. Why matters more than How. = Accept that context can change. è Document your decisions in, eg: ADR Laws of Architecture
4. A 87 VictorRentea.ro a training by REST API
5. A 88 VictorRentea.ro a training by Be conservative in what you do, be liberal in what you accept from others - Robustness Law, aka. Postel's Law (author of TCP/IP)
6. A 89 VictorRentea.ro a training by Seman&c Versioning SemVer.org v1.2.0 Major Version ++ on breaking change Minor Version ++ on new feature Patch ++ on bugfix
7. A 90 VictorRentea.ro a training by If you use v1.2.0, can you use instead: a) 1.2.1 b) 1.3.0 c) 1.1.0 ⬇ d) 2.0.1 SemVer Exercise Major Version ++ on breaking change Minor Version ++ on new feature Patch ++ on bugfix ✅ Yes. Probably... 🤞 ✅ Yes... 🤞 🚫 Probably no... 🤨 🚫 No
8. A 91 VictorRentea.ro a training by 1 2 3 4 5 6 7 8 9 10 PUT /user/{id}/prefs { // request: "firstName": "John", "lastName": "DOE", "email": "a/com", "phone": "+407129", "currency": "EUR" } One of EUR|USD|CHF Hack Lab 🧪 Cause a Breaking Change that would force clients to update { // another response: "date": "03/01/2023", "email": "a@b.com", "age": "37", "country": "ROU" } One of ROU|BGL|HUN ------ (no longer supported) |BEL [ ] fullName: { } - - (number) POST /users/ regex valid *required min=7char 2023-03-01 Number {email} "address": ... (required)
9. A 93 VictorRentea.ro a training by §Expose 2+ versions for a limited 4meframe - Pollutes provider's logic with if(ver=2)... §Iden4fy your clients via: - Authen5ca5on: tokens, API Keys ... - TraceIDs / logs - Check their past calls payloads – do they ever send me that field? §Convince your clients to upgrade 😳🙏😲🐗🥷🤬💰🤜🪓!^*%#: - Old version: Rate limit, Fee$, Warnings, Spurious Failures😫 §tl;dr major versioning sucks: avoid it! Major Version Upgrade: How To? 😬
10. A 94 VictorRentea.ro a training by 😏 To avoid breaking changes, I'll make my API "future proof"
11. A 95 VictorRentea.ro a training by { emails: ["a@b.com"], phones: [{type:"work", value: "..."}], attributes: [ {name:"age", value: 12}, {name:"gender", value:"M"}.. // we can add more keys tomorrow 😈 ] } Overly-Generic API = Unclear API 👏 You just brain-damaged your clients 🤯 // today I always have 1 email // today we only support work phone number To avoid breaking changes, I'll make my API "future proof"
12. A 96 VictorRentea.ro a training by §new app😱: om-v2.intra, with separate Git/CI/DB §per-service⭐: om.intra/v2/customer/{id} §per-endpoint: /customer/{id}/v2 (is this a monolith?) §per request: Content-Type: applica<on/json+v3 Versioning Scope
13. A 103 VictorRentea.ro a training by Ø Domain Model Freezes: ❄ as clients depend on it Ø Security Risk: 🔐 Excessive Data Exposure [API-SEC] Ø Domain Pollu4on with presentaIon: @JsonIgnore... Expose Domain Model in your API Only expose Data Transfer Objects (DTOs) in your API @GetMapping("...") // REST public Customer findById(... Opinions? 🧐 DON'T // Domain Model (± @Entity) public class Customer {...
14. A 104 VictorRentea.ro a training by Separate Contract from ImplementaCon Stable API for client(s) to rely on Evolving to simplify my implementaNon documented
15. A 105 VictorRentea.ro a training by §1. Serialize ORM @En1ty as JSON - Can trigger unwanted lazy-loading (+design leak⚠) èExpose dedicated DTOs §2. Only expose get-one-by-id: GET /products/{id} - Clients could do: for (id in [..]) {.. yourApi.get(id)} èExpose a bulk-retrieve: - GET /products/?id=1,3,5,8,1200 ⚠but URL.length <= 2000 chars - POST /products/get-many + body: [1,2...5000] - GET /products + body: [1,2,3..] (It's legal to send body in an HTTP GET since 2021😎) §3. API-in-the-middle - A à B (forwards C's response) à C è Bypass B: A à C API design to trash performance? network-call-in-a-loop anti-pattern
16. A 106 VictorRentea.ro a training by Dto ftw!
17. A 107 VictorRentea.ro a training by
18. A 109 VictorRentea.ro a training by Reusing the same DTO for GET + POST/PUT InventoryItemDto { "id": 13, "name": "Chair", "supplierName": "ACME", "supplierId": 78, "description": "Soft friend", "stock": 10, "status": "ACTIVE", "deactivationReason": null, "creationDate": "2021-10-01", "createdBy": "Wonder Woman" } { "id": null, "supplierName": null, . "status": null, . "deactivationReason": null, "creationDate": null, . "createdBy": null . } @GetMapping("{id}") InventoryItemDto get(id) { @PostMapping void create(InventoryItemDto) { Ab ß always null in create flow What's wrong? The Contract (OpenAPI) is: – misleading for clients 🥴 – confusing to implement – couples endpoints
19. A 110 VictorRentea.ro a training by InventoryItemDto { "id": 13, "name": "Chair", "supplierName": "ACME", "supplierId": 78, "description": "Soft friend", "stock": 10, "status": "ACTIVE", "deactivationReason": null, "creationDate": "2021-10-01", "createdBy": "Wonder Woman" } CQRS Dedicated Request/Response Structures = CQRS at the API Level CreateItemRequest { "name": "Chair", "supplierId": 78, "description": "Soft friend", "stock": 10 } GetItemResponse { "id": 13, "name": "Chair", "supplierName": "ACME", "supplierId": 78, "description": "Soft friend", "stock": 10, "status": "ACTIVE", "deactivationReason": null, "creationDate": "2021-10-01", "createdBy": "Wonder Woman" } @GetMapping("{id}") GetItemResponse get(id) { @PostMapping void create(CreateItemRequest){ @NotNull in dto package
20. A 111 VictorRentea.ro a training by CQRS Command/Query Responsibility SegregaCon Update Data Read Data = SeparaCon
21. A 112 VictorRentea.ro a training by Command/Query Responsibility SegregaCon Most people perceive soTware systems as stores of records: that they Create, Read, Update, Delete and Search As system grows complex: - READ: - aggregates (SUM..) or enriches the data (JOIN, API calls..) - ⭐ key concern: fast & available - UPDATE: - stores addi5onal metadata: createdBy=, ... - ⭐ key concern: preserve consistency CQRS = use separate WRITE / READ data models https://www.eventstore.com/blog/transcript-of-greg-youngs-talk-at-code-on-the-beach-2014-cqrs-and-event-sourcing
22. A 113 VictorRentea.ro a training by CQRS Levels CQRS at API level to Clarify Contract 👍 GetItemResponse -- query CreateItemRequest -- command CQRS at DB InteracCon to Improve Read Performance when using an ORM SELECT new dto.SearchResult(e.id, e.name) - fetch less ⚠ But change data via domain en55es Distributed CQRS for Performance & Availability Write to an SQL or Event Store but Read from: - Redis, Memcached,.. or Materialized Views, poten5ally stale⏱ - ElasCc Search, Mongo, ... updated async⏱ higher performance 🔽 Eventual⏱ Consistency More DB Indexes = faster reads but slower inserts 🔼 Strong Consistency
23. A 114 VictorRentea.ro a training by B B Q C Distributed CQRS UI READ DB ElasScSearch, Mongo, Redis, Memcache,.. WRITE DB SQL/Event Store Query (reads🔥) Command (writes) update sync result - Can store pre-built JSONs - No need for DB constraints - Horizontal scalable = autonomous component MQ by Greg Young, 2011 CQRS by Greg Young (BEST video)👉 hXps://www.youtube.com/watch?v=JHGkaShoyNs Events/CDC 💥 change
24. A 115 VictorRentea.ro a training by ⚠ WARNING ⚠ Many systems fit well with the notion of an information base that is updated in the same way that it's read. Adding Distributed CQRS (prev. slide) to such a system can add complexity, lower productivity, and add unwarranted risk to the project, even in the hands of a capable 🥷 team. CQRS is difficult to use well. - Martin Fowler Mainly due to async
25. A 116 VictorRentea.ro a training by §Availability+ of query when command is down §Performance+: spread queries to separate apps/DB instances But! CQRS means more than read replicas: §Different storage: in-memory (fast access), ES (full-text search) §Restructure data in warehouse for BI/ML/AI ~ ETLs §Pre-fetch / pre-aggregate response (eg: home page) §Read-projec4on of an event stream (in Event Sourcing) Why CQRS
26. A 117 VictorRentea.ro a training by Separate Commands from Queries Read Write CQRS
27. A 118 VictorRentea.ro a training by ? CQRS Should PUT (C) Return Data (Q) Only with strong arguments in an ADR. (FE convenience, unmeasured performance)
28. A 119 VictorRentea.ro a training by Task-Based UI A large CRUDe PUT vs.
29. A 120 VictorRentea.ro a training by Inventory Item id:13 Chair Name Description Supplier Supplier Cost (EUR) Stock Status Deactivation Reason Soft Friend ACME 120 10 ACTIVE Update Cancel PUT or PATCH /item/13 { "id": 13, "name": "Chair", "supplierId": 78, "description": "Soft Friend", "cost": 120, "count": 10, "status": "ACTIVE", "deactivationReason": null, "version": 17 //🔒 } Large Edit Screen What's wrong with this screen? PUT
30. A 121 VictorRentea.ro a training by 2° CONCURRENT UPDATES Inventory Item id:13 Chair Name Description Supplier Supplier Cost (EUR) Stock Status Deactivation Reason Soft Friend ACME 120 10 ACTIVE Update Cancel Large Edit Screen - Problems The user updates the descripSon If you change status ACTIVEàINACTIVE, user must provide a reason 1° BAD UX: not obvious rule Can cause: - data loss by blind overwrite - op6mis6c locking errors, frustraSng users: - pessimis6c locking (bo;lenecks) tracking who edits this record now in DB columns ItemSoldEvent arrived that decreased the stock , but meanwhile 3° Server must DIFF changes UNABLE TO SAVE Someone else already changed this item. Refresh the page and re-do your updates. OK🤬 👍 Avoid a Large PUT/PATCH CANNOT OPEN EDIT SCREEN Item under edit by vrentea since 3h ago. OK🤬
31. A 122 VictorRentea.ro a training by Name ^ Supplier Active Supplier Cost Stock Chair ACME 120 10 Armchair ACME 160 12 Table ACME 255 5 Sofa ACME 980 4 Inventory Item id:13 Chair Name Description Supplier Supplier Cost (EUR) Stock Status Deactivation Reason Soft Friend ACME 120 10 ACTIVE Update Cancel What do users usually do?
32. A 123 VictorRentea.ro a training by Task-Based UI https://cqrs.wordpress.com/documents/task-based-ui/ split by typical user👑 actions Name ^ Supplier Active Supplier Cost Stock Chair ACME 120 10 Armchair ACME 160 12 Table ACME 255 5 Sofa ACME 980 4 PUT /item/13/deactivate { reason } Deactivate Inventory Item Reason*: stopped manufacturing| Cancel Deactivate Update Supplier Cost New cost*: 120 Cancel OK PUT /item/13/cost { newCost } POST /item/13/sell { quantity:2 } (POST = not idempotent = not retryable) PUT /item/13/details { name: supplier: description:🦄 } ❌ Requires User Research 💖 ❌ More APIs ❌ More Screens è Full-stack 💖 ✅ Intentional, Semantic API & UX ✅ Less concurrency risk ✅ Cleaner impl. (no need to diff) sub-resource ✅ ac[on (verb) ✅ sub-resource ✅ action (verb) ✅
33. A 124 VictorRentea.ro a training by Religious REST Fallacy Can lead to: §Unequal distribuNon of complexity - GET🧠, POST🧠🧠🧠, PUT/PATCH 🤯, DELETE, Search🧠🧠 §LimiNng semanNcs. è Enhance it with: - Sub-resources: GET|PUT /item/13/cost - Ac4ons: POST /item/13/sell (verb) ⚠ But sNck to REST for as long as it's decent REST next level : Crafting domain driven web APIs By Julien Topçu This is still REST: https://martinfowler.com/articles/richardsonMaturityModel.html
34. A 125 VictorRentea.ro a training by REST PATCH PATCH /item/1 = partial update; payload: § Raw: skip unchanged fields = hard to parse { "status": "INACTIVE", "deactivationReason": "reason", "description": null // clear value // unchaged fields missing } § Using jsonpatch.com standard [ {"op":"set", "path":"/status", "value":"INACTIVE"}, {"op":"set", "path":"/deactivationReason", "value":"reason"}, {"op":"rem", "path":"/description"} ] Lacks Seman&cs 😵💫😵 Why do status, deacGvaGonReason and descripGon change at once??
35. A 126 VictorRentea.ro a training by Separate unrelated updates from one another by studying typical User💖 actions different endpoints/screens
36. A 127 VictorRentea.ro a training by Summary: REST API Design PiTalls •Breaking Changes è SemVer •Expose internal Domain Model è expose DTOs •Leak Sensi4ve Data •Force your clients to remote-call-in-loop è bulk ops •Reuse same DTO for POST/PUT/GET è separate structures •Large PUT or PATCH, CRUD forever! è task-based endpoints •Command/Query Responsibility SegregaIon. PUT returns data •Religious REST è sub-resources + verbs can enrich semanIcs That is, don't do this
37. A 128 VictorRentea.ro a training by THANK YOU! TRAINING + CONSULTING = VICTORRENTEA.RO