API Design: It's Not Rocket Surgery (ODI)

An updated version of the API talk, given at the Open Data Institute on Thursday 6th December 2012.

1. API design It’s Not Rocket Surgery Dave Ingram @dmi December 6, 2012
2. Who am I? • Was developer/RM at GroupSpaces for 41/2 years • Now DevOps freelancer • Twitter: @dmi • Github: dingram • Way too many projects • Involved in open source, including Phabricator • Occasionally found at London Hackspace
3. Why am I giving this talk? • I’ve built a number of APIs • Both for GroupSpaces and my own projects • Sadly most are not public (yet!) • I’m also a consumer of many other APIs • Twitter • Foursquare • Tumblr • TfL • Spotify • . . . blah blah blah. . .
4. Feedback/thoughts via Twitter: #apirocketsurgery (or @dmi)
5. What this isn’t
6. T T REST REST REST REST REST REST REST EST EST ST ST ST T T REST REST REST REST REST REST REST REST REST REST REST REST REST REST R REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST R REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST R REST REST REST REST REST REST REST REST REST REST REST REST REST REST RES RES REST REST REST REST REST REST REST REST REST REST REST REST REST R R R RE RE RE RES RES REST R
7. T T REST REST REST REST REST REST REST EST EST ST ST ST T T REST REST REST REST REST REST REST REST REST REST REST REST REST REST R REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST R REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST R REST REST REST REST REST REST REST REST REST REST REST REST REST REST RES RES REST REST REST REST REST REST REST REST REST REST REST REST REST R R R RE RE RE RES RES REST R
8. T T REST REST REST REST REST REST REST EST EST ST ST ST T T REST REST REST REST REST REST REST REST REST REST REST REST REST REST R REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST R REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST REST R REST REST REST REST REST REST REST REST REST REST REST REST REST REST RES RES REST REST REST REST REST REST REST REST REST REST REST REST REST R R R RE RE RE RES RES REST R
9. There’s so much more!
10. URLs
11. URLs verbs
12. URLs verbs headers
13. URLs verbs headers authentication
14. URLs verbs headers authentication formats
15. URLs verbs headers authentication formats integrity
16. URLs verbs headers authentication formats integrity documentation
17. DOCUMENTATION
18. DOCUMENTATION (later)
19. Part 1: URLs
20. Make them versioned
21. Make them versioned, hackable
22. Make them versioned, hackable & meaningful
23. http://api.com/v1/
24. http://api.com/v1/users/
25. http://api.com/v1/users/31337
26. http://api.com/v1/users/31337/posts/
27. Look at URLs from a consumer perspective
28. They don’t care about internal implementation details
29. Use common sense
30. Don’t be afraid of the query string
31. Use query arguments to filter output or for (some) alternate representations
32. Good: searching & verbosity Bad: output format control (XML vs JSON)
33. Good: http://api.com/v1/search?q=foobar Bad: http://api.com/v1/search/foobar
34. Good: http://api.com/v1/foo.json Bad: http://api.com/v1/foo?format=json
35. Part 2: Verbs
36. • GET = get() • PUT = setAll() / new Obj($id) • POST = new Obj() / doStuff() / set() • DELETE = delete() • HEAD ≈ getMetadata() • OPTIONS ≈ reflection/permissions
37. Must at least handle GET/POST
38. Can emulate PUT/DELETE
39. For example: POST http://api.com/v1/foo/42!PUT POST http://api.com/v1/foo/11!DELETE
40. Remember that POST can have side-effects and is never cached
41. PUT and DELETE are idempotent, which means they have the same effect even if they’re repeated
42. HEAD is rare and can probably be ignored
43. OPTIONS is used by CORS, but otherwise rare
44. CORS?
45. Cross-Origin Resource Sharing
46. A way to allow in-browser cross-origin XMLHTTPRequests Support: FF3.5+, Chrome 4+, Safari 4+, Opera 12+, IE8+ (partial), IE10+ (full), iOS 3.2+, Android 2.1+ http://www.w3.org/TR/cors/ http://caniuse.com/cors
47. Origin & Allow-Origin A way to allow in-browser cross-origin XMLHTTPRequests Support: FF3.5+, Chrome 4+, Safari 4+, Opera 12+, IE8+ (partial), IE10+ (full), iOS 3.2+, Android 2.1+ http://www.w3.org/TR/cors/ http://caniuse.com/cors
48. Part 3: Headers
49. Headers are important too! (and you can be clever)
50. Accept The MIME types the client will accept. No need to use file extensions to decide what content type to serve! Accept-Language The languages the client will accept. No need to ask clients or (worse) just assume English responses.
51. Some headers reduce traffic (important for mobile)
52. • ETag – A unique tag for the content • If-(None-)Match – Check ETag • If-(Un)Modified-Since – Is it newer? • Cache-Control – Can it be cached? • Expires – How long is it valid? • Vary – Additional caching rules
53. Beware: some proxies may not pass custom headers
54. Beware: even some standard headers are not safe
55. Prefer headers, but accept other methods
56. Part 4: Authentication & Authorization
57. OAuth2 over HTTPS
58. Much simpler than OAuth1 (similar to session cookies)
59. Many libraries for OAuth2 for many platforms as it’s a popular standard
60. For extra security use OAuth2-MAC
61. OAuth2-MAC uses signatures instead of bearer tokens, so secrets stay secret
62. Then again, the author of OAuth2 has now advised not upgrading from OAuth 1.0a and either using OAuth 1.0a for new sites or staying close to a large provider’s implementation
63. No need to worry about rate-limiting (to start with*) 3scale, Apiaxle, Mashery can help
64. Part 5: Formats
65. Sane default: JSON, plus envelope with metadata
66. { "meta": { "code": 2 , "dev_notes ": [ "This endpoint is deprecated" ] }, "response ": { ... } }
67. Metadata envelope helps with JSONP and limited clients that discard errors
68. Having metadata allows out-of-band information, like deprecation warnings
69. What about XML?
70. *Hypertext as the Engine of Application State HATEOAS* is a nice idea, although it’s very verbose (but useful for building an API explorer)
71. HATEOAS adds links to related endpoints in the response
72. At the very most, HATEOAS should be opt-in for clients that will use it
73. Don’t forget: mobile bandwidth is still very limited
74. Timestamps • ISO-8601 2 12- 5- 3T19: : Z Human-readable, but needs parsing • UTC seconds since epoch: 1336 716 Easily machine-usable
75. Should your API really be human-readable? It’s better to help your consumers.
76. Part 6: Data integrity
77. Allow your consumers to cache data whenever possible
78. Encourage use of request headers: • GET: • If-Modified-Since • If-None-Match • POST/PUT/DELETE: • If-Unmodified-Since • If-Match
79. Return useful response headers: • Last-Modified • Expires • ETag • Cache-Control
80. Deal with preconditions and give correct response codes
81. Useful status codes: 4 5 Method Not Allowed 4 6 Not Acceptable 412 Precondition Failed 428 Precondition Required* 429 Too Many Requests* *New in RFC6585
82. For example: ETag and Last-Modified can help prevent race conditions
83. PUT /v1/wiki/dealing -with -conflicts HTTP /1.1 Host: api.com Content -Type: text/html ... 428 Precondition Required ETag: "x-rev -11294" Last -Modified: Sat , 18 Feb 2 12 11: 9:21 GMT ...
84. PUT /v1/wiki/dealing -with -conflicts HTTP /1.1 Host: api.com Content -Type: text/html ... 428 Precondition Required ETag: "x-rev -11294" Last -Modified: Sat , 18 Feb 2 12 11: 9:21 GMT ...
85. PUT /v1/wiki/dealing -with -conflicts HTTP /1.1 Host: api.com Content -Type: text/html ... 428 Precondition Required ETag: "x-rev -11294" Last -Modified: Sat , 18 Feb 2 12 11: 9:21 GMT ...
86. PUT /v1/wiki/dealing -with -conflicts HTTP /1.1 Host: api.com Content -Type: text/html ... 428 Precondition Required ETag: "x-rev -11294" Last -Modified: Sat , 18 Feb 2 12 11: 9:21 GMT ...
87. PUT /v1/wiki/dealing -with -conflicts HTTP /1.1 Host: api.com If -Unmodified -Since: Sat , 18 Feb 2 12 11: 9:21 GMT If -Match: "x-rev -11294" Content -Type: text/html ... 412 Precondition Failed ETag: "x-rev -11467" Last -Modified: Sat , 25 Feb 2 12 14:42:53 GMT ...
88. PUT /v1/wiki/dealing -with -conflicts HTTP /1.1 Host: api.com If -Unmodified -Since: Sat , 18 Feb 2 12 11: 9:21 GMT If -Match: "x-rev -11294" Content -Type: text/html ... 412 Precondition Failed ETag: "x-rev -11467" Last -Modified: Sat , 25 Feb 2 12 14:42:53 GMT ...
89. PUT /v1/wiki/dealing -with -conflicts HTTP /1.1 Host: api.com If -Unmodified -Since: Sat , 18 Feb 2 12 11: 9:21 GMT If -Match: "x-rev -11294" Content -Type: text/html ... 412 Precondition Failed ETag: "x-rev -11467" Last -Modified: Sat , 25 Feb 2 12 14:42:53 GMT ...
90. PUT /v1/wiki/dealing -with -conflicts HTTP /1.1 Host: api.com If -Unmodified -Since: Sat , 18 Feb 2 12 11: 9:21 GMT If -Match: "x-rev -11294" Content -Type: text/html ... 412 Precondition Failed ETag: "x-rev -11467" Last -Modified: Sat , 25 Feb 2 12 14:42:53 GMT ...
91. Part 7: Documentation
92. Documentation is important
93. If people can’t understand your API, they won’t use it
94. Use examples liberally. . . and make sure they’re both up-to-date and correct!
95. Assume nothing; explain everything
96. Try getting people you know to build something using only your own API docs
97. Be helpful: provide an interactive console e.g. Apiary, Apigee console, Mashery’s iodocs
98. And finally: Statistics
99. Record numbers for anything and everything you can imagine
100. Everybody loves graphs (and they’re important for understanding API usage and performance)
101. Things to measure Counters: HTTP status (200, 400, 500, . . . ) Tokens issued Token refreshes Clients created Client usage Auth successes Auth failures Auth grants Auth rejections DB queries API version/release Client type . . . Avg response time: Auth requests General requests Specific subsystems . . . Other Memory usage Client location . . .
102. Thanks! Any questions? Feedback via Twitter: #apirocketsurgery or @dmi Slides: http://www.dmi.me.uk/talks/ Built in L ATEX Inspired by: http://goo.gl/ mT55