API First Architecture Transformation at Etsy

At Etsy we have been doing some pioneering work with our web APIs. We switched to API-first design, have experimented with concurrency handling in our composition layer, introduced strong typing into our API design, experimented with code generation, and built distributed tracing tools for the API as part of this project. We faced a common challenge: much of our logic was implemented twice. All of the code that was built for the website then had to be rebuilt in our API to be used by our iOS and Android apps. We wanted an approach where we built everything on reusable API components that could be shared between the web and API. Unfortunately, our existing API framework couldn’t support this shared approach. The solution we settled on was to abandon the existing framework and rebuild it from scratch. Follow along this case study of building an API First architecture. Hear what problems prompted this drastic change. Learn which new tools we had to build to be able to work with the new system and what mistakes we made along the way. Finally, how did it end? How did the team adopt the new system and have we succeeded in our goals of API First? From our learnings, can we make the adoption of new systems and ideas easier for everyone?

Transcript

1. API-First Architecture Transformation at Etsy Stefanie Schirmer @linse
2. API-First Architecture Transformation at Etsy Stefanie Schirmer @linse
4. 1: Cross-platform consistency?
5. DB model controller Logic A controller Logic B view view Logic A Logic B
6. DB model controller Logic A controller Logic B view view Logic A Logic B API v2 Logic A Logic B iOS App Logic A Logic B Android App Logic A Logic B
7. 2: Performance?
8. time to glass non-concurrent PHP .. .. .. .
9. “We desperately need to figure out a scheme for allowing concurrency or else we're going to have performance problems forever.” - Andrew Morrison (Etsy)
10. 3: Who has control?
11. tied to DB schema user controls request API
12. API platform NEW! v2 v3
13. keep it REST (as in v2) declaration of endpoints in code input validation Ideas we kept v3!
14. decouple endpoints from framework auto-generate routes, clients: PHP, JS meta-endpoints that consume our API New ideas v3!
15. decouple endpoints from framework auto-generate routes, clients: PHP, JS meta-endpoints that consume our API New ideas v3!
16. cURL
18. homepage favorites favorite shops friends activity … cURL multi cURL multi …
20. How we built it
21. Etsy API Perspectives
22. https://www.etsy.com/api/v3/ajax/public/teams /member/ /shop/
23. Etsy API Services
24. Ajax Atlas Internal EtsyApps OpenAPI …
27. shop updates favorites picks for you favorite shops friends activity shop explore from blog
28. shop updates favorites picks for you favorite shops friends activity shop explore from blog favorites favorite shops friends activity from blog featured shop story card story card listing card listing card shop card shop card listing card …
29. bespoke homepage favorites favorite shops friends activity story card story card listing card listing card shop card shop card listing card … cURL multi cURL multi cURL multi cURL multi …
30. bespoke homepage favorites favorite shops friends activity story card story card listing card listing card shop card shop card listing card … cURL multi cURL multi cURL multi cURL multi … bespoke
31. bespoke homepage favorites favorite shops friends activity story card story card listing card listing card shop card shop card listing card … cURL multi cURL multi cURL multi cURL multi … bespoke concurrent
32. bespoke homepage favorites favorite shops friends activity story card story card listing card listing card shop card shop card listing card … cURL multi cURL multi cURL multi cURL multi … bespoke concurrent component
33. Język dziedzinowy DSL of Etsy endpoints domänenspezifische Sprache von Endpunkten langage dédié هنماد هب صتخم نابز ϮαЀࢴํ᥺承
34. route() desc() resultType() handle($client, $input, $response) required
35. declareInput($input) includedServices() cacheTtlSeconds() optional
36. Tooling: Route compiler & Auto-client generation https://www.etsy.com/listing/192043815/1880s-vintage-toolbox-full-of-tools
37. Architecture implications
38. pain point of shared nothing: bootstrap
39. Problem: PHP time to request
40. Skip optional work Optimize bootstrap Caching HHVM
41. HHVM Average Response Time 12 Hour PHP 5 / Zend Average Response Time 12 Hour
43. Where wouldn’t this work?
44. Operations: Tooling
45. Tooling: CrossStitch https://www.etsy.com/listing/192043815/1880s-vintage-toolbox-full-of-tools
47. <someserver>
48. Tooling: Fanout-Manager https://www.etsy.com/listing/192043815/1880s-vintage-toolbox-full-of-tools
49. Tooling: datafindr https://www.etsy.com/listing/192043815/1880s-vintage-toolbox-full-of-tools
51. Developer adoption
52. new platform (& retire) scaling/ maintenance incremental improvements Phases of adoption
53. How did we “sell” it?
54. ..too fast to keep up!
55. White space gets filled with code
56. Better documentation!! https://www.etsy.com/listing/275337876/eclectic-elements-documentation-half
57. Pockets of patterns
58. iOS team: contract between set of items an endpoint will return, set of items that the app knows how to format. code organization: compiler in same space as endpoints ಠ_ಠ inventory team: we can compose endpoints, but for composable views we still have to repeat things. Can we lift composability into the view? android team: can we mock a typed resource? Can we automatically traverse it? homepage team: make API type errors more prominent to enforce guarantees.
59. Trust + Guidance
60. Conclusion
61. What did we learn?
62. What did we learn? Did we succeed?
63. “We desperately need to figure out a scheme for allowing concurrency or else we're going to have performance problems forever.” - Andrew Morrison (Etsy) ✓
64. Where we are now: Responsibility division between device and server
65. Thank you <3 I’m Stefanie Schirmer @linse https://codeascraft.com -> post series on the API