"API First" Development

API First Development at Etsy. Presented on August 15th at Converge.

Transcript

1. “API First” Development Dan Miller Software Engineer, Core Platform Etsy
2. @jazzdan Metadata bit.ly/api-first
3. @jazzdan Metadata Insert picture of notes here
4. @jazzdan The Problem Transition engineers from desktop web to non-desktop.
5. @jazzdan The Problem Transition humans from desktop web to non-desktop.
6. Model Controller API Business Logic A Business Logic B Business Logic A
7. @jazzdan What is Etsy? Etsy is the marketplace we make together
8. @jazzdan What is Etsy? ObjectifyHomeware in Dunedin, New Zealand
9. @jazzdan What is Etsy? AliciaWatkins in United States
10. 2005 2006 2007 2008 2009 2010 2011 2012 2013 GROSS MERCHANDISE SALES $0.17 M $26 M $87.5 M $180.6 M $314.3 M $525.6 M $895.1 M $1.35 B $3.8 M
12. @jazzdan
16. The Story of Etsy’s Architecture
17. 2005 2014 ‘12 ‘11 ‘10 ‘09 ‘08 ‘07 ‘06 ‘13 • LAMP in a Box • Explosive growth • Chiseling away at the monolith
18. • Four Employees • One Web Server* • One Database Server
 * until getting Slashdotted in August, 2005 2005 2014 ‘12 ‘11 ‘10 ‘09 ‘08 ‘07 ‘06 ‘13
19. • Etsy goes down all the time (often for a long time) • Code deployments are incredibly painful • Developer, DBA, Operations silos 2005 2014 ‘12 ‘11 ‘10 ‘09 ‘08 ‘07 ‘06 ‘13
21. • New CTO • New CEO • New VP of Operations 2005 2014 ‘12 ‘11 ‘10 ‘09 ‘08 ‘07 ‘06 ‘13
22. • Metrics Driven Development • DevOps • Continuous Deployment • Sharded Databases • Tons of Hiring 2005 2014 ‘12 ‘11 ‘10 ‘09 ‘08 ‘07 ‘06 ‘13
23. • Deployed over 30 times per day (Deployinator) • Hundreds of thousands of graphs (StatsD) • 6x as many engineers • $895 million in Gross Merchandise Sales/year 2005 2014 ‘12 ‘11 ‘10 ‘09 ‘08 ‘07 ‘06 ‘13
24. @jazzdan Then mobile happened
27. 2005 2014 ‘12 ‘11 ‘10 ‘09 ‘08 ‘07 ‘06 ‘13
28. 2005 Desktop Web
29. 2011 iOS
30. 2012 Android
31. 2014 Sell on Etsy
33. Technical Problems of 2013
34. Model Controller API Business Logic A Business Logic B Business Logic A
35. @jazzdan REST Not always best
37. @jazzdan A REST API is a Slow API Uniform Interface ! “Individual resources are identified in requests” ! (Fielding 2000, §5.1.5)
38. @jazzdan A REST API is a Slow API 1.GET /listing/12345 2.GET /shop/98765 3.GET /user/cart 4.GET /shop/98765/reviews 5.GET /shop/98765/shipping_profiles 200 ms (best case) 200 ms (best case) 200 ms (best case) 200 ms (best case) 200 ms (best case)
40. @jazzdan A REST API is a Slow API
41. @jazzdan A REST API is a Slow API 1000 ms -900 ms (3G network + client rendering) —————————————————— 100ms
42. @jazzdan A REST API is a Slow API 1.GET /listing/12345 2.GET /shop/98765 3.GET /user/cart 4.GET /shop/98765/reviews 5.GET /shop/98765/shipping_profiles 200 ms (best case) 200 ms (best case) 200 ms (best case) 200 ms (best case) 200 ms (best case)
43. @jazzdan International Performance
46. @jazzdan Data Access Templates Models Controllers Component Database Access
47. Solution
48. @jazzdan Part 1: Human
49. @jazzdan Human Problems • Most Engineers speak PHP and JavaScript • iOS apps are in Objective-C, Android apps are in Java • 1/10 Engineers were native apps engineers
50. @jazzdan Human Problems • Separate teams • iOS team for iPhone, iPad • Android team for Android phone and tablet
51. @jazzdan The Mobile App Training Program
52. @jazzdan Option 1: Learn on your Own • Curated guides and tutorials • Getting started with Objective-C at Etsy • Obj-C/Java style guides • Lunch and Learn videos • App architecture documentation
53. @jazzdan Option 2: The Classroom
54. @jazzdan Option 2: The Classroom • Week 1: Prerequisites • Week 2: Class • Week 3: Post-Class Hacking
55. @jazzdan End Goal • Everyone will be a mobile engineer • All product teams can develop their products on all platforms: Desktop, iOS, Android • Infrastructure teams can triage issues on all platforms
56. @jazzdan Part 2: Technological
57. Model Controller API Business Logic A Business Logic B Business Logic A
58. Model API v3 Business Logic
59. @jazzdan Data Access Templates Models Controllers Component Database Access
60. @jazzdan API First Architecture Models API v3 Bespoke Endpoint Edge No Data Access Data Access
61. @jazzdan Data Flowing In Models API v3 Bespoke Endpoint Edge No Data Access Data Access
62. @jazzdan Data Flowing Out Models API v3 Bespoke Endpoint Edge No Data Access Data Access
63. @jazzdan API First Architecture Models API v3 Bespoke Endpoint Edge No Data Access Data Access
64. The Edge
66. @jazzdan Move data closer to the user faster Origin <!DOCTYPE html> ! <html xmlns="http://www.w3.org/1999/xhtml" ! xml:lang="en" ! lang="en" ! xmlns:og="http://ogp.me/ns#" xmlns:fb="https:// www.facebook.com/2008/fbml" ! data-user-id="15515418" > ! <head> ! <title>Etsy :: Your place to buy and sell all things handmade</title>
67. @jazzdan Move data closer to the user faster Origin Edge
68. @jazzdan <!DOCTYPE html> ! <html xmlns="http://www.w3.org/1999/xhtml" ! xml:lang="en" ! lang="en" ! xmlns:og="http://ogp.me/ns#" xmlns:fb="https:// www.facebook.com/2008/fbml" ! data-user-id="15515418" > ! <head> ! <title>Etsy :: Your place to buy and sell all things handmade</title> Move data closer to the user faster Origin Edge {! "listing": {! "listing_id": 150002520,! "state": "active",! "user_id": 5029420,! "category_id": 69150467,! "title": "ETSY Buttons Variety Pack (Set of 6) copy 3",! ...! } 70% reduction in payload size to Edge
69. @jazzdan Data Access Templates Models Controllers Component Database Access
71. API v3
72. @jazzdan API First Architecture Models API v3 Bespoke Endpoint Edge No Data Access Data Access
73. @jazzdan Listing and Shop Relationship Shop Listing 1 Listing 2 Listing 3
74. @jazzdan v2 Row Oriented aka REST 12345 54321 active 5 123478 Pug Hat 20.00 Cool Pug Hat USD shop_id currency_code price description title category_id user_id state listing_id { “listing_id”: 12345, “state”: “active”, “user_id”: 123478, “title: “Pug Hat”, “description”: “Cool Pug Hat” “price”: “20.00”, “currency_code”: “USD”, “shop_id”: 54321 }
75. @jazzdan v3 Resource Oriented { listing_id: 12345, shop: { name: “Awesome shop”, url: “et.sy/abcdf” }, price: “20.00”, currency: “USD”, currency_symbol: “$”, … }
76. @jazzdan Cacheable Today Tomorrow API v2 Models GET /listings/12345?include=foo&bar API v3 Models GET /public/listings/12345
77. Bespoke Endpoints
79. @jazzdan A REST API is a Slow API 1000 ms -900 ms (network + client processing) —————————————————— 100ms
80. @jazzdan A REST API is a Slow API 1000 ms -900 ms (network + client processing) —————————————————— 100ms = Bespoke
81. @jazzdan API First Architecture Models API v3 Bespoke Endpoint Edge No Data Access Data Access
82. @jazzdan Bespoke Endpoints etsy.com/listing/124740565 Bespoke Recent Shop Overview Related Seller Listing
83. @jazzdan Bespoke Endpoints etsy.com/listing/124740565 Bespoke Recent Shop Overview Related Seller Listing listing->id listing->id listing->id listing->user_id listing->shop_id listing->shop_id
84. @jazzdan Bespoke Endpoints etsy.com/listing/124740565 Bespoke Recent Shop Overview Related Seller Listing listing->id listing->id
85. @jazzdan Bespoke Endpoints etsy.com/listing/124740565 Bespoke Recent Shop Overview Related Seller Listing listing->id listing->id
86. @jazzdan Overview Related Listing Bespoke Endpoints etsy.com/listing/124740565 Bespoke Recent Shop Seller listing->shop_id listing->shop_id
87. @jazzdan API First Architecture Models API v3 Bespoke Endpoint Edge No Data Access Data Access
88. @jazzdan Data Flowing In Models API v3 Bespoke Endpoint Edge No Data Access Data Access
89. @jazzdan Data Flowing Out Models API v3 Bespoke Endpoint Edge No Data Access Data Access
90. Wrap Up
91. @jazzdan Cultural Benefits • Fewer Silos • Easier team collaboration • Career Growth
92. @jazzdan Technical Benefits • Same API endpoints power web and apps • Enables multi-colo • More easily understood codebase • Enable infrastructure experimentation
93. @jazzdan Many, Many Thanks!