Upgrade to Pro — share decks privately, control downloads, hide ads and more …

"API First" Development

"API First" Development

API First Development at Etsy. Presented on August 15th at Converge.

3b544695362c1fd66dae6a2c46f6ff91?s=128

Dan Miller

August 15, 2014
Tweet

Transcript

  1. “API First” Development Dan Miller Software Engineer, Core Platform Etsy

  2. @jazzdan Metadata bit.ly/api-first

  3. @jazzdan Metadata Insert picture of notes here

  4. @jazzdan The Problem Transition engineers from desktop web to non-desktop.

  5. @jazzdan The Problem Transition humans from desktop web to non-desktop.

  6. Model Controller API Business Logic A Business Logic B Business

    Logic A
  7. @jazzdan What is Etsy? Etsy is the marketplace we make

    together
  8. @jazzdan What is Etsy? ObjectifyHomeware in Dunedin, New Zealand

  9. @jazzdan What is Etsy? AliciaWatkins in United States

  10. 2005 2006 2007 2008 2009 2010 2011 2012 2013 GROSS

    MERCHANDISE SALES $0.17 M $26 M $87.5 M $180.6 M $314.3 M $525.6 M $895.1 M $1.35 B $3.8 M
  11. None
  12. @jazzdan

  13. None
  14. None
  15. None
  16. The Story of Etsy’s Architecture

  17. 2005 2014 ‘12 ‘11 ‘10 ‘09 ‘08 ‘07 ‘06 ‘13

    • LAMP in a Box • Explosive growth • Chiseling away at the monolith
  18. • Four Employees • One Web Server* • One Database

    Server
 * until getting Slashdotted in August, 2005 2005 2014 ‘12 ‘11 ‘10 ‘09 ‘08 ‘07 ‘06 ‘13
  19. • Etsy goes down all the time (often for a

    long time) • Code deployments are incredibly painful • Developer, DBA, Operations silos 2005 2014 ‘12 ‘11 ‘10 ‘09 ‘08 ‘07 ‘06 ‘13
  20. None
  21. • New CTO • New CEO • New VP of

    Operations 2005 2014 ‘12 ‘11 ‘10 ‘09 ‘08 ‘07 ‘06 ‘13
  22. • Metrics Driven Development • DevOps • Continuous Deployment •

    Sharded Databases • Tons of Hiring 2005 2014 ‘12 ‘11 ‘10 ‘09 ‘08 ‘07 ‘06 ‘13
  23. • Deployed over 30 times per day (Deployinator) • Hundreds

    of thousands of graphs (StatsD) • 6x as many engineers • $895 million in Gross Merchandise Sales/year 2005 2014 ‘12 ‘11 ‘10 ‘09 ‘08 ‘07 ‘06 ‘13
  24. @jazzdan Then mobile happened

  25. None
  26. None
  27. 2005 2014 ‘12 ‘11 ‘10 ‘09 ‘08 ‘07 ‘06 ‘13

  28. 2005 Desktop Web

  29. 2011 iOS

  30. 2012 Android

  31. 2014 Sell on Etsy

  32. None
  33. Technical Problems of 2013

  34. Model Controller API Business Logic A Business Logic B Business

    Logic A
  35. @jazzdan REST Not always best

  36. None
  37. @jazzdan A REST API is a Slow API Uniform Interface

    ! “Individual resources are identified in requests” ! (Fielding 2000, §5.1.5)
  38. @jazzdan A REST API is a Slow API 1.GET /listing/12345

    2.GET /shop/98765 3.GET /user/cart 4.GET /shop/98765/reviews 5.GET /shop/98765/shipping_profiles 200 ms (best case) 200 ms (best case) 200 ms (best case) 200 ms (best case) 200 ms (best case)
  39. None
  40. @jazzdan A REST API is a Slow API

  41. @jazzdan A REST API is a Slow API 1000 ms

    -900 ms (3G network + client rendering) —————————————————— 100ms
  42. @jazzdan A REST API is a Slow API 1.GET /listing/12345

    2.GET /shop/98765 3.GET /user/cart 4.GET /shop/98765/reviews 5.GET /shop/98765/shipping_profiles 200 ms (best case) 200 ms (best case) 200 ms (best case) 200 ms (best case) 200 ms (best case)
  43. @jazzdan International Performance

  44. None
  45. None
  46. @jazzdan Data Access Templates Models Controllers Component Database Access

  47. Solution

  48. @jazzdan Part 1: Human

  49. @jazzdan Human Problems • Most Engineers speak PHP and JavaScript

    • iOS apps are in Objective-C, Android apps are in Java • 1/10 Engineers were native apps engineers
  50. @jazzdan Human Problems • Separate teams • iOS team for

    iPhone, iPad • Android team for Android phone and tablet
  51. @jazzdan The Mobile App Training Program

  52. @jazzdan Option 1: Learn on your Own • Curated guides

    and tutorials • Getting started with Objective-C at Etsy • Obj-C/Java style guides • Lunch and Learn videos • App architecture documentation
  53. @jazzdan Option 2: The Classroom

  54. @jazzdan Option 2: The Classroom • Week 1: Prerequisites •

    Week 2: Class • Week 3: Post-Class Hacking
  55. @jazzdan End Goal • Everyone will be a mobile engineer

    • All product teams can develop their products on all platforms: Desktop, iOS, Android • Infrastructure teams can triage issues on all platforms
  56. @jazzdan Part 2: Technological

  57. Model Controller API Business Logic A Business Logic B Business

    Logic A
  58. Model API v3 Business Logic

  59. @jazzdan Data Access Templates Models Controllers Component Database Access

  60. @jazzdan API First Architecture Models API v3 Bespoke Endpoint Edge

    No Data Access Data Access
  61. @jazzdan Data Flowing In Models API v3 Bespoke Endpoint Edge

    No Data Access Data Access
  62. @jazzdan Data Flowing Out Models API v3 Bespoke Endpoint Edge

    No Data Access Data Access
  63. @jazzdan API First Architecture Models API v3 Bespoke Endpoint Edge

    No Data Access Data Access
  64. The Edge

  65. None
  66. @jazzdan Move data closer to the user faster Origin <!DOCTYPE

    html> ! <html xmlns="http://www.w3.org/1999/xhtml" ! xml:lang="en" ! lang="en" ! xmlns:og="http://ogp.me/ns#" xmlns:fb="https:// www.facebook.com/2008/fbml" ! data-user-id="15515418" > ! <head> ! <title>Etsy :: Your place to buy and sell all things handmade</title>
  67. @jazzdan Move data closer to the user faster Origin Edge

  68. @jazzdan <!DOCTYPE html> ! <html xmlns="http://www.w3.org/1999/xhtml" ! xml:lang="en" ! lang="en"

    ! xmlns:og="http://ogp.me/ns#" xmlns:fb="https:// www.facebook.com/2008/fbml" ! data-user-id="15515418" > ! <head> ! <title>Etsy :: Your place to buy and sell all things handmade</title> Move data closer to the user faster Origin Edge {! "listing": {! "listing_id": 150002520,! "state": "active",! "user_id": 5029420,! "category_id": 69150467,! "title": "ETSY Buttons Variety Pack (Set of 6) copy 3",! ...! } 70% reduction in payload size to Edge
  69. @jazzdan Data Access Templates Models Controllers Component Database Access

  70. None
  71. API v3

  72. @jazzdan API First Architecture Models API v3 Bespoke Endpoint Edge

    No Data Access Data Access
  73. @jazzdan Listing and Shop Relationship Shop Listing 1 Listing 2

    Listing 3
  74. @jazzdan v2 Row Oriented aka REST 12345 54321 active 5

    123478 Pug Hat 20.00 Cool Pug Hat USD shop_id currency_code price description title category_id user_id state listing_id { “listing_id”: 12345, “state”: “active”, “user_id”: 123478, “title: “Pug Hat”, “description”: “Cool Pug Hat” “price”: “20.00”, “currency_code”: “USD”, “shop_id”: 54321 }
  75. @jazzdan v3 Resource Oriented { listing_id: 12345, shop: { name:

    “Awesome shop”, url: “et.sy/abcdf” }, price: “20.00”, currency: “USD”, currency_symbol: “$”, … }
  76. @jazzdan Cacheable Today Tomorrow API v2 Models GET /listings/12345?include=foo&bar API

    v3 Models GET /public/listings/12345
  77. Bespoke Endpoints

  78. None
  79. @jazzdan A REST API is a Slow API 1000 ms

    -900 ms (network + client processing) —————————————————— 100ms
  80. @jazzdan A REST API is a Slow API 1000 ms

    -900 ms (network + client processing) —————————————————— 100ms = Bespoke
  81. @jazzdan API First Architecture Models API v3 Bespoke Endpoint Edge

    No Data Access Data Access
  82. @jazzdan Bespoke Endpoints etsy.com/listing/124740565 Bespoke Recent Shop Overview Related Seller

    Listing
  83. @jazzdan Bespoke Endpoints etsy.com/listing/124740565 Bespoke Recent Shop Overview Related Seller

    Listing listing->id listing->id listing->id listing->user_id listing->shop_id listing->shop_id
  84. @jazzdan Bespoke Endpoints etsy.com/listing/124740565 Bespoke Recent Shop Overview Related Seller

    Listing listing->id listing->id
  85. @jazzdan Bespoke Endpoints etsy.com/listing/124740565 Bespoke Recent Shop Overview Related Seller

    Listing listing->id listing->id
  86. @jazzdan Overview Related Listing Bespoke Endpoints etsy.com/listing/124740565 Bespoke Recent Shop

    Seller listing->shop_id listing->shop_id
  87. @jazzdan API First Architecture Models API v3 Bespoke Endpoint Edge

    No Data Access Data Access
  88. @jazzdan Data Flowing In Models API v3 Bespoke Endpoint Edge

    No Data Access Data Access
  89. @jazzdan Data Flowing Out Models API v3 Bespoke Endpoint Edge

    No Data Access Data Access
  90. Wrap Up

  91. @jazzdan Cultural Benefits • Fewer Silos • Easier team collaboration

    • Career Growth
  92. @jazzdan Technical Benefits • Same API endpoints power web and

    apps • Enables multi-colo • More easily understood codebase • Enable infrastructure experimentation
  93. @jazzdan Many, Many Thanks!