Making Maps: The Role of Frontend Infrastructure at Etsy

Making Maps: The Role of Frontend Infrastructure at Etsy

Video: http://vimeo.com/109912254
Given October 9th at Fronteers 2014 in Amsterdam.

The scope of Etsy's frontend is massive - our JavaScript codebase has grown by 50% in the last year to more than 3,000 files totaling almost 800,000 lines. Because of Etsy's dedication to continuous deployment, the code running in production changes 25 to 50 times every day. Because of our experimentation-driven development cycle, there may be multiple production versions of our features at any given time. These factors can lead to uncertainty and fear with rolling out upgrades, deleting old code, or confidently making changes. I share how the Frontend Infrastructure team works within this ecosystem to mitigate those risks, manage the asset build pipeline, build tools to understand our frontend, automate migrations & delete as much code as possible.

757c233176d3ad14136df4a547ac4234?s=128

Daniel Espeset

October 09, 2014
Tweet

Transcript

  1. Daniel Espeset @danielespeset Fronteers Amsterdam, October 2014 The Role of

    Frontend Infrastructure at Etsy Making Maps
  2. talks.desp.in/fronteers2014 Resources for this talk

  3. Daniel Espeset @danielespeset Frontend Infrastructure 3 Daniel Espeset Daniel Na

    Takashi Mizohata Etsy Seth Walker Sufian Rhazi
  4. 4

  5. Daniel Espeset @danielespeset 1 Million Active Sellers 26 Million Active

    Listings 5
  6. 6 Etsy Engineering

  7. Daniel Espeset @danielespeset 7 Anybody can touch anything

  8. Daniel Espeset @danielespeset 8 Blameless postmortems

  9. Daniel Espeset @danielespeset 9 Everyone deploys

  10. 10 Continuous Deployment

  11. Daniel Espeset @danielespeset "The Button"

  12. pages/search_results.php if (Feature::isEnabled('my_new_feature')) { // TODO build this part }

    else { // Existing code } 20 21 22 .. 100 12 config/production.php // Owner: despeset, frontend infrastructure server_config['my_new_feature'] = ( 'enabled' => 0 ); 890 891 892 893
  13. 13 desp .join pushbot has changed topic to: "<prod> seth

    dna* gio* | desp" pushbot desp jklein : You're up desp .in pushbot has changed topic to: "desp jklein" pushbot has changed topic to: "desp* jklein" jklein .in pushbot has changed topic to: "desp* jklein*" pushbot desp jklein : everyone is ready
  14. None
  15. None
  16. None
  17. None
  18. None
  19. 19 25-50Pushes a day

  20. 20 config/production.php // Owner: despeset, frontend infrastructure server_config['my_new_feature'] = (

    'enabled' => 0, 'group' => 9903 ); 890 891 892 893 894
  21. 21 config/production.php // Owner: despeset, frontend infrastructure server_config['my_new_feature'] = (

    'enabled' => 0, 'group' => 9903, 'admin' => 'on' ); 890 891 892 893 894 895
  22. 22 config/production.php // Owner: despeset, frontend infrastructure server_config['my_new_feature'] = (

    'enabled' => 50, 'group' => 9903, 'admin' => 'on' ); 890 891 892 893 894 895
  23. 23 Continuous Experimentation

  24. Daniel Espeset @danielespeset 24 idea code release idea code release?

    A/B test idea prototype release A/B test validate A/B test refine 2005 2010 2012 Shamelessly stolen from Dan McKinley mcfunley.com/data-driven-products-now Evolution of Continuous Experimentation
  25. Daniel Espeset @danielespeset 25 • Engineers are trusted and have

    massive access • The production codebase is constantly changing • That codebase is filled with experiments TL;DR
  26. Daniel Espeset @danielespeset 26

  27. Daniel Espeset @danielespeset 27 Our frontend is made up of

    300  different “pages”
  28. Daniel Espeset @danielespeset 28 At any given time we may

    be running dozens of experiments per page
  29. Daniel Espeset @danielespeset 29 4,096 212 =

  30. Daniel Espeset @danielespeset 30 212 300 x = 1.2 M

  31. Daniel Espeset @danielespeset 31 Reasoning about our frontend can be

    hard
  32. Daniel Espeset @danielespeset 32 Deleting code is often scary

  33. Daniel Espeset @danielespeset 33 Testing changes is difficult

  34. 34 Frontend Infrastructure

  35. Daniel Espeset @danielespeset 35 source code deploy build system

  36. Daniel Espeset @danielespeset 36 source code production build deploy build

    system
  37. Daniel Espeset @danielespeset 37 Frontend Infrastructure

  38. 38 Builda [Build Assets]

  39. Daniel Espeset @danielespeset • • Wraps RequireJS AMD compiler •

    Parallelizes work across CPU cores • Serves fully compiled assets on development VMS 39 Builda [Build Assets]
  40. Daniel Espeset @danielespeset 40 Techniques for serving built assets in

    dev • Build on the fly in response to HTTP requests for asset files
  41. Daniel Espeset @danielespeset 41 Techniques for serving built assets in

    dev • Build on the fly in response to HTTP requests for asset files • Watch FS for any changes and rebuild everything
  42. Daniel Espeset @danielespeset 42 Techniques for serving built assets in

    dev • Build on the fly in response to HTTP requests for asset files • Watch FS for any changes and rebuild everything • Watch FS for any changes and only rebuild affected targets
  43. Daniel Espeset @danielespeset 43 Techniques for serving built assets in

    dev • Build on the fly in response to HTTP requests for asset files • Watch FS for any changes and rebuild everything • Watch FS for any changes and only rebuild affected targets this is... complicated
  44. Daniel Espeset @danielespeset 44 source code built assets Builda Source

    file is changed
  45. Daniel Espeset @danielespeset 45 source code built assets Builda Builda

    looks up dependencies
  46. Daniel Espeset @danielespeset 46 source code built assets Builda Only

    rebuilds files that are now stale
  47. Daniel Espeset @danielespeset This solution gave us something valuable 47

    A dependency graph
  48. 48 dependency_graph.json "mobile/modules/rwd": { "info": { "version": "20140815195505" }, "children":

    { "common/etsy.namespace": "amd" }, "parents": { "mobile/base-container": "amd" } } 201 202 203 204 205 206 207 208 209 210 211
  49. 49 Ranger

  50. Daniel Espeset @danielespeset 50 + + JS graph render graph

    CSS graph Ranger combines 3 dependency graphs
  51. 51

  52. 52

  53. 53

  54. 54

  55. 55

  56. 56

  57. 57

  58. 58

  59. Daniel Espeset @danielespeset Ranger gave us something valuable 59 The

    set of files that aren't used
  60. 60 Shrinkray

  61. Daniel Espeset @danielespeset • Pick a stylesheet loaded on the

    page 61 Shrinkray - clientside CSS analysis
  62. Daniel Espeset @danielespeset • Pick a stylesheet loaded on the

    page • Randomly pick 50 selectors via CSSOM 62 Shrinkray - clientside CSS analysis
  63. Daniel Espeset @danielespeset • Pick a stylesheet loaded on the

    page • Randomly pick 50 selectors via CSSOM • Search for matches with document.querySelector 63 Shrinkray - clientside CSS analysis
  64. Daniel Espeset @danielespeset • Pick a stylesheet loaded on the

    page • Randomly pick 50 selectors via CSSOM • Search for matches with document.querySelector • Send results to server 64 Shrinkray - clientside CSS analysis
  65. Daniel Espeset @danielespeset • Pick a stylesheet loaded on the

    page • Randomly pick 50 selectors via CSSOM • Search for matches with document.querySelector • Send results to server • Aggregate data with Map Reduce 65 Shrinkray - clientside CSS analysis
  66. 66

  67. 67 Moving to SCSS

  68. Daniel Espeset @danielespeset 68 Existing preprocessor inlines @import, versions images

  69. Daniel Espeset @danielespeset 69 Linted for syntax (sorta)

  70. 70 Every color defined in our CSS

  71. Daniel Espeset @danielespeset • No @extends ever • No @mixins

    (for now) • Nesting level < 4 • Lints to enforce these and strict code-style adherence 71 Restricting SCSS functionality
  72. Daniel Espeset @danielespeset • @import statements needed changed • Converter

    fixes 171,244 scss-lint errors • Had to resolve hundreds of invalid CSS rules & selectors 72 Converter script
  73. Daniel Espeset @danielespeset 73 Will fixing this break something? color:

    rgb(25555, -10, 0);
  74. Daniel Espeset @danielespeset 74 What about this one? color: rgb(0,

    0, 0, 0);
  75. Daniel Espeset @danielespeset 75 CSS Source Legacy CSS Build Legacy

    Builda
  76. Daniel Espeset @danielespeset 76 CSS Source Legacy CSS Build Legacy

    Builda Converter SCSS Builda New CSS Build
  77. Daniel Espeset @danielespeset 77 CSS Source Legacy CSS Build Legacy

    Builda Converter SCSS Builda New CSS Build AST Differ Parse to AST Parse to AST
  78. Daniel Espeset @danielespeset 78 CSS Source Legacy CSS Build Legacy

    Builda Converter SCSS Builda 100% New CSS Build
  79. Daniel Espeset @danielespeset 79 CSS Source Legacy CSS Build Legacy

    Builda Converter SCSS Builda 100% Etsy New CSS Build
  80. Daniel Espeset @danielespeset 80 CSS Source Legacy CSS Build Legacy

    Builda Converter SCSS Builda 50% 50% New CSS Build
  81. Daniel Espeset @danielespeset 81 CSS Source Legacy CSS Build New

    CSS Build Legacy Builda Converter SCSS Builda 100%
  82. Daniel Espeset @danielespeset 82 SCSS Source New CSS Build SCSS

    Builda 100%
  83. What have we learned?

  84. Daniel Espeset @danielespeset 84 Disposability > Modularity

  85. Daniel Espeset @danielespeset 85 The frontend is getting more complex

    everywhere.
  86. Daniel Espeset @danielespeset 86 We need new strategies to manage

    and understand it.
  87. Daniel Espeset @danielespeset 87 The browser is part of your

    distributed system, not just a client you support.
  88. Daniel Espeset @danielespeset 88 The more extensible your tool the

    more useful it will be.