Frontend Infrastructure at Etsy

Delivered at Smashing Conf NYC, June 17th 2015

Transcript

1. Daniel Espeset @danielespeset Smashing Conf NYC, June 2015 at Etsy Frontend Infrastructure
2. talks.desp.in/smashing Resources for this talk
4. Daniel Espeset @danielespeset A front-end operations engineer is not a title you’ve likely come across, but hopefully one that you will [...] an expert at serving and hosting front-end resources. [finding] the best ways to piece together the parts of a Web application, and [being a pro] at versioning, caching and deployment. Alex Sexton @slexaxton "Front-End Ops", Smashing Magazine, July 2013 “ ” 4
5. Daniel Espeset @danielespeset Frontend Infrastructure 5 Daniel Espeset Daniel Na Takashi Mizohata Etsy Seth Walker Sufian Rhazi Jonathan Lai
6. Etsy Engineering
7. Daniel Espeset @danielespeset Anybody can touch (mostly) anything 7
8. Daniel Espeset @danielespeset Blameless postmortems 8
9. Daniel Espeset @danielespeset Everyone deploys 9
10. Continuous Deployment
11. Daniel Espeset @danielespeset Push Button Deploys 11
12. pages/search_results.php if (Feature::isEnabled('my_new_feature')) { // TODO build this part } else { // Existing code } 20 21 22 .. 100 12 config/production.php // Owner: despeset, frontend infrastructure server_config['my_new_feature'] = ( 'enabled' => 0 ); 890 891 892 893 Checking a flag Defining a flag
13. 13 desp .join pushbot has changed topic to: "<prod> seth dna* gio* | desp" pushbot desp jklein : You're up desp .in pushbot has changed topic to: "desp jklein" pushbot has changed topic to: "desp* jklein" jklein .in pushbot has changed topic to: "desp* jklein*" pushbot desp jklein : everyone is ready Coordinating Deploys
19. 19 25-50Pushes a day
20. 20 // Owner: despeset, frontend infrastructure server_config['my_new_feature'] = ( 'enabled' => 0, 'group' => 9903 ); 890 891 892 893 894 config/production.php
21. 21 config/production.php // Owner: despeset, frontend infrastructure server_config['my_new_feature'] = ( 'enabled' => 0, 'group' => 9903, 'admin' => 'on' ); 890 891 892 893 894 895
22. 22 // Owner: despeset, frontend infrastructure server_config['my_new_feature'] = ( 'enabled' => 50, 'group' => 9903, 'admin' => 'on' ); 890 891 892 893 894 895 config/production.php
23. Continuous Experimentation
24. Daniel Espeset @danielespeset idea code release idea code release? A/B test idea prototy release A/B test validate A/B test refine 2005 2010 2012 Shamelessly stolen from Dan McKinley mcfunley.com/data-driven-products-now Evolution of Continuous Experimentation 24
25. Daniel Espeset @danielespeset • Engineers are trusted and given agency • The production codebase is constantly changing • That codebase is filled with feature flags In a nutshell 25
26. Daniel Espeset @danielespeset 26
27. Daniel Espeset @danielespeset 27 Sass conversion
28. Daniel Espeset @danielespeset Our frontend is made up of >300
    different “pages” 28
29. Daniel Espeset @danielespeset At any given time we may be running dozens of tests per page 29
30. Daniel Espeset @danielespeset 4,096 212= 30
31. Daniel Espeset @danielespeset 212 300 x =1.2 M 31
32. Daniel Espeset @danielespeset Reasoning about our frontend can be hard 32
33. Daniel Espeset @danielespeset Deleting code is often scary 33
34. Daniel Espeset @danielespeset Verifying changes is difficult 34
35. Frontend Infrastructure
36. Daniel Espeset @danielespeset 36 Performance Allison McKnight Kristyn Reith Etsy Natalya Hoota Lara Hogan
37. Daniel Espeset @danielespeset source code deploy build system 37
38. Daniel Espeset @danielespeset source code production build 38 deploy build system
39. Daniel Espeset @danielespeset Frontend Infrastructure 39
40. Daniel Espeset @danielespeset 40 ...the bridge between an application's intent and an application's reality. Alex Sexton (JavaScript Laureate of Texas) "Front-End Ops", Smashing Magazine, July 2013 “ ”
41. {% css files="base" %}{% /css %} Template Code <link href="css/base.20150607182402.css" ... /> Rendered HTML most recent last modified date of file and all dependencies.
42. {% css files="base" %}{% /css %} Template Code <link href="css/base.20150607182402.css" ... /> Rendered HTML adds version number, always happens.
43. {% css files="base" %}{% /css %} Template Code <link href="css/base.20150607182402.css" ... /> Rendered HTML if (rand() > 0.01) { log(template, asset); }
44. Where are my things? (Ranger)
45. Daniel Espeset @danielespeset + + JS graph Asset logs CSS graph Ranger combines 3 dependency graphs 45
54. Daniel Espeset @danielespeset Ranger gave us something valuable The set of files that aren't used 54
55. Profiling CSS (Shrinkray)
56. Daniel Espeset @danielespeset 56 • Pick a stylesheet loaded on the page Shrinkray - CSS analysis
57. Daniel Espeset @danielespeset 57 • Pick a stylesheet loaded on the page • Randomly pick 50 selectors via CSSOM Shrinkray - CSS analysis
58. Daniel Espeset @danielespeset 58 • Pick a stylesheet loaded on the page • Randomly pick 50 selectors via CSSOM • Search for matches with document.querySelector Shrinkray - CSS analysis
59. Daniel Espeset @danielespeset 59 • Pick a stylesheet loaded on the page • Randomly pick 50 selectors via CSSOM • Search for matches with document.querySelector • Send results to server Shrinkray - CSS analysis
60. Daniel Espeset @danielespeset • Pick a stylesheet loaded on the page • Randomly pick 50 selectors via CSSOM • Search for matches with document.querySelector • Send results to server • Aggregate data with Map Reduce Shrinkray - CSS analysis 60
62. Daniel Espeset @danielespeset • Low confidence on long-tail pages • Doesn't capture CSS appearing by user action Why doesn't it work yet? 62
63. Daniel Espeset @danielespeset • Increase measurement volume if perf allows • Add time skew to measurements • Intelligently target selectors based on the data • Add a second pass using an more invasive but thorough technique Possible solutions 63
64. 64 #container .button-cool ul { color: pink; float: left; } #container .button-cool ul:before { content: ''; background-image: url('selector-used?id=1'); } 890 891 892 893 894 895 896 899 some_page.css CSS Background Beacon
65. Moving to Sass
66. Daniel Espeset @danielespeset Existing preprocessor inlines @import, versions images 66
67. Daniel Espeset @danielespeset Linted for syntax (sorta) 67
68. 68 Every color defined in our CSS
69. Daniel Espeset @danielespeset • @import statements needed changed • Converter fixes 171,244 Sass-lint errors • Had to resolve hundreds of invalid CSS rules & selectors Converter script 69
70. Daniel Espeset @danielespeset Will fixing this break something? color: rgb(25555, -10, 0); 70
71. Daniel Espeset @danielespeset What about this one? color: rgb(0, 0, 0, 0); 71
72. Daniel Espeset @danielespeset CSS Source Legacy CSS Build Legacy Build 72
73. Daniel Espeset @danielespeset CSS Source Legacy CSS Build Converter Sass Build New CSS Build 73 Legacy Build
74. Daniel Espeset @danielespeset CSS Source Legacy CSS Build Converter Sass Build New CSS Build AST Differ Parse to AST Parse to AST 74 Legacy Build
75. Daniel Espeset @danielespeset CSS Source Legacy CSS Build Converter Sass Build 100% New CSS Build 75 Legacy Build
76. Daniel Espeset @danielespeset CSS Source Legacy CSS Build Converter Sass Build 100% Etsy New CSS Build 76 Legacy Build
77. Daniel Espeset @danielespeset CSS Source Legacy CSS Build Converter Sass Build 50% 50% New CSS Build 77 Legacy Build
78. Daniel Espeset @danielespeset CSS Source Legacy CSS Build New CSS Build Converter Sass Build 100% 78 Legacy Build
79. Daniel Espeset @danielespeset Sass Source New CSS Build Sass Build 100% 79
80. Daniel Espeset @danielespeset • No @extends or functions, ever • @mixins only defined in one global file • Nesting level < 4 • Variable definitions should in a dedicated file Restricting Sass 80
82. Daniel Espeset @danielespeset • Enforce globally unique variable names • Sass AST! More to do 82
83. What have we learned?
84. Daniel Espeset @danielespeset Disposability > Modularity 84
85. Daniel Espeset @danielespeset The frontend is getting more complex everywhere. 85
86. Daniel Espeset @danielespeset We need new strategies to manage and understand it. 86
87. Daniel Espeset @danielespeset It helps to have a team dedicated to this work. (or even just a person) 87
88. Daniel Espeset @danielespeset The browser is part of your distributed system, not just a client you support. 88
89. codeascraft.com etsy.com/careers Thank you. talks.desp.in/smashing