Frontend Infrastructure at Etsy

Slide 1

Slide 1 text

Daniel Espeset @danielespeset Smashing Conf NYC, June 2015 at Etsy Frontend Infrastructure

Slide 2

Slide 2 text

talks.desp.in/smashing Resources for this talk

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Daniel Espeset @danielespeset A front-end operations engineer is not a title you’ve likely come across, but hopefully one that you will [...] an expert at serving and hosting front-end resources. [finding] the best ways to piece together the parts of a Web application, and [being a pro] at versioning, caching and deployment. Alex Sexton @slexaxton "Front-End Ops", Smashing Magazine, July 2013 “ ” 4

Slide 5

Slide 5 text

Daniel Espeset @danielespeset Frontend Infrastructure 5 Daniel Espeset Daniel Na Takashi Mizohata Etsy Seth Walker Sufian Rhazi Jonathan Lai

Slide 6

Slide 6 text

Etsy Engineering

Slide 7

Slide 7 text

Daniel Espeset @danielespeset Anybody can touch (mostly) anything 7

Slide 8

Slide 8 text

Daniel Espeset @danielespeset Blameless postmortems 8

Slide 9

Slide 9 text

Daniel Espeset @danielespeset Everyone deploys 9

Slide 10

Slide 10 text

Continuous Deployment

Slide 11

Slide 11 text

Daniel Espeset @danielespeset Push Button Deploys 11

Slide 12

Slide 12 text

pages/search_results.php if (Feature::isEnabled('my_new_feature')) { // TODO build this part } else { // Existing code } 20 21 22 .. 100 12 config/production.php // Owner: despeset, frontend infrastructure server_config['my_new_feature'] = ( 'enabled' => 0 ); 890 891 892 893 Checking a flag Defining a flag

Slide 13

Slide 13 text

13 desp .join pushbot has changed topic to: " seth dna* gio* | desp" pushbot desp jklein : You're up desp .in pushbot has changed topic to: "desp jklein" pushbot has changed topic to: "desp* jklein" jklein .in pushbot has changed topic to: "desp* jklein*" pushbot desp jklein : everyone is ready Coordinating Deploys

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

19 25-50Pushes a day

Slide 20

Slide 20 text

20 // Owner: despeset, frontend infrastructure server_config['my_new_feature'] = ( 'enabled' => 0, 'group' => 9903 ); 890 891 892 893 894 config/production.php

Slide 21

Slide 21 text

21 config/production.php // Owner: despeset, frontend infrastructure server_config['my_new_feature'] = ( 'enabled' => 0, 'group' => 9903, 'admin' => 'on' ); 890 891 892 893 894 895

Slide 22

Slide 22 text

22 // Owner: despeset, frontend infrastructure server_config['my_new_feature'] = ( 'enabled' => 50, 'group' => 9903, 'admin' => 'on' ); 890 891 892 893 894 895 config/production.php

Slide 23

Slide 23 text

Continuous Experimentation

Slide 24

Slide 24 text

Daniel Espeset @danielespeset idea code release idea code release? A/B test idea prototy release A/B test validate A/B test refine 2005 2010 2012 Shamelessly stolen from Dan McKinley mcfunley.com/data-driven-products-now Evolution of Continuous Experimentation 24

Slide 25

Slide 25 text

Daniel Espeset @danielespeset • Engineers are trusted and given agency • The production codebase is constantly changing • That codebase is filled with feature flags In a nutshell 25

Slide 26

Slide 26 text

Daniel Espeset @danielespeset 26

Slide 27

Slide 27 text

Daniel Espeset @danielespeset 27 Sass conversion

Slide 28

Slide 28 text

Daniel Espeset @danielespeset Our frontend is made up of >300 different “pages” 28

Slide 29

Slide 29 text

Daniel Espeset @danielespeset At any given time we may be running dozens of tests per page 29

Slide 30

Slide 30 text

Daniel Espeset @danielespeset 4,096 212= 30

Slide 31

Slide 31 text

Daniel Espeset @danielespeset 212 300 x =1.2 M 31

Slide 32

Slide 32 text

Daniel Espeset @danielespeset Reasoning about our frontend can be hard 32

Slide 33

Slide 33 text

Daniel Espeset @danielespeset Deleting code is often scary 33

Slide 34

Slide 34 text

Daniel Espeset @danielespeset Verifying changes is difficult 34

Slide 35

Slide 35 text

Frontend Infrastructure

Slide 36

Slide 36 text

Daniel Espeset @danielespeset 36 Performance Allison McKnight Kristyn Reith Etsy Natalya Hoota Lara Hogan

Slide 37

Slide 37 text

Daniel Espeset @danielespeset source code deploy build system 37

Slide 38

Slide 38 text

Daniel Espeset @danielespeset source code production build 38 deploy build system

Slide 39

Slide 39 text

Daniel Espeset @danielespeset Frontend Infrastructure 39

Slide 40

Slide 40 text

Daniel Espeset @danielespeset 40 ...the bridge between an application's intent and an application's reality. Alex Sexton (JavaScript Laureate of Texas) "Front-End Ops", Smashing Magazine, July 2013 “ ”

Slide 41

Slide 41 text

{% css files="base" %}{% /css %} Template Code Rendered HTML most recent last modified date of file and all dependencies.

Slide 42

Slide 42 text

{% css files="base" %}{% /css %} Template Code Rendered HTML adds version number, always happens.

Slide 43

Slide 43 text

{% css files="base" %}{% /css %} Template Code Rendered HTML if (rand() > 0.01) { log(template, asset); }

Slide 44

Slide 44 text

Where are my things? (Ranger)

Slide 45

Slide 45 text

Daniel Espeset @danielespeset + + JS graph Asset logs CSS graph Ranger combines 3 dependency graphs 45

Slide 46

Slide 46 text

Slide 47

Slide 47 text

Slide 48

Slide 48 text

Slide 49

Slide 49 text

Slide 50

Slide 50 text

Slide 51

Slide 51 text

Slide 52

Slide 52 text

Slide 53

Slide 53 text

Slide 54

Slide 54 text

Daniel Espeset @danielespeset Ranger gave us something valuable The set of files that aren't used 54

Slide 55

Slide 55 text

Profiling CSS (Shrinkray)

Slide 56

Slide 56 text

Daniel Espeset @danielespeset 56 • Pick a stylesheet loaded on the page Shrinkray - CSS analysis

Slide 57

Slide 57 text

Daniel Espeset @danielespeset 57 • Pick a stylesheet loaded on the page • Randomly pick 50 selectors via CSSOM Shrinkray - CSS analysis

Slide 58

Slide 58 text

Daniel Espeset @danielespeset 58 • Pick a stylesheet loaded on the page • Randomly pick 50 selectors via CSSOM • Search for matches with document.querySelector Shrinkray - CSS analysis

Slide 59

Slide 59 text

Daniel Espeset @danielespeset 59 • Pick a stylesheet loaded on the page • Randomly pick 50 selectors via CSSOM • Search for matches with document.querySelector • Send results to server Shrinkray - CSS analysis

Slide 60

Slide 60 text

Daniel Espeset @danielespeset • Pick a stylesheet loaded on the page • Randomly pick 50 selectors via CSSOM • Search for matches with document.querySelector • Send results to server • Aggregate data with Map Reduce Shrinkray - CSS analysis 60

Slide 61

Slide 61 text

Slide 62

Slide 62 text

Daniel Espeset @danielespeset • Low confidence on long-tail pages • Doesn't capture CSS appearing by user action Why doesn't it work yet? 62

Slide 63

Slide 63 text

Daniel Espeset @danielespeset • Increase measurement volume if perf allows • Add time skew to measurements • Intelligently target selectors based on the data • Add a second pass using an more invasive but thorough technique Possible solutions 63

Slide 64

Slide 64 text

64 #container .button-cool ul { color: pink; float: left; } #container .button-cool ul:before { content: ''; background-image: url('selector-used?id=1'); } 890 891 892 893 894 895 896 899 some_page.css CSS Background Beacon

Slide 65

Slide 65 text

Moving to Sass

Slide 66

Slide 66 text

Daniel Espeset @danielespeset Existing preprocessor inlines @import, versions images 66

Slide 67

Slide 67 text

Daniel Espeset @danielespeset Linted for syntax (sorta) 67

Slide 68

Slide 68 text

68 Every color defined in our CSS

Slide 69

Slide 69 text

Daniel Espeset @danielespeset • @import statements needed changed • Converter fixes 171,244 Sass-lint errors • Had to resolve hundreds of invalid CSS rules & selectors Converter script 69

Slide 70

Slide 70 text

Daniel Espeset @danielespeset Will fixing this break something? color: rgb(25555, -10, 0); 70

Slide 71

Slide 71 text

Daniel Espeset @danielespeset What about this one? color: rgb(0, 0, 0, 0); 71

Slide 72

Slide 72 text

Daniel Espeset @danielespeset CSS Source Legacy CSS Build Legacy Build 72

Slide 73

Slide 73 text

Daniel Espeset @danielespeset CSS Source Legacy CSS Build Converter Sass Build New CSS Build 73 Legacy Build

Slide 74

Slide 74 text

Daniel Espeset @danielespeset CSS Source Legacy CSS Build Converter Sass Build New CSS Build AST Differ Parse to AST Parse to AST 74 Legacy Build

Slide 75

Slide 75 text

Daniel Espeset @danielespeset CSS Source Legacy CSS Build Converter Sass Build 100% New CSS Build 75 Legacy Build

Slide 76

Slide 76 text

Daniel Espeset @danielespeset CSS Source Legacy CSS Build Converter Sass Build 100% Etsy New CSS Build 76 Legacy Build

Slide 77

Slide 77 text

Daniel Espeset @danielespeset CSS Source Legacy CSS Build Converter Sass Build 50% 50% New CSS Build 77 Legacy Build

Slide 78

Slide 78 text

Daniel Espeset @danielespeset CSS Source Legacy CSS Build New CSS Build Converter Sass Build 100% 78 Legacy Build

Slide 79

Slide 79 text

Daniel Espeset @danielespeset Sass Source New CSS Build Sass Build 100% 79

Slide 80

Slide 80 text

Daniel Espeset @danielespeset • No @extends or functions, ever • @mixins only defined in one global file • Nesting level < 4 • Variable definitions should in a dedicated file Restricting Sass 80

Slide 81

Slide 81 text

Slide 82

Slide 82 text

Daniel Espeset @danielespeset • Enforce globally unique variable names • Sass AST! More to do 82

Slide 83

Slide 83 text

What have we learned?

Slide 84

Slide 84 text

Daniel Espeset @danielespeset Disposability > Modularity 84

Slide 85

Slide 85 text

Daniel Espeset @danielespeset The frontend is getting more complex everywhere. 85

Slide 86

Slide 86 text

Daniel Espeset @danielespeset We need new strategies to manage and understand it. 86

Slide 87

Slide 87 text

Daniel Espeset @danielespeset It helps to have a team dedicated to this work. (or even just a person) 87

Slide 88

Slide 88 text

Daniel Espeset @danielespeset The browser is part of your distributed system, not just a client you support. 88

Slide 89

Slide 89 text

codeascraft.com etsy.com/careers Thank you. talks.desp.in/smashing