Slide 1

Slide 1 text

Daniel Espeset @danielespeset Fronteers Amsterdam, October 2014 The Role of Frontend Infrastructure at Etsy Making Maps

Slide 2

Slide 2 text

talks.desp.in/fronteers2014 Resources for this talk

Slide 3

Slide 3 text

Daniel Espeset @danielespeset Frontend Infrastructure 3 Daniel Espeset Daniel Na Takashi Mizohata Etsy Seth Walker Sufian Rhazi

Slide 4

Slide 4 text

4

Slide 5

Slide 5 text

Daniel Espeset @danielespeset 1 Million Active Sellers 26 Million Active Listings 5

Slide 6

Slide 6 text

6 Etsy Engineering

Slide 7

Slide 7 text

Daniel Espeset @danielespeset 7 Anybody can touch anything

Slide 8

Slide 8 text

Daniel Espeset @danielespeset 8 Blameless postmortems

Slide 9

Slide 9 text

Daniel Espeset @danielespeset 9 Everyone deploys

Slide 10

Slide 10 text

10 Continuous Deployment

Slide 11

Slide 11 text

Daniel Espeset @danielespeset "The Button"

Slide 12

Slide 12 text

pages/search_results.php if (Feature::isEnabled('my_new_feature')) { // TODO build this part } else { // Existing code } 20 21 22 .. 100 12 config/production.php // Owner: despeset, frontend infrastructure server_config['my_new_feature'] = ( 'enabled' => 0 ); 890 891 892 893

Slide 13

Slide 13 text

13 desp .join pushbot has changed topic to: " seth dna* gio* | desp" pushbot desp jklein : You're up desp .in pushbot has changed topic to: "desp jklein" pushbot has changed topic to: "desp* jklein" jklein .in pushbot has changed topic to: "desp* jklein*" pushbot desp jklein : everyone is ready

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

19 25-50Pushes a day

Slide 20

Slide 20 text

20 config/production.php // Owner: despeset, frontend infrastructure server_config['my_new_feature'] = ( 'enabled' => 0, 'group' => 9903 ); 890 891 892 893 894

Slide 21

Slide 21 text

21 config/production.php // Owner: despeset, frontend infrastructure server_config['my_new_feature'] = ( 'enabled' => 0, 'group' => 9903, 'admin' => 'on' ); 890 891 892 893 894 895

Slide 22

Slide 22 text

22 config/production.php // Owner: despeset, frontend infrastructure server_config['my_new_feature'] = ( 'enabled' => 50, 'group' => 9903, 'admin' => 'on' ); 890 891 892 893 894 895

Slide 23

Slide 23 text

23 Continuous Experimentation

Slide 24

Slide 24 text

Daniel Espeset @danielespeset 24 idea code release idea code release? A/B test idea prototype release A/B test validate A/B test refine 2005 2010 2012 Shamelessly stolen from Dan McKinley mcfunley.com/data-driven-products-now Evolution of Continuous Experimentation

Slide 25

Slide 25 text

Daniel Espeset @danielespeset 25 • Engineers are trusted and have massive access • The production codebase is constantly changing • That codebase is filled with experiments TL;DR

Slide 26

Slide 26 text

Daniel Espeset @danielespeset 26

Slide 27

Slide 27 text

Daniel Espeset @danielespeset 27 Our frontend is made up of 300 different “pages”

Slide 28

Slide 28 text

Daniel Espeset @danielespeset 28 At any given time we may be running dozens of experiments per page

Slide 29

Slide 29 text

Daniel Espeset @danielespeset 29 4,096 212 =

Slide 30

Slide 30 text

Daniel Espeset @danielespeset 30 212 300 x = 1.2 M

Slide 31

Slide 31 text

Daniel Espeset @danielespeset 31 Reasoning about our frontend can be hard

Slide 32

Slide 32 text

Daniel Espeset @danielespeset 32 Deleting code is often scary

Slide 33

Slide 33 text

Daniel Espeset @danielespeset 33 Testing changes is difficult

Slide 34

Slide 34 text

34 Frontend Infrastructure

Slide 35

Slide 35 text

Daniel Espeset @danielespeset 35 source code deploy build system

Slide 36

Slide 36 text

Daniel Espeset @danielespeset 36 source code production build deploy build system

Slide 37

Slide 37 text

Daniel Espeset @danielespeset 37 Frontend Infrastructure

Slide 38

Slide 38 text

38 Builda [Build Assets]

Slide 39

Slide 39 text

Daniel Espeset @danielespeset • • Wraps RequireJS AMD compiler • Parallelizes work across CPU cores • Serves fully compiled assets on development VMS 39 Builda [Build Assets]

Slide 40

Slide 40 text

Daniel Espeset @danielespeset 40 Techniques for serving built assets in dev • Build on the fly in response to HTTP requests for asset files

Slide 41

Slide 41 text

Daniel Espeset @danielespeset 41 Techniques for serving built assets in dev • Build on the fly in response to HTTP requests for asset files • Watch FS for any changes and rebuild everything

Slide 42

Slide 42 text

Daniel Espeset @danielespeset 42 Techniques for serving built assets in dev • Build on the fly in response to HTTP requests for asset files • Watch FS for any changes and rebuild everything • Watch FS for any changes and only rebuild affected targets

Slide 43

Slide 43 text

Daniel Espeset @danielespeset 43 Techniques for serving built assets in dev • Build on the fly in response to HTTP requests for asset files • Watch FS for any changes and rebuild everything • Watch FS for any changes and only rebuild affected targets this is... complicated

Slide 44

Slide 44 text

Daniel Espeset @danielespeset 44 source code built assets Builda Source file is changed

Slide 45

Slide 45 text

Daniel Espeset @danielespeset 45 source code built assets Builda Builda looks up dependencies

Slide 46

Slide 46 text

Daniel Espeset @danielespeset 46 source code built assets Builda Only rebuilds files that are now stale

Slide 47

Slide 47 text

Daniel Espeset @danielespeset This solution gave us something valuable 47 A dependency graph

Slide 48

Slide 48 text

48 dependency_graph.json "mobile/modules/rwd": { "info": { "version": "20140815195505" }, "children": { "common/etsy.namespace": "amd" }, "parents": { "mobile/base-container": "amd" } } 201 202 203 204 205 206 207 208 209 210 211

Slide 49

Slide 49 text

49 Ranger

Slide 50

Slide 50 text

Daniel Espeset @danielespeset 50 + + JS graph render graph CSS graph Ranger combines 3 dependency graphs

Slide 51

Slide 51 text

51

Slide 52

Slide 52 text

52

Slide 53

Slide 53 text

53

Slide 54

Slide 54 text

54

Slide 55

Slide 55 text

55

Slide 56

Slide 56 text

56

Slide 57

Slide 57 text

57

Slide 58

Slide 58 text

58

Slide 59

Slide 59 text

Daniel Espeset @danielespeset Ranger gave us something valuable 59 The set of files that aren't used

Slide 60

Slide 60 text

60 Shrinkray

Slide 61

Slide 61 text

Daniel Espeset @danielespeset • Pick a stylesheet loaded on the page 61 Shrinkray - clientside CSS analysis

Slide 62

Slide 62 text

Daniel Espeset @danielespeset • Pick a stylesheet loaded on the page • Randomly pick 50 selectors via CSSOM 62 Shrinkray - clientside CSS analysis

Slide 63

Slide 63 text

Daniel Espeset @danielespeset • Pick a stylesheet loaded on the page • Randomly pick 50 selectors via CSSOM • Search for matches with document.querySelector 63 Shrinkray - clientside CSS analysis

Slide 64

Slide 64 text

Daniel Espeset @danielespeset • Pick a stylesheet loaded on the page • Randomly pick 50 selectors via CSSOM • Search for matches with document.querySelector • Send results to server 64 Shrinkray - clientside CSS analysis

Slide 65

Slide 65 text

Daniel Espeset @danielespeset • Pick a stylesheet loaded on the page • Randomly pick 50 selectors via CSSOM • Search for matches with document.querySelector • Send results to server • Aggregate data with Map Reduce 65 Shrinkray - clientside CSS analysis

Slide 66

Slide 66 text

66

Slide 67

Slide 67 text

67 Moving to SCSS

Slide 68

Slide 68 text

Daniel Espeset @danielespeset 68 Existing preprocessor inlines @import, versions images

Slide 69

Slide 69 text

Daniel Espeset @danielespeset 69 Linted for syntax (sorta)

Slide 70

Slide 70 text

70 Every color defined in our CSS

Slide 71

Slide 71 text

Daniel Espeset @danielespeset • No @extends ever • No @mixins (for now) • Nesting level < 4 • Lints to enforce these and strict code-style adherence 71 Restricting SCSS functionality

Slide 72

Slide 72 text

Daniel Espeset @danielespeset • @import statements needed changed • Converter fixes 171,244 scss-lint errors • Had to resolve hundreds of invalid CSS rules & selectors 72 Converter script

Slide 73

Slide 73 text

Daniel Espeset @danielespeset 73 Will fixing this break something? color: rgb(25555, -10, 0);

Slide 74

Slide 74 text

Daniel Espeset @danielespeset 74 What about this one? color: rgb(0, 0, 0, 0);

Slide 75

Slide 75 text

Daniel Espeset @danielespeset 75 CSS Source Legacy CSS Build Legacy Builda

Slide 76

Slide 76 text

Daniel Espeset @danielespeset 76 CSS Source Legacy CSS Build Legacy Builda Converter SCSS Builda New CSS Build

Slide 77

Slide 77 text

Daniel Espeset @danielespeset 77 CSS Source Legacy CSS Build Legacy Builda Converter SCSS Builda New CSS Build AST Differ Parse to AST Parse to AST

Slide 78

Slide 78 text

Daniel Espeset @danielespeset 78 CSS Source Legacy CSS Build Legacy Builda Converter SCSS Builda 100% New CSS Build

Slide 79

Slide 79 text

Daniel Espeset @danielespeset 79 CSS Source Legacy CSS Build Legacy Builda Converter SCSS Builda 100% Etsy New CSS Build

Slide 80

Slide 80 text

Daniel Espeset @danielespeset 80 CSS Source Legacy CSS Build Legacy Builda Converter SCSS Builda 50% 50% New CSS Build

Slide 81

Slide 81 text

Daniel Espeset @danielespeset 81 CSS Source Legacy CSS Build New CSS Build Legacy Builda Converter SCSS Builda 100%

Slide 82

Slide 82 text

Daniel Espeset @danielespeset 82 SCSS Source New CSS Build SCSS Builda 100%

Slide 83

Slide 83 text

What have we learned?

Slide 84

Slide 84 text

Daniel Espeset @danielespeset 84 Disposability > Modularity

Slide 85

Slide 85 text

Daniel Espeset @danielespeset 85 The frontend is getting more complex everywhere.

Slide 86

Slide 86 text

Daniel Espeset @danielespeset 86 We need new strategies to manage and understand it.

Slide 87

Slide 87 text

Daniel Espeset @danielespeset 87 The browser is part of your distributed system, not just a client you support.

Slide 88

Slide 88 text

Daniel Espeset @danielespeset 88 The more extensible your tool the more useful it will be.