Scaling Deployment at Etsy

Slide 1

Slide 1 text

Scaling Deployment at Etsy Daniel Schauenberg [email protected] @mrtazz Thursday, April 18, 13

Slide 2

Slide 2 text

Thursday, April 18, 13

Slide 3

Slide 3 text

the marketplace we make together Thursday, April 18, 13

Slide 4

Slide 4 text

February 2013 •1.49 billion page views •4,215,169 items sold •$94.7 million of goods sold •22+ million members •800,000+ active shops http://www.etsy.com/blog/news/2013/etsy-statistics-february-2013-weather-report/ | Items by RockerDollJewellery, ZulamimiLand, codice, 42Things Thursday, April 18, 13

Slide 5

Slide 5 text

LAMMP Item by TheBackPackShoppe Thursday, April 18, 13

Slide 6

Slide 6 text

Item by FrankelPhotos Monolithic App Thursday, April 18, 13

Slide 7

Slide 7 text

No Branching Item by NurseryWallArt Thursday, April 18, 13

Slide 8

Slide 8 text

Deploy Frequency Thursday, April 18, 13

Slide 9

Slide 9 text

First Day Item by ﬂowersandﬂeurons Thursday, April 18, 13

Slide 10

Slide 10 text

Thursday, April 18, 13

Slide 11

Slide 11 text

IRC Thursday, April 18, 13

Slide 12

Slide 12 text

Thursday, April 18, 13

Slide 13

Slide 13 text

Developer VMs •KVM •Dev version of full Etsy stack •Chef •DevTools Thursday, April 18, 13

Slide 14

Slide 14 text

Thursday, April 18, 13

Slide 15

Slide 15 text

Item by codecards Thursday, April 18, 13

Slide 16

Slide 16 text

% review -r dschauenberg Thursday, April 18, 13

Slide 17

Slide 17 text

automatically assigned automatically assigned Thursday, April 18, 13

Slide 18

Slide 18 text

Try Item by CSSDesign Thursday, April 18, 13

Slide 19

Slide 19 text

Actually ... Thursday, April 18, 13

Slide 20

Slide 20 text

CI - The Old Way •~10 buildtest servers, single SSD •running multiple Jenkins executors •tests pegging IO •Jenkins is much better at running single executors Thursday, April 18, 13

Slide 21

Slide 21 text

The Bobs Item by Signz Thursday, April 18, 13

Slide 22

Slide 22 text

The Bobs •LXC containers on buildtests •Multiple SSDs •Labels for heavy/any execution •One heavy executor per disk Thursday, April 18, 13

Slide 23

Slide 23 text

CI/Try •~200 Bobs •Mostly for try •Constant monitoring for slow tests Thursday, April 18, 13

Slide 24

Slide 24 text

push train Item by decomodwalls Thursday, April 18, 13

Slide 25

Slide 25 text

#push •IRC channel to organize push trains •Join a train if you want to deploy changes •Schedule is planned via the channel topic •First in the train is the driver (controls the deploy) •Opening hours: 7am - 10pm NYC time Thursday, April 18, 13

Slide 26

Slide 26 text

#push kseever* + jameslee | jpaul | DanielConvissor (c) Thursday, April 18, 13

Slide 27

Slide 27 text

#push bateman* + krunal* + enorris* | tristan (c) + jameslee (c) + jlaster (c) | dawa + corey + sandosh + jklein + magera + seth_home + mpascual + nathan | bateman | russp (c) Thursday, April 18, 13

Slide 28

Slide 28 text

pushbot •.join •.in •.good •.done Thursday, April 18, 13

Slide 29

Slide 29 text

pushbot Thursday, April 18, 13

Slide 30

Slide 30 text

https://github.com/etsy/PushBot Thursday, April 18, 13

Slide 31

Slide 31 text

Item by EsalonPhotography Thursday, April 18, 13

Slide 32

Slide 32 text

Deployinator Thursday, April 18, 13

Slide 33

Slide 33 text

Thursday, April 18, 13

Slide 34

Slide 34 text

Thursday, April 18, 13

Slide 35

Slide 35 text

stale commits Thursday, April 18, 13

Slide 36

Slide 36 text

version checks buttons disabled buttons disabled Thursday, April 18, 13

Slide 37

Slide 37 text

version checks Thursday, April 18, 13

Slide 38

Slide 38 text

lock down deploys Thursday, April 18, 13

Slide 39

Slide 39 text

https://github.com/etsy/deployinator Thursday, April 18, 13

Slide 40

Slide 40 text

Dashboards deploy lines deploy lines Thursday, April 18, 13

Slide 41

Slide 41 text

https://github.com/etsy/dashboard Thursday, April 18, 13

Slide 42

Slide 42 text

Supergrep Thursday, April 18, 13

Slide 43

Slide 43 text

https://github.com/etsy/supergrep Thursday, April 18, 13

Slide 44

Slide 44 text

Thursday, April 18, 13

Slide 45

Slide 45 text

Summary •Current setup has scaled to ~150 people •Constantly trying to improve the speed of deployment •Find weak parts in the process and make them more robust/faster •Bring Dev closer to Prod •Not being able to deploy has the same status as the site being down Thursday, April 18, 13

Slide 46

Slide 46 text

codeascraft.etsy.com www.etsy.com/codeascraft/talks etsy.github.com www.etsy.com/careers Thursday, April 18, 13

Slide 47

Slide 47 text

Questions? Thursday, April 18, 13

Slide 48

Slide 48 text

Scaling Deployment at Etsy Daniel Schauenberg [email protected] @mrtazz Thursday, April 18, 13