August 2013 •1.8 billion page views •5,483,399 items sold •$109.1 million of goods sold •> 30 million members •> 1 million active shops http://www.etsy.com/blog/news/2013/etsy-statistics-august-2013-weather-report/ | Items by RockerDollJewellery, ZulamimiLand, codice, 42Things Thursday, October 10, 13
#push •IRC channel to organize push trains •Join a train if you want to deploy changes •Schedule is planned via the channel topic •First in the train is the driver (controls the deploy) •Opening hours: 7am - 10pm NYC time Thursday, October 10, 13
Downsides •Deploys not atomic on the request level •Limbo during the time of the local rsync •Common strategy was to split commits into 3 deploys Thursday, October 10, 13
Problems •Symlink swap during requests •Code needs to be guaranteed to finish on the docroot it started •Code inclusion mid request Thursday, October 10, 13
etsy/incpath •PHP module to set the incpath •Gets docroot from Apache or realpath() itself •Looks for a pattern to replace in include_path •Restores include_path at the end of the request Thursday, October 10, 13
What did we get? •Remove functions and call site in same deploy •No restarts necessary •Opcode caches stay warm for files that don’t change between 2 deploys Thursday, October 10, 13
Things to watch out for •Code that uses full path names to scripts •Atomic symlink swapping with `mv -T` •Realpath caching to not stress the filesystem •Opcode cache needs to fit 2x code size •Only request atomicity Thursday, October 10, 13
Summary •Current setup has scaled to ~150 people •Constantly trying to improve the speed of deployment •Find weak parts in the process and make them more robust/faster •Bring Dev closer to Prod •Not being able to deploy has the same status as the site being down Thursday, October 10, 13