Continuous-Delivery-Meetup-NYC-2013.pdf

How we tackle deployments and keep them fast at Etsy.

Reprised version with some additions for the Continuous Delivery NYC meetup October 2013.

Transcript

1. Scaling Deployment at Etsy
   Daniel Schauenberg
   dschauenberg@etsy.com
   @mrtazz
   Thursday, October 10, 13
   

   View Slide

2. Thursday, October 10, 13
   

   View Slide

3. August 2013
   •1.8 billion page views
   •5,483,399 items sold
   •$109.1 million of goods sold
   •> 30 million members
   •> 1 million active shops
   http://www.etsy.com/blog/news/2013/etsy-statistics-august-2013-weather-report/ | Items by RockerDollJewellery, ZulamimiLand, codice, 42Things
   Thursday, October 10, 13
   

   View Slide

4. LAMMP
   Item by TheBackPackShoppe
   Thursday, October 10, 13
   

   View Slide

5. Item by FrankelPhotos
   Monolithic App
   Thursday, October 10, 13
   

   View Slide

6. No Branching
   Item by NurseryWallArt
   Thursday, October 10, 13
   

   View Slide

7. Deploy Frequency
   Thursday, October 10, 13
   

   View Slide

8. First Day
   Item by flowersandfleurons
   Thursday, October 10, 13
   

   View Slide

9. Thursday, October 10, 13
   

   View Slide

10. IRC
    Thursday, October 10, 13
    

    View Slide

11. Thursday, October 10, 13
    

    View Slide

12. Developer VMs
    •KVM
    •Dev version of full Etsy stack
    •Chef
    •DevTools
    Thursday, October 10, 13
    

    View Slide

13. Thursday, October 10, 13
    

    View Slide

14. Item by codecards
    Thursday, October 10, 13
    

    View Slide

15. % review -r dschauenberg
    Thursday, October 10, 13
    

    View Slide

16. automatically assigned
    automatically assigned
    Thursday, October 10, 13
    

    View Slide

17. Try
    Item by CSSDesign
    Thursday, October 10, 13
    

    View Slide

18. Actually ...
    Thursday, October 10, 13
    

    View Slide

19. The Bobs
    Item by Signz
    Thursday, October 10, 13
    

    View Slide

20. The Bobs
    •LXC containers on buildtests
    •Multiple SSDs
    •Labels for heavy/any execution
    •One heavy executor per disk
    Thursday, October 10, 13
    

    View Slide

21. CI/Try
    •~260 Bobs
    •Mostly for try
    •Constant monitoring for slow tests
    Thursday, October 10, 13
    

    View Slide

22. push train
    Item by decomodwalls
    Thursday, October 10, 13
    

    View Slide

23. #push
    •IRC channel to organize push trains
    •Join a train if you want to deploy changes
    •Schedule is planned via the channel topic
    •First in the train is the driver (controls the deploy)
    •Opening hours: 7am - 10pm NYC time
    Thursday, October 10, 13
    

    View Slide

24. #push
    kseever* + jameslee | jpaul | DanielConvissor (c)
    Thursday, October 10, 13
    

    View Slide

25. #push
    bateman* + krunal* + enorris* | tristan (c) + jameslee (c)
    + jlaster (c) | dawa + corey + sandosh + jklein + magera
    + seth_home + mpascual + nathan | bateman | russp (c)
    Thursday, October 10, 13
    

    View Slide

26. pushbot
    •.join
    •.in
    •.good
    •.done
    Thursday, October 10, 13
    

    View Slide

27. pushbot
    Thursday, October 10, 13
    

    View Slide

28. Item by EsalonPhotography
    Thursday, October 10, 13
    

    View Slide

29. Deployinator
    Thursday, October 10, 13
    

    View Slide

30. Thursday, October 10, 13
    

    View Slide

31. Thursday, October 10, 13
    

    View Slide

32. stale commits
    Thursday, October 10, 13
    

    View Slide

33. version checks
    buttons disabled
    buttons disabled
    Thursday, October 10, 13
    

    View Slide

34. version checks
    Thursday, October 10, 13
    

    View Slide

35. lock down deploys
    Thursday, October 10, 13
    

    View Slide

36. https://github.com/etsy/deployinator
    Thursday, October 10, 13
    

    View Slide

37. Downsides
    •Deploys not atomic on the request level
    •Limbo during the time of the local rsync
    •Common strategy was to split commits into 3 deploys
    Thursday, October 10, 13
    

    View Slide

38. Item by Geographicsart
    Thursday, October 10, 13
    

    View Slide

39. Atomic Deploys
    Thursday, October 10, 13
    

    View Slide

40. Basic Idea
    Yin Yang
    Active Docroot
    Thursday, October 10, 13
    

    View Slide

41. Basic Idea
    Yin Yang
    Active Docroot
    rsync
    Thursday, October 10, 13
    

    View Slide

42. Basic Idea
    Yin Yang
    Active Docroot
    Thursday, October 10, 13
    

    View Slide

43. Basic Idea
    Yin Yang
    Active Docroot
    Thursday, October 10, 13
    

    View Slide

44. Problems
    •Symlink swap during requests
    •Code needs to be guaranteed to finish on the docroot
    it started
    •Code inclusion mid request
    Thursday, October 10, 13
    

    View Slide

45. etsy/mod_realdoc
    •Apache post_read_request hook
    •Whole request works on realpath of docroot
    •Caches realpath for 2s
    Thursday, October 10, 13
    

    View Slide

46. ini_set('include_path', $_SERVER['DOCUMENT_ROOT'].'/../include');
    Thursday, October 10, 13
    

    View Slide

47. etsy/incpath
    •PHP module to set the incpath
    •Gets docroot from Apache or realpath() itself
    •Looks for a pattern to replace in include_path
    •Restores include_path at the end of the request
    Thursday, October 10, 13
    

    View Slide

48. What did we get?
    •Remove functions and call site in same deploy
    •No restarts necessary
    •Opcode caches stay warm for files that don’t change
    between 2 deploys
    Thursday, October 10, 13
    

    View Slide

49. Things to watch out for
    •Code that uses full path names to scripts
    •Atomic symlink swapping with `mv -T`
    •Realpath caching to not stress the filesystem
    •Opcode cache needs to fit 2x code size
    •Only request atomicity
    Thursday, October 10, 13
    

    View Slide

50. The Plateau
    Item by finandfancy
    Thursday, October 10, 13
    

    View Slide

51. The Plateau
    •Regular deploys took ~15 mins
    •Config deploys about half
    •10am - 6pm => ~ 32 deploys
    •Long waiting times
    Thursday, October 10, 13
    

    View Slide

52. Item by KlaireWarren
    Thursday, October 10, 13
    

    View Slide

53. Waiting for push queue
    Waiting for push queue
    Thursday, October 10, 13
    

    View Slide

54. Split The Queues
    Item by KlaireWarren
    Thursday, October 10, 13
    

    View Slide

55. HELLO SPLIT QUEUES
    HELLO SPLIT QUEUES
    Thursday, October 10, 13
    

    View Slide

56. Dashboards
    deploy lines
    deploy lines
    Thursday, October 10, 13
    

    View Slide

57. Supergrep
    Thursday, October 10, 13
    

    View Slide

58. Thursday, October 10, 13
    

    View Slide

59. Summary
    •Current setup has scaled to ~150 people
    •Constantly trying to improve the speed of deployment
    •Find weak parts in the process and make them more
    robust/faster
    •Bring Dev closer to Prod
    •Not being able to deploy has the same status as the site
    being down
    Thursday, October 10, 13
    

    View Slide

60. codeascraft.etsy.com
    www.etsy.com/codeascraft/talks
    etsy.github.com
    www.etsy.com/careers
    Thursday, October 10, 13
    

    View Slide

61. Questions?
    Thursday, October 10, 13
    

    View Slide

62. Scaling Deployment at Etsy
    Daniel Schauenberg
    dschauenberg@etsy.com
    @mrtazz
    Thursday, October 10, 13
    

    View Slide