Wayfair's Deployment Evolution and Tooling

Dan Rowe (Wayfair)

Dan shares the evolution of Wayfair's deployment tooling, from FTPing files onto a shared server to building their own tools to improve the process.

Presented at the Boston DevOps Meetup May 18 2016.

http://www.meetup.com/Boston-Devops/events/230870643/

Transcript

1. Wayfair's Deployment Evolution and Tooling May 2016
2. 2 Who Am I? Dan Rowe I lead the InfraSwat team at Wayfair About me: • Building deploy systems since before it was kool • Passionate about making everything self service (monitoring, alerting, deploying, ….) • Comfortable with Python and pythons (yes it’s an anaconda and not a python -->)
3. 3 Agenda Evolution of: • Deploys • Collaboration • Testing and tooling
4. 4 Evolution of Deploys • Wayfair moves fast and we iterate and improve on our tooling and workflows as fast as we develop and deploy new site features. • Wayfair didn’t start with a Deployment system with advanced features, but it’s evolved into that over time.
5. 5 Early Days of Rapid Deployment • 1 web server • Developer checked out a file to work on it • All developers deployed via FTP • Deploy: 1. Upload file 2. Cross fingers 3. Check site 4. Upload previous version if broken
6. 6 Scaling Rapid Deployment • Added more servers. • Developers still deployed via FTP • Synced out to all servers via Rsync like tool • Deploy: 1. Upload File 2. Cross fingers and toes 3. Check site 4. Upload previous version if broken
7. 7 Deploy Tool • Solved replication issues • Integrated with SVN and Git • Prevented the dreaded accidental folder drag • Was the beginning of a deploy pipeline • Solved auditability
8. 8 Deploy Tool Improved • Only Unit tests initially • Added Selenium tests later • Added Staging servers • Was great except developers had a hard time testing in dev • Hard to configure/reproduce Selenium issues for debugging
9. 9 Deploy Trains • Scaling past 100s of deploys a day. • Conductor to help identify issues in other areas • Grouping of integrated and tested code in one deploy. • Runs code through tests, staging and then if verified off to prod it goes • More eyes on deploy, more time to bake before next change set goes out
10. 10 Collaboration and Testing Evolution • Deploying fast is great, but we needed a way to show each other what our changes do • This included other developers as well as business and creative users. • This also evolved as the teams scaled and new tooling was developed
11. 11 Early day collaboration • In a small team it’s easy to have someone just swing over. • Collaborate at the developers desk
12. 12 Dev Switcher • Give ability for developers to share what they are working on. • Give ability for Business users to switch to any engineers environment and test functionality • Increased number of engineers and parallel projects caused developer idle time
13. 13 QaGit: Business Validation • Developer Pushes to remote git branch with Ticket ID in name • Business users can checkout an environment to test/verify the changes • Allows developers to keep coding on a different branch
14. 14 Autotest: Selenium test framework • Homegrown testing framework • Managed by a small team • Tests were created and maintained by the testing team • Gave us better confidence in changes working • Tests constantly broke with developer changes • Did not scale with the size of the Engineering team
15. 15 Codeception: New Selenium test framework • Switched to an Opensource framework • Trained Developers to write and run tests just like their unit tests • Troubleshooting and test maintenance owned by the teams that manage the features a tests exercises. • Testing team able to focus on framework improvements
16. 16 Wayrunner • Command line tool to tie together all the pieces of an engineers workflow • Runs code sniffs and unit tests local to the developers VM. • Pushes patch of developer changes to a VM to run selenium tests against. • Pushes changes to Reveiwboard for codereview. • Allows developers to join train with their changes. • Has similar functionality to other Try implementations. https://wiki.mozilla.org/ReleaseEngineering/TryServer https://github.com/etsy/TryLib http://docs.buildbot.net/0.8.4/try.html#try
17. 17 Never Done • Testing Containers as the next iteration of deployment pipeline. • Testing a Local VM vs. Centrally managed VMs. • And many more things on the roadmap
18. 18 The End Questions? Wayfair is Hiring Checkout our openings at: http://wayfair.com/careers