PuppetCamp Dublin

Roland Tritsch - VP Technical Operations, Gilt Groupe
"Building a continuous delivery platform for the biggest spike in e-commerce" at PuppetCamp Dublin 2012.

Transcript

1. Puppet Camp Dublin - 2012 Andrés Pereira <apereira@gilt.com> Roland Tritsch <rtritsch@gilt.com>
2. Gilt in a nutshell ... • 5, 500, 1000/200, 300, 1, 18, ...
3. Gilt in a nutshell ... • 2 DCs + AWS EC2 • CentOS Linux • CDN (Akamai) • Puppet, Cobbler, ... • Zenoss, Nagios, ... • Git & Gerrit • Jenkins Frontend (HTML[5], CSS, JS, ...) Load Balancing (Zeus) Aggregation (Java, Scala, ...) Load Balancing (Zeus) Services (Java, Scala, Ruby, ...) Database (PostgreSQL, MongoDB, ...)
4. Puppet at Gilt ... • We've been using Puppet for almost 3 years ◦ ~ 1000 puppet modules • Typical setup ◦ Apache httpd+passenger for Puppetmaster ◦ Puppetcommander (MCollective plugin) to schedule puppet runs ◦ Puppet Dashboard (Reporting) ◦ In-house External Node Classifier (aka 'Mothership')
5. Puppet - Incremental rollout
6. Puppet - Incremental rollout • Deployment of changes ◦ Commit to 'development' branch ◦ Push changes for code review ◦ Once changes have been reviewed and verified, they get merged into 'development' ◦ A 'canary' flag can be set for a number of nodes ◦ Change is cherry-picked to 'master' branch ◦ A 'tag' from master is created and deployed ▪ See behavior in the canary environment ▪ Expand the canary env. Rinse & repeat ▪ Remove canary flag
7. http://mothership.sourceforge.net/ Puppet - External node Classifier
8. Puppet - External node Classifier • Mothership ◦ In-house assets management & provisioning tool. It also acts as an External Node Classifier for Puppet ▪ Can provision bare-metal & virtual machines ▪ Integrated with Cobbler ▪ Also manages users/groups & sudoers • Synchronized to LDAP • Puppet deploys sudoers file ▪ DNS management ▪ Hosts can have one or more labels (tags) that are mapped to Puppet modules
11. Lessons learned / Best practices ... Puppet ... • Don't just run it! • Node mgmt can become difficult (+100) • Keep change in mind (e.g. OS upgrades) Mothership ... • ... is not enough. You need at least 2 views ◦ Logical (deployed/reserved), Physical • You need to live with Physical != Logical ◦ Physical will be generated. It is a feedback loop.
12. Next steps ...
13. Requirements ... • Accelerate our ability to do incremental deployments (multiple times a day) ◦ ... with easy rollback ◦ ... accelerating our ability to innovate ◦ ... while maintaining 100% uptime • The "Happy Path" needs to be without human intervention ◦ From Commit to A/B test
14. Architecture ... Ion Cannon Code - Gerrit Build - Jenkins/Nexus QA - Selenium / TestComplete Performance Deployment - Puppet Analytics
15. Takeaways ... • Keep your modules small, keep them simple • Keep change in mind ◦ Do incremental rollout of changes ◦ Provide a 'default' case in your puppet modules and use 'fail' (sdtlib) as a default case. • If you are looking for a complete CD platform, then you need to embed Puppet into a larger solution
16. Questions?