Upgrade to Pro — share decks privately, control downloads, hide ads and more …

PuppetCamp Dublin

A8dd98306ac2cc522f9bff45872a33a7?s=47 Puppet Labs
October 24, 2012

PuppetCamp Dublin

Roland Tritsch - VP Technical Operations, Gilt Groupe
"Building a continuous delivery platform for the biggest spike in e-commerce" at PuppetCamp Dublin 2012.

A8dd98306ac2cc522f9bff45872a33a7?s=128

Puppet Labs

October 24, 2012
Tweet

Transcript

  1. Puppet Camp Dublin - 2012 Andrés Pereira <apereira@gilt.com> Roland Tritsch

    <rtritsch@gilt.com>
  2. Gilt in a nutshell ... • 5, 500, 1000/200, 300,

    1, 18, ...
  3. Gilt in a nutshell ... • 2 DCs + AWS

    EC2 • CentOS Linux • CDN (Akamai) • Puppet, Cobbler, ... • Zenoss, Nagios, ... • Git & Gerrit • Jenkins Frontend (HTML[5], CSS, JS, ...) Load Balancing (Zeus) Aggregation (Java, Scala, ...) Load Balancing (Zeus) Services (Java, Scala, Ruby, ...) Database (PostgreSQL, MongoDB, ...)
  4. Puppet at Gilt ... • We've been using Puppet for

    almost 3 years ◦ ~ 1000 puppet modules • Typical setup ◦ Apache httpd+passenger for Puppetmaster ◦ Puppetcommander (MCollective plugin) to schedule puppet runs ◦ Puppet Dashboard (Reporting) ◦ In-house External Node Classifier (aka 'Mothership')
  5. Puppet - Incremental rollout

  6. Puppet - Incremental rollout • Deployment of changes ◦ Commit

    to 'development' branch ◦ Push changes for code review ◦ Once changes have been reviewed and verified, they get merged into 'development' ◦ A 'canary' flag can be set for a number of nodes ◦ Change is cherry-picked to 'master' branch ◦ A 'tag' from master is created and deployed ▪ See behavior in the canary environment ▪ Expand the canary env. Rinse & repeat ▪ Remove canary flag
  7. http://mothership.sourceforge.net/ Puppet - External node Classifier

  8. Puppet - External node Classifier • Mothership ◦ In-house assets

    management & provisioning tool. It also acts as an External Node Classifier for Puppet ▪ Can provision bare-metal & virtual machines ▪ Integrated with Cobbler ▪ Also manages users/groups & sudoers • Synchronized to LDAP • Puppet deploys sudoers file ▪ DNS management ▪ Hosts can have one or more labels (tags) that are mapped to Puppet modules
  9. None
  10. None
  11. Lessons learned / Best practices ... Puppet ... • Don't

    just run it! • Node mgmt can become difficult (+100) • Keep change in mind (e.g. OS upgrades) Mothership ... • ... is not enough. You need at least 2 views ◦ Logical (deployed/reserved), Physical • You need to live with Physical != Logical ◦ Physical will be generated. It is a feedback loop.
  12. Next steps ...

  13. Requirements ... • Accelerate our ability to do incremental deployments

    (multiple times a day) ◦ ... with easy rollback ◦ ... accelerating our ability to innovate ◦ ... while maintaining 100% uptime • The "Happy Path" needs to be without human intervention ◦ From Commit to A/B test
  14. Architecture ... Ion Cannon Code - Gerrit Build - Jenkins/Nexus

    QA - Selenium / TestComplete Performance Deployment - Puppet Analytics
  15. Takeaways ... • Keep your modules small, keep them simple

    • Keep change in mind ◦ Do incremental rollout of changes ◦ Provide a 'default' case in your puppet modules and use 'fail' (sdtlib) as a default case. • If you are looking for a complete CD platform, then you need to embed Puppet into a larger solution
  16. Questions?