Upgrade to Pro — share decks privately, control downloads, hide ads and more …

PuppetCamp Dublin

Puppet Labs
October 24, 2012

PuppetCamp Dublin

Roland Tritsch - VP Technical Operations, Gilt Groupe
"Building a continuous delivery platform for the biggest spike in e-commerce" at PuppetCamp Dublin 2012.

Puppet Labs

October 24, 2012
Tweet

More Decks by Puppet Labs

Other Decks in Technology

Transcript

  1. Gilt in a nutshell ... • 2 DCs + AWS

    EC2 • CentOS Linux • CDN (Akamai) • Puppet, Cobbler, ... • Zenoss, Nagios, ... • Git & Gerrit • Jenkins Frontend (HTML[5], CSS, JS, ...) Load Balancing (Zeus) Aggregation (Java, Scala, ...) Load Balancing (Zeus) Services (Java, Scala, Ruby, ...) Database (PostgreSQL, MongoDB, ...)
  2. Puppet at Gilt ... • We've been using Puppet for

    almost 3 years ◦ ~ 1000 puppet modules • Typical setup ◦ Apache httpd+passenger for Puppetmaster ◦ Puppetcommander (MCollective plugin) to schedule puppet runs ◦ Puppet Dashboard (Reporting) ◦ In-house External Node Classifier (aka 'Mothership')
  3. Puppet - Incremental rollout • Deployment of changes ◦ Commit

    to 'development' branch ◦ Push changes for code review ◦ Once changes have been reviewed and verified, they get merged into 'development' ◦ A 'canary' flag can be set for a number of nodes ◦ Change is cherry-picked to 'master' branch ◦ A 'tag' from master is created and deployed ▪ See behavior in the canary environment ▪ Expand the canary env. Rinse & repeat ▪ Remove canary flag
  4. Puppet - External node Classifier • Mothership ◦ In-house assets

    management & provisioning tool. It also acts as an External Node Classifier for Puppet ▪ Can provision bare-metal & virtual machines ▪ Integrated with Cobbler ▪ Also manages users/groups & sudoers • Synchronized to LDAP • Puppet deploys sudoers file ▪ DNS management ▪ Hosts can have one or more labels (tags) that are mapped to Puppet modules
  5. Lessons learned / Best practices ... Puppet ... • Don't

    just run it! • Node mgmt can become difficult (+100) • Keep change in mind (e.g. OS upgrades) Mothership ... • ... is not enough. You need at least 2 views ◦ Logical (deployed/reserved), Physical • You need to live with Physical != Logical ◦ Physical will be generated. It is a feedback loop.
  6. Requirements ... • Accelerate our ability to do incremental deployments

    (multiple times a day) ◦ ... with easy rollback ◦ ... accelerating our ability to innovate ◦ ... while maintaining 100% uptime • The "Happy Path" needs to be without human intervention ◦ From Commit to A/B test
  7. Architecture ... Ion Cannon Code - Gerrit Build - Jenkins/Nexus

    QA - Selenium / TestComplete Performance Deployment - Puppet Analytics
  8. Takeaways ... • Keep your modules small, keep them simple

    • Keep change in mind ◦ Do incremental rollout of changes ◦ Provide a 'default' case in your puppet modules and use 'fail' (sdtlib) as a default case. • If you are looking for a complete CD platform, then you need to embed Puppet into a larger solution