PuppetCamp Dublin

Roland Tritsch - VP Technical Operations, Gilt Groupe
"Building a continuous delivery platform for the biggest spike in e-commerce" at PuppetCamp Dublin 2012.

Transcript

1. Puppet Camp
   Dublin - 2012
   Andrés Pereira
   Roland Tritsch
   

   View Slide

2. Gilt in a nutshell ...
   ● 5, 500, 1000/200, 300, 1, 18, ...
   

   View Slide

3. Gilt in a nutshell ...
   ● 2 DCs + AWS EC2
   ● CentOS Linux
   ● CDN (Akamai)
   ● Puppet, Cobbler, ...
   ● Zenoss, Nagios, ...
   ● Git & Gerrit
   ● Jenkins
   Frontend (HTML[5], CSS, JS, ...)
   Load Balancing (Zeus)
   Aggregation (Java, Scala, ...)
   Load Balancing (Zeus)
   Services (Java, Scala, Ruby, ...)
   Database (PostgreSQL, MongoDB, ...)
   

   View Slide

4. Puppet at Gilt ...
   ● We've been using Puppet for almost 3 years
   ○ ~ 1000 puppet modules
   ● Typical setup
   ○ Apache httpd+passenger for Puppetmaster
   ○ Puppetcommander (MCollective plugin) to schedule
   puppet runs
   ○ Puppet Dashboard (Reporting)
   ○ In-house External Node Classifier (aka 'Mothership')
   

   View Slide

5. Puppet - Incremental rollout
   

   View Slide

6. Puppet - Incremental rollout
   ● Deployment of changes
   ○ Commit to 'development' branch
   ○ Push changes for code review
   ○ Once changes have been reviewed and verified,
   they get merged into 'development'
   ○ A 'canary' flag can be set for a number of nodes
   ○ Change is cherry-picked to 'master' branch
   ○ A 'tag' from master is created and deployed
   ■ See behavior in the canary environment
   ■ Expand the canary env. Rinse & repeat
   ■ Remove canary flag
   

   View Slide

7. http://mothership.sourceforge.net/
   Puppet - External node Classifier
   

   View Slide

8. Puppet - External node Classifier
   ● Mothership
   ○ In-house assets management & provisioning tool. It
   also acts as an External Node Classifier for Puppet
   ■ Can provision bare-metal & virtual machines
   ■ Integrated with Cobbler
   ■ Also manages users/groups & sudoers
   ● Synchronized to LDAP
   ● Puppet deploys sudoers file
   ■ DNS management
   ■ Hosts can have one or more labels (tags) that
   are mapped to Puppet modules
   

   View Slide

9. View Slide

10. View Slide

11. Lessons learned / Best practices ...
    Puppet ...
    ● Don't just run it!
    ● Node mgmt can become difficult (+100)
    ● Keep change in mind (e.g. OS upgrades)
    Mothership ...
    ● ... is not enough. You need at least 2 views
    ○ Logical (deployed/reserved), Physical
    ● You need to live with Physical != Logical
    ○ Physical will be generated. It is a feedback loop.
    

    View Slide

12. Next steps ...
    

    View Slide

13. Requirements ...
    ● Accelerate our ability to do incremental
    deployments (multiple times a day)
    ○ ... with easy rollback
    ○ ... accelerating our ability to innovate
    ○ ... while maintaining 100% uptime
    ● The "Happy Path" needs to be without
    human intervention
    ○ From Commit to A/B test
    

    View Slide

14. Architecture ...
    Ion Cannon
    Code - Gerrit
    Build -
    Jenkins/Nexus
    QA - Selenium /
    TestComplete
    Performance
    Deployment -
    Puppet
    Analytics
    

    View Slide

15. Takeaways ...
    ● Keep your modules small, keep them simple
    ● Keep change in mind
    ○ Do incremental rollout of changes
    ○ Provide a 'default' case in your puppet modules and
    use 'fail' (sdtlib) as a default case.
    ● If you are looking for a complete CD
    platform, then you need to embed Puppet
    into a larger solution
    

    View Slide

16. Questions?
    

    View Slide