Scaling Continuous Integration for Puppet

Supporting multiple teams doing parallel Puppet development

Transcript

1. Scaling Continuous Integration for Puppet Supporting multiple teams doing parallel Puppet development ​ Alan Vaghti ​ Senior Member of Technical Staff ​ avaghti@salesforce.com ​ in/avaghti ​ 
2. ​ Safe harbor statement under the Private Securities Litigation Reform Act of 1995: ​ This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such uncertainties materialize or if any of the assumptions proves incorrect, the results of salesforce.com, inc. could differ materially from the results expressed or implied by the forward-looking statements we make. All statements other than statements of historical fact could be deemed forward-looking, including any projections of product or service availability, subscriber growth, earnings, revenues, or other financial items and any statements regarding strategies or plans of management for future operations, statements of belief, any statements concerning new, planned, or upgraded services or technology developments and customer contracts or use of our services. ​ The risks and uncertainties referred to above include – but are not limited to – risks associated with developing and delivering new functionality for our service, new products and services, our new business model, our past operating losses, possible fluctuations in our operating results and rate of growth, interruptions or delays in our Web hosting, breach of our security measures, the outcome of any litigation, risks associated with completed and any possible mergers and acquisitions, the immature market in which we operate, our relatively limited operating history, our ability to expand, retain, and motivate our employees and manage our growth, new releases of our service and successful customer deployment, our limited history reselling non-salesforce.com products, and utilization and selling to larger enterprise customers. Further information on potential factors that could affect the financial results of salesforce.com, inc. is included in our annual report on Form 10-K for the most recent fiscal year and in our quarterly report on Form 10-Q for the most recent fiscal quarter. These documents and others containing important disclosures are available on the SEC Filings section of the Investor Information section of our Web site. ​ Any unreleased services or features referenced in this or other presentations, press releases or public statements are not currently available and may not be delivered on time or at all. Customers who purchase our services should make the purchase decisions based upon features that are currently available. Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements. Safe Harbor
3. §  A 15 year old cloud computing pioneer §  Data centers around the world §  Rapid growth and expansion §  Tens of thousands of servers §  Existing in-house automation tools Growth required consistency and an automated process for making reliable, repeatable changes. Salesforce
4. §  Ever growing number of servers built using automation and managed by Puppet §  Used for converting existing servers and building new ones §  Team is cross functional; consists of developers, quality, and system engineers Puppet at Salesforce
5. §  Jenkins §  Handful of machines responsible for testing, packaging, and shipping our Puppet code §  Vagrant §  Configures and manages our VirtualBox based development environment §  Rouster §  Abstraction layer for managing Vagrant virtual machines §  https://github.com/chorankates/rouster §  Git §  Version control; use GitHub Enterprise as a repository hosting service §  puppet-lint §  Make sure Puppet manifests conform to the style guide §  rspec-puppet §  Testing Puppet’s behavior when it compiles manifests into a catalog of Puppet resources Open source tools used for developing and testing Puppet code
6. ​ The team took on converting existing Kickstart & post-install scripts into the Puppet infrastructure and modules used for the Puppetization of the initial server types. -  Functional tests ensure successful/idempotent Puppet runs, resource validation, and more -  Each server type has its own functional tests -  Tests are kicked off on every commit to the integration branch and at least once a day The initial iteration of Puppet development
7. ​ Enabling other teams to contribute Puppet modules for their servers §  Teams would submit pull requests to the Puppet repo with proposed changes §  The Puppet Team would code review, test, and merge (or work on iterating with the contributor) Now more teams want to use Puppet for managing their servers Supporting external teams
8. ​ External teams were contributing Puppet code, but.. §  Teams were gated by the Puppet Team’s availability to code review & test pull requests §  This caused long feedback loops and slow iterations §  Not scalable. Could only support a handful of teams at a time We needed a new self-service contribution model to support multiple teams doing parallel Puppet development without requiring any intervention from the Puppet team. We also needed to keep the build healthy. What wasn’t working
9. ​ r10k allowed us to break up the monolithic Puppet repo. Each module was now its own Git repository. ​ A Puppetfile lives at the root of the Puppet directory. It specifies the name, location, and commit hash of the modules that will be deployed. ​ Modules now reside in their respective team’s GitHub organization. r10k
10. ​ Before After Before and after r10k
11. §  Every module is its own Git repository §  Development, code reviews, and testing of Puppet modules are all done by the contributing team §  When a change is ready for deployment, a pull request is submitted to the Puppet repo updating the modules commit hash in the Puppetfile §  Pull requests are automatically tested by an in-house tool called PAI (Puppet auto integration) §  Runs puppet-lint and rspec-puppet on modules that were changed §  Runs functional tests on all server types that are effected by the changes §  If the pull request passes, it is merged into the integration branch of Puppet §  Contributors are alerted on any test failures §  Changes to shared, core functionality (such as the external node classifier) are left open for code review from the Puppet Team New contribution model
12. Current pipeline
13. That other continuous stuff Delivery and Deployment
14. Every Puppet RPM is deployed to Puppet masters soon after creation Internal environments: continuous deployment
15. Releasing Puppet changes to production involves: §  Publishing a diff file & summary between the last release and the current release §  A thumbs up from Site Reliability §  Pressing the shiny red button & letting post deployment smoke tests run §  Canary releases: §  Utilizing Puppet’s directory environments, new releases are consumed only by a subset of representative servers (“canary servers”) §  Other servers continue to consume the previous Puppet release §  Releases are automatically consumed by non-canary servers after 24 hours §  Nagios and Graphite are used to monitor, alert, and gather metrics on Puppet health and performance Production environments: continuous delivery
16. Production environments: continuous delivery
17. Thank you