Gonzo - Increasing Agility by Understanding Risk

Gonzo lets you assess the impact of your Puppet change before rolling it out across your server estate. It takes over after continuous integration tools have run their tests and deployed the release to the Puppet Masters, but before clients have been updated.

Gonzo's goal is to increase confidence in Puppet changes by making it easier to verify that all changes are intentional and understood.

github.com/croomes/gonzo

Transcript

1. Increasing Agility by Understanding Risk Simon Croome simon@croome.org
2. $ whoami Simon Croome Infrastructure Engineer Financial Industry Independent Consultant
3. Why Puppet?
4. Velocity
5. Configuration Management is only part of the solution
7. Software is deployable throughout its lifecycle
8. Software is deployable throughout its lifecycle ! Priority is keeping the software deployable over new features
9. Software is deployable throughout its lifecycle ! Priority is keeping the software deployable over new features ! Anybody can get fast, automated feedback on the production readiness of their systems any time a change is made
10. Software is deployable throughout its lifecycle ! Priority is keeping the software deployable over new features ! Anybody can get fast, automated feedback on the production readiness of their systems any time a change is made ! Push-button deployments of any version to any environment
11. Continuous Delivery
12. Continuous Delivery Reduces the transaction cost of making change
13. Continuous Delivery Reduces the transaction cost of making change Faster ROI
14. Continuous Delivery Reduces the transaction cost of making change Faster ROI Reduces Risk
15. Risk
16. Concept: John Allspaw http://blog.vincentbrouillet.com/costs-and-risks-benefits-of- continuous-delivery-in-one-picture/ !
17. If it hurts Do it more often
18. Controls
19. Controls Change Management
20. Controls Change Management Separation of Duties
21. Controls Change Management Separation of Duties Audit
22. An example workflow
23. Jira Ticket Engineering Puppet Data Code Branch Puppet Modules Test Servers Dev Puppet Engineering UNIX team member is assigned ticket. Jira creates development sandbox using Stash integration. Peer Review Code Format Tests Integr’tn Tests Unit Tests Code Release Once code is merged into Stash’s Release Branch, a UNIX team member may create a release containing one or more changes. Bamboo automates deployment of the release to UAT, further automated testing, then release to Production. Note that it will not yet be deployed to client systems. Create Release Deploy to UAT Integration Tests Deploy to PROD New Requirement Ticket is raised against UNIX Jira Issue tracker, e.g. New project, change to OS build. Test failures block the release. Initiate Deployment Release is selected by UNIX team member. Select Release Dry-run Change Detail Impacted Roles Build Status Non- Impacted Servers Impacted Servers Manual Review Impact Analysis Change is run across the environment in read-only mode. Servers that would be changed report back changes. UNIX Change Manager assesses impact and assigns risk. Scheduling & Change Management Each impacted server role enters separate change management workflow Maintenance windows and change risk feed into scheduling. DEV SIT DIT EBF PERF EBF Impacted Roles BCP PROD CAT Deployment Completed approvals trigger deployment during change window. . Approved CHG Tickets Apply Change Review Request Risk Maintenance Windows ServiceNow CHG Tickets Reporting Web console to track deployment of changes across the environment, including summary view of health by server role and business area. Un-approved tickets reset change workflow. Code Review UNIX team member creates “Pull Request”, indicating change is ready to be reviewed. Bamboo automates code quality review, unit and integration testing. If all tests pass, Engineering team alerted to perform peer review in Stash.
24. Jira Ticket Engineering Puppet Data Code Branch Puppet Modules Test Servers Dev Puppet Engineering UNIX team member is assigned ticket. Jira creates development sandbox using Stash integration. New Requirement Ticket is raised against UNIX Jira Issue tracker, e.g. New project, change to OS build. Test fail
25. Jira Ticket Engineering Puppet Data Code Branch Puppet Modules Test Servers Dev Puppet Engineering UNIX team member is assigned ticket. Jira creates development sandbox using Stash integration. New Requirement Ticket is raised against UNIX Jira Issue tracker, e.g. New project, change to OS build. Test failures block the release. Review Request Code Review UNIX team member change is ready to b Bamboo automates integration testing. If all tests pass, Eng peer review in Stash
26. Puppet Data Puppet Modules et. sing Stash Peer Review Code Format Tests Integr’tn Tests Unit Tests Code Release Once code is merged into member may create a rel Bamboo automates deplo automated testing, then r Note that it will not yet be Create Release Deploy UAT Test failures block the release. Review Request Code Review UNIX team member creates “Pull Request”, indicating change is ready to be reviewed. Bamboo automates code quality review, unit and integration testing. If all tests pass, Engineering team alerted to perform peer review in Stash.
27. Peer Review Code Release Once code is merged into Stash’s Release Branch, a UNIX team member may create a release containing one or more changes. Bamboo automates deployment of the release to UAT, further automated testing, then release to Production. Note that it will not yet be deployed to client systems. Create Release Deploy to UAT Integration Tests Deploy to PROD st”, indicating unit and to perform
28. Initiate Deployment Release is selected by UNIX team member. Select Release Dry-run Change Detail Impacted Roles Build Status Non- Impacted Servers Impacted Servers Manual Review Impact Analysis Change is run across the environment in read-only mode. Servers that would be changed report back changes. UNIX Change Manager assesses impact and assigns risk.
29. nitiate Deployment Release is selected by UNIX team member. Select Release Dry-run Change Detail Impacted Roles Build Status Non- Impacted Servers Impacted Servers Manual Review Impact Analysis Change is run across the environment in read-only mode. Servers that would be changed report back changes. UNIX Change Manager assesses impact and assigns risk. Scheduling & Ch Each impacted serv workflow Maintenance window DE Impacted Roles Reporting
30. nge ail ld tus Manual Review read-only mode. ack changes. t and assigns risk. Scheduling & Change Management Each impacted server role enters separate change management workflow Maintenance windows and change risk feed into scheduling. DEV SIT DIT EBF PERF EBF Impacted Roles BCP PROD CAT Deployment Completed app window. . Approved CHG Ticke Risk Maintenance Windows ServiceNow CHG Tickets Reporting Un-approved tickets r change workflow.
31. Management nters separate change management hange risk feed into scheduling. T EBF PERF EBF BCP PROD Deployment Completed approvals trigger deployment during change window. . Approved CHG Tickets Apply Change Risk Maintenance Windows ServiceNow CHG Tickets Un-approved tickets reset change workflow.
32. DEV SIT DIT EBF PERF EBF Impacted Roles BCP PROD CAT Approved CHG Tickets Risk Maintenance Windows ServiceNow CHG Tickets Reporting Web console to track deployment of changes across the environment, including summary view of health by server role and business area. Un-approved tickets reset change workflow.
33. Jira Ticket Engineering Puppet Data Code Branch Puppet Modules Test Servers Dev Puppet Engineering UNIX team member is assigned ticket. Jira creates development sandbox using Stash integration. Peer Review Code Format Tests Integr’tn Tests Unit Tests Code Release Once code is merged into Stash’s Release Branch, a UNIX team member may create a release containing one or more changes. Bamboo automates deployment of the release to UAT, further automated testing, then release to Production. Note that it will not yet be deployed to client systems. Create Release Deploy to UAT Integration Tests Deploy to PROD New Requirement Ticket is raised against UNIX Jira Issue tracker, e.g. New project, change to OS build. Test failures block the release. Initiate Deployment Release is selected by UNIX team member. Select Release Dry-run Change Detail Impacted Roles Build Status Non- Impacted Servers Impacted Servers Manual Review Impact Analysis Change is run across the environment in read-only mode. Servers that would be changed report back changes. UNIX Change Manager assesses impact and assigns risk. Scheduling & Change Management Each impacted server role enters separate change management workflow Maintenance windows and change risk feed into scheduling. DEV SIT DIT EBF PERF EBF Impacted Roles BCP PROD CAT Deployment Completed approvals trigger deployment during change window. . Approved CHG Tickets Apply Change Review Request Risk Maintenance Windows ServiceNow CHG Tickets Reporting Web console to track deployment of changes across the environment, including summary view of health by server role and business area. Un-approved tickets reset change workflow. Code Review UNIX team member creates “Pull Request”, indicating change is ready to be reviewed. Bamboo automates code quality review, unit and integration testing. If all tests pass, Engineering team alerted to perform peer review in Stash.
36. Technology Stack ! Puppet - Dynamic environments MCollective - Comms CouchDB - Reports PouchDB - Replication to browser AngularJS - Web interface Rails - API (to be replaced) Resque / Redis - Job scheduling !
37. github.com/croomes/gonzo ! Questions?