Infrastructure Testing: Grey Matter

As more organizations start to embrace the DevOps culture, there are very unique hurdles that each must overcome. One of the major hurdles we have had to overcome is the ability to verify our infrastructure prior to rolling out each of the pieces. The two major pieces of this are idempotent provisioning and the ability to monitor all necessary aspects of the environment. For this we utilize Chef and Sensu respectively, allowing us to have with pretty good certainty as to what state the systems are in and how well they are running. There will always be cases were we can not run these tools or the tools are not able to provide all of the necessary information, putting us into a grey area with the amount of confidence we have in the system.

Transcript

1. Infrastructure Testing Grey Matter Anthony Spring DevOps Days Boston - 2014
2. What is grey matter? Grey matter is the area of your infrastructure where you have the least amount of confidence.
3. Monitoring Cacti Icinga Nagios Sensu Zabbix Infrastructure Provisioning Ansible CF-Engine Chef Kickstart Powershell Puppet Salt Shell scripts
4. Confidence Trust Experience Control Knowledge http://www.skipprichard.com/
5. What about… images we did not create? machines we did not provision? machines we can not/do not monitor?
7. Why ServerSpec? Easy to use Extensible Put tests into source control Only need SSH access on remote machine
8. Integration Testing Utilize KitchenCI + ServerSpec for integration testing of Chef cookbooks and roles. Perform a basic set of specs that are applicable to all tests in a cookbook. Create highly specific tests to help increase confidence in a cookbook and cookbook dependencies.
9. Developer Environments A spec was created for all of the pieces of the developer stack Organize the specs based off of a team Two commands to evaluate the configuration of the environment
10. Box Creation and Delivery Utilize Packer and Racker to build boxes. Execute set of specs as part of build process. Lets us test kickstart, provisioning scripts, as well as Chef provisioned assets.
11. Third Party Machines Guarantee compliance with requirements — disks, mounts, cpu, packages, etc. Continuous compliance, process, and system monitoring. Reduce the feedback loop for identifying issues.
12. “Reducing grey matter is essential to increasing confidence.”
13. How can you reduce grey matter?
14. Anthony Spring Minister of DevOps Culture aspring@yieldbot.com @devopsanthony