ITT 2019 - Kief Morris - Building Evolvable Infrastructure

Design strategies for building infrastructure that can be continuously evolved, by using pipelines, automated tests, and loosely integrated stacks to enable a continuous flow of changes and improvements.

People are adopting dynamic infrastructure technologies like cloud, containers, and serverless so that they can easily make changes to it. Defining infrastructure as code should make systems consistent, reliable, and easy to manage. But an infrastructure codebase can easily become a complicated, fragile mess that is scary to change. Teams can gain confidence to make frequent, rapid changes to continuously improve their infrastructure by applying appropriate design patterns and implementation practices.

Transcript

1. BUILDING EVOLVABLE INFRASTRUCTURE (as code)
2. kief@thoughtworks.com Head of Cloud Engineering Twitter: @kief Book: http://oreil.ly/1JKIBVe Site: http://infrastructure-as-code.com
3. SPEED
4. IMPEDIMENT
5. CLOUD
6. QUALITY
7. Fast Slow Careless Careful
8. State of the DevOps Report https://devops-research.com/ Accelerate, Nicole Forsgren, PhD, Jez Humble, Gene Kim
9. Graphic from State of the DevOps Report 2018 The four metrics
10. "The highest performers excel at throughput and stability" State of the DevOps Report 2018 Nicole Forsgren, PhD, Jez Humble, Gene Kim https://devops-research.com/
11. Optimize for the capability to deliver changes RAPIDLY and RELIABLY
12. "Since we can't avoid change, we need to exploit it" Building Evolutionary Architectures Neal Ford, Rebecca Parsons, Pat Kua
13. How do we make our infrastructure evolvable?
14. OPTIMIZE FOR CHANGE AS CODE Define all your stuff as code So that everything is visible, repeatable, and changes are actionable
15. OPTIMIZE FOR CHANGE CONTINUOUS VALIDATION Integrate and test all work in progress as we go So that problems are discovered and fixed up front
16. OPTIMIZE FOR CHANGE Build small, independently releasable components So that they are easier and faster to change, test, and release SMALL PIECES
17. Define all your stuff AS CODE
18. Define infrastructure as code Infrastructure configuration is: § Reusable § Consistent § Visible § Versioned
19. Configuration drift Servers start out identical But changes accumulate over time
20. Synchronize continuously
21. INFRASTRUCTURE STACK
22. Some platforms: • AWS • Azure • Google Cloud Platform • VMWare • Digital Ocean • Bare metal clouds Infrastructure Platform A dynamic pool of compute, storage, and networking resources
23. Infrastructure stack A collection of infrastructure resources provisioned and updated as a unit
24. Stack management tool Stack instances Platform API Stack definition Some tools: • Terraform • AWS CloudFormation • Azure Resource Manager • Google Deployment Manager • Ansible Cloud Modules
25. A stack often includes servers
26. Server configuration tool Configuration tool Configuration definitions Server instance Some tools: • Ansible • Chef • Puppet • Saltstack
27. MANAGING MULTIPLE ENVIRONMENTS
28. MANY-HEADED STACK antipattern Test Staging Production our_env/ └── test.tf └── staging.tf └── production.tf
29. Changes have a wide blast radius Test Staging Production our_env/ └── test.tf └── staging.tf └── production.tf !
30. SINGLETON STACK antipattern our_env/ └── test/ └── servers.tf our_env/ └── staging/ └── servers.tf our_env/ └── production/ └── servers.tf Test Staging Production
31. Code changes by copy/paste our_env/ └── test/ └── servers.tf our_env/ └── staging/ └── servers.tf our_env/ └── production/ └── servers.tf Test Staging Production !
32. test instance staging instance production instance TEMPLATE STACK pattern stack source code
33. id=test Instance parameter values id=staging id=production
34. ! Blast radius is managed ! Environments are consistent ! Testing is more reliable " Adds moving parts # Requires versioning and parameterization mechanisms
35. CONTINUOUSLY VALIDATE all work in progress as you go
36. Automatically test every change before applying it
37. Promote changes to environments using a pipeline BUILD LOCAL APPLY TO QA APPLY TO PROD Sandbox QA Production APPLY TO TEST Test
38. Processes and controls are enforced by code Every change is logged and traceable, from commit to production Enable governance with pipelines Environment Definitions Test Code Compliance Specifications Pipeline Definitions
39. Challenge: Feedback cycles (This stuff is slow. Very. Very. Slow)
40. Solution: Break stacks apart into smaller pieces
41. BUILD SMALL, independently releasable components
42. A stack with servers Stack provisioning includes creating and configuring the servers
43. Testing the stack is slow Provision the entire stack on the platform, with all elements Or update an environment that we keep running all the time Repeat for every stage
44. Break out a server role Application server role Stack Server configurations Java Cookbook Tomcat Cookbook
45. Stack Java Cookbook Tomcat Cookbook Appserver Role Server configuration tool provisions the server separately from the stack
46. Test server configurations Test configuration elements separately Provision and test using virtual machines or containers Test locally or on build agents
47. Extract separate pipeline stages for server configuration Test stack definition Test server configuration Test integrated system
48. How do we scale?
49. Monolithic stack
50. Monolithic stack Wide blast radius, high coordination overhead !
51. Divide infrastructure into multiple, independently changeable stacks
52. Each stack has its own pipeline to deliver changes
53. Draw boundaries to minimize changes that cross stacks
54. CONCLUSION: Optimize for change in order to achieve reliability
55. kief@thoughtworks.com Head of Cloud Engineering Twitter: @kief Book: http://oreil.ly/1JKIBVe Site: http://infrastructure-as-code.com