Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ITT 2019 - Kief Morris - Building Evolvable Infrastructure

ITT 2019 - Kief Morris - Building Evolvable Infrastructure

Design strategies for building infrastructure that can be continuously evolved, by using pipelines, automated tests, and loosely integrated stacks to enable a continuous flow of changes and improvements.

People are adopting dynamic infrastructure technologies like cloud, containers, and serverless so that they can easily make changes to it. Defining infrastructure as code should make systems consistent, reliable, and easy to manage. But an infrastructure codebase can easily become a complicated, fragile mess that is scary to change. Teams can gain confidence to make frequent, rapid changes to continuously improve their infrastructure by applying appropriate design patterns and implementation practices.

990b89ca5f918a94ef6523d399eda9a4?s=128

Istanbul Tech Talks

April 02, 2019
Tweet

Transcript

  1. BUILDING EVOLVABLE INFRASTRUCTURE (as code)

  2. kief@thoughtworks.com Head of Cloud Engineering Twitter: @kief Book: http://oreil.ly/1JKIBVe Site:

    http://infrastructure-as-code.com
  3. SPEED

  4. IMPEDIMENT

  5. CLOUD

  6. QUALITY

  7. Fast Slow Careless Careful

  8. State of the DevOps Report https://devops-research.com/ Accelerate, Nicole Forsgren, PhD,

    Jez Humble, Gene Kim
  9. Graphic from State of the DevOps Report 2018 The four

    metrics
  10. "The highest performers excel at throughput and stability" State of

    the DevOps Report 2018 Nicole Forsgren, PhD, Jez Humble, Gene Kim https://devops-research.com/
  11. Optimize for the capability to deliver changes RAPIDLY and RELIABLY

  12. "Since we can't avoid change, we need to exploit it"

    Building Evolutionary Architectures Neal Ford, Rebecca Parsons, Pat Kua
  13. How do we make our infrastructure evolvable?

  14. OPTIMIZE FOR CHANGE AS CODE Define all your stuff as

    code So that everything is visible, repeatable, and changes are actionable
  15. OPTIMIZE FOR CHANGE CONTINUOUS VALIDATION Integrate and test all work

    in progress as we go So that problems are discovered and fixed up front
  16. OPTIMIZE FOR CHANGE Build small, independently releasable components So that

    they are easier and faster to change, test, and release SMALL PIECES
  17. Define all your stuff AS CODE

  18. Define infrastructure as code Infrastructure configuration is: § Reusable §

    Consistent § Visible § Versioned
  19. Configuration drift Servers start out identical But changes accumulate over

    time
  20. Synchronize continuously

  21. INFRASTRUCTURE STACK

  22. Some platforms: • AWS • Azure • Google Cloud Platform

    • VMWare • Digital Ocean • Bare metal clouds Infrastructure Platform A dynamic pool of compute, storage, and networking resources
  23. Infrastructure stack A collection of infrastructure resources provisioned and updated

    as a unit
  24. Stack management tool Stack instances Platform API Stack definition Some

    tools: • Terraform • AWS CloudFormation • Azure Resource Manager • Google Deployment Manager • Ansible Cloud Modules
  25. A stack often includes servers

  26. Server configuration tool Configuration tool Configuration definitions Server instance Some

    tools: • Ansible • Chef • Puppet • Saltstack
  27. MANAGING MULTIPLE ENVIRONMENTS

  28. MANY-HEADED STACK antipattern Test Staging Production our_env/ └── test.tf └──

    staging.tf └── production.tf
  29. Changes have a wide blast radius Test Staging Production our_env/

    └── test.tf └── staging.tf └── production.tf !
  30. SINGLETON STACK antipattern our_env/ └── test/ └── servers.tf our_env/ └──

    staging/ └── servers.tf our_env/ └── production/ └── servers.tf Test Staging Production
  31. Code changes by copy/paste our_env/ └── test/ └── servers.tf our_env/

    └── staging/ └── servers.tf our_env/ └── production/ └── servers.tf Test Staging Production !
  32. test instance staging instance production instance TEMPLATE STACK pattern stack

    source code
  33. id=test Instance parameter values id=staging id=production

  34. ! Blast radius is managed ! Environments are consistent !

    Testing is more reliable " Adds moving parts # Requires versioning and parameterization mechanisms
  35. CONTINUOUSLY VALIDATE all work in progress as you go

  36. Automatically test every change before applying it

  37. Promote changes to environments using a pipeline BUILD LOCAL APPLY

    TO QA APPLY TO PROD Sandbox QA Production APPLY TO TEST Test
  38. Processes and controls are enforced by code Every change is

    logged and traceable, from commit to production Enable governance with pipelines Environment Definitions Test Code Compliance Specifications Pipeline Definitions
  39. Challenge: Feedback cycles (This stuff is slow. Very. Very. Slow)

  40. Solution: Break stacks apart into smaller pieces

  41. BUILD SMALL, independently releasable components

  42. A stack with servers Stack provisioning includes creating and configuring

    the servers
  43. Testing the stack is slow Provision the entire stack on

    the platform, with all elements Or update an environment that we keep running all the time Repeat for every stage
  44. Break out a server role Application server role Stack Server

    configurations Java Cookbook Tomcat Cookbook
  45. Stack Java Cookbook Tomcat Cookbook Appserver Role Server configuration tool

    provisions the server separately from the stack
  46. Test server configurations Test configuration elements separately Provision and test

    using virtual machines or containers Test locally or on build agents
  47. Extract separate pipeline stages for server configuration Test stack definition

    Test server configuration Test integrated system
  48. How do we scale?

  49. Monolithic stack

  50. Monolithic stack Wide blast radius, high coordination overhead !

  51. Divide infrastructure into multiple, independently changeable stacks

  52. Each stack has its own pipeline to deliver changes

  53. Draw boundaries to minimize changes that cross stacks

  54. CONCLUSION: Optimize for change in order to achieve reliability

  55. kief@thoughtworks.com Head of Cloud Engineering Twitter: @kief Book: http://oreil.ly/1JKIBVe Site:

    http://infrastructure-as-code.com