ITT 2019 - Kief Morris - Building Evolvable Infrastructure

Design strategies for building infrastructure that can be continuously evolved, by using pipelines, automated tests, and loosely integrated stacks to enable a continuous flow of changes and improvements.

People are adopting dynamic infrastructure technologies like cloud, containers, and serverless so that they can easily make changes to it. Defining infrastructure as code should make systems consistent, reliable, and easy to manage. But an infrastructure codebase can easily become a complicated, fragile mess that is scary to change. Teams can gain confidence to make frequent, rapid changes to continuously improve their infrastructure by applying appropriate design patterns and implementation practices.

Transcript

1. BUILDING
   EVOLVABLE
   INFRASTRUCTURE
   (as code)
   

   View Slide

2. kief@thoughtworks.com
   Head of Cloud Engineering
   Twitter: @kief
   Book: http://oreil.ly/1JKIBVe
   Site: http://infrastructure-as-code.com
   

   View Slide

3. SPEED
   

   View Slide

4. IMPEDIMENT
   

   View Slide

5. CLOUD
   

   View Slide

6. QUALITY
   

   View Slide

7. Fast
   Slow Careless
   Careful
   

   View Slide

8. State of the DevOps Report
   https://devops-research.com/
   Accelerate,
   Nicole Forsgren, PhD,
   Jez Humble,
   Gene Kim
   

   View Slide

9. Graphic from State of the DevOps Report 2018
   The four
   metrics
   

   View Slide

10. "The highest
    performers excel at
    throughput and
    stability"
    State of the DevOps Report 2018
    Nicole Forsgren, PhD, Jez Humble, Gene Kim
    https://devops-research.com/
    

    View Slide

11. Optimize for the capability
    to deliver changes
    RAPIDLY and RELIABLY
    

    View Slide

12. "Since we can't avoid
    change, we need to
    exploit it"
    Building Evolutionary Architectures
    Neal Ford, Rebecca Parsons, Pat Kua
    

    View Slide

13. How do we make our
    infrastructure evolvable?
    

    View Slide

14. OPTIMIZE
    FOR
    CHANGE
    AS CODE
    Define all your stuff as code
    So that everything is visible,
    repeatable, and changes
    are actionable
    

    View Slide

15. OPTIMIZE
    FOR
    CHANGE
    CONTINUOUS
    VALIDATION
    Integrate and test all work
    in progress as we go
    So that problems are
    discovered and fixed up
    front
    

    View Slide

16. OPTIMIZE
    FOR
    CHANGE
    Build small, independently
    releasable components
    So that they are easier and faster
    to change, test, and release
    SMALL
    PIECES
    

    View Slide

17. Define all
    your stuff
    AS CODE
    

    View Slide

18. Define infrastructure as code
    Infrastructure configuration is:
    § Reusable
    § Consistent
    § Visible
    § Versioned
    

    View Slide

19. Configuration drift
    Servers start
    out identical
    But changes
    accumulate
    over time
    

    View Slide

20. Synchronize continuously
    

    View Slide

21. INFRASTRUCTURE STACK
    

    View Slide

22. Some platforms:
    • AWS
    • Azure
    • Google Cloud Platform
    • VMWare
    • Digital Ocean
    • Bare metal clouds
    Infrastructure Platform
    A dynamic pool of
    compute, storage,
    and networking
    resources
    

    View Slide

23. Infrastructure stack
    A collection of infrastructure
    resources provisioned and
    updated as a unit
    

    View Slide

24. Stack management tool
    Stack instances
    Platform API
    Stack definition
    Some tools:
    • Terraform
    • AWS CloudFormation
    • Azure Resource Manager
    • Google Deployment Manager
    • Ansible Cloud Modules
    

    View Slide

25. A stack often
    includes servers
    

    View Slide

26. Server configuration tool
    Configuration
    tool
    Configuration
    definitions
    Server instance
    Some tools:
    • Ansible
    • Chef
    • Puppet
    • Saltstack
    

    View Slide

27. MANAGING
    MULTIPLE
    ENVIRONMENTS
    

    View Slide

28. MANY-HEADED STACK antipattern
    Test Staging Production
    our_env/
    └── test.tf
    └── staging.tf
    └── production.tf
    

    View Slide

29. Changes have a wide blast radius
    Test Staging Production
    our_env/
    └── test.tf
    └── staging.tf
    └── production.tf
    !
    

    View Slide

30. SINGLETON STACK antipattern
    our_env/
    └── test/
    └── servers.tf
    our_env/
    └── staging/
    └── servers.tf
    our_env/
    └── production/
    └── servers.tf
    Test
    Staging
    Production
    

    View Slide

31. Code changes by copy/paste
    our_env/
    └── test/
    └── servers.tf
    our_env/
    └── staging/
    └── servers.tf
    our_env/
    └── production/
    └── servers.tf
    Test
    Staging
    Production
    !
    

    View Slide

32. test instance
    staging instance
    production instance
    TEMPLATE STACK pattern stack
    source
    code
    

    View Slide

33. id=test
    Instance parameter values
    id=staging
    id=production
    

    View Slide

34. ! Blast radius is managed
    ! Environments are consistent
    ! Testing is more reliable
    " Adds moving parts
    # Requires versioning and
    parameterization mechanisms
    

    View Slide

35. CONTINUOUSLY
    VALIDATE all
    work in progress
    as you go
    

    View Slide

36. Automatically test every change before
    applying it
    

    View Slide

37. Promote changes to environments using
    a pipeline
    BUILD
    LOCAL
    APPLY
    TO QA
    APPLY
    TO
    PROD
    Sandbox
    QA Production
    APPLY
    TO TEST
    Test
    

    View Slide

38. Processes and
    controls are
    enforced by code
    Every change is logged
    and traceable, from
    commit to production
    Enable
    governance with
    pipelines
    Environment
    Definitions
    Test Code
    Compliance
    Specifications
    Pipeline
    Definitions
    

    View Slide

39. Challenge: Feedback cycles
    (This stuff is slow. Very. Very.
    Slow)
    

    View Slide

40. Solution: Break stacks apart
    into smaller pieces
    

    View Slide

41. BUILD SMALL,
    independently
    releasable
    components
    

    View Slide

42. A stack with servers
    Stack provisioning
    includes creating and
    configuring the servers
    

    View Slide

43. Testing the stack is slow
    Provision the entire
    stack on the platform,
    with all elements
    Or update an
    environment that we
    keep running all the time
    Repeat for
    every stage
    

    View Slide

44. Break out a server role
    Application
    server role
    Stack
    Server
    configurations
    Java
    Cookbook
    Tomcat
    Cookbook
    

    View Slide

45. Stack
    Java
    Cookbook
    Tomcat
    Cookbook
    Appserver
    Role
    Server configuration tool
    provisions the server
    separately from the stack
    

    View Slide

46. Test server configurations
    Test configuration
    elements separately
    Provision and test using
    virtual machines or
    containers
    Test locally or on
    build agents
    

    View Slide

47. Extract separate pipeline stages
    for server configuration
    Test stack definition
    Test server configuration
    Test integrated system
    

    View Slide

48. How do we scale?
    

    View Slide

49. Monolithic stack
    

    View Slide

50. Monolithic stack
    Wide blast radius, high coordination overhead
    !
    

    View Slide

51. Divide infrastructure into multiple,
    independently changeable stacks
    

    View Slide

52. Each stack has its own pipeline to deliver
    changes
    

    View Slide

53. Draw boundaries to minimize changes
    that cross stacks
    

    View Slide

54. CONCLUSION:
    Optimize for
    change in order to
    achieve reliability
    

    View Slide

55. kief@thoughtworks.com
    Head of Cloud Engineering
    Twitter: @kief
    Book: http://oreil.ly/1JKIBVe
    Site: http://infrastructure-as-code.com
    

    View Slide