Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Environment as Code: Moving away from Imperative Pipelines

Adarsh Shah
February 06, 2023

Environment as Code: Moving away from Imperative Pipelines

Cloud Native tools like Infrastructure as Code (IaC) & Helm have made managing infrastructure as well as cloud-native apps easier in a lot of ways, but there are many challenges that companies accept as the cost of adopting them. IaC is good at provisioning individual infrastructure components (a few resources together), and tools like helm are good at deploying applications, but engineering teams want an entire environment with various components like networking, platform (ec2/eks), database, k8s apps, etc. to run their business applications.

To provision and teardown an entire environment, these teams have two options. They can either hand-roll imperative pipelines to manage individual components and then manage complex dependencies between these components or create a monolith deployment for the entire environment. These approaches are inefficient and slow down feature development and innovation. They also make replicating, visualizing & understanding environments difficult. What if there were a better way?

This talk digs into these challenges and then looks at how to resolve them. We will introduce Environment as Code (EaC) which is an abstraction over cloud-native tools that provides a declarative way of defining an entire environment. It has a Control Plane that manages the state of the environment, including component dependencies, and detects drift, and enables reconciliation.

Key Takeaways:

* Challenges adopting Cloud Native tools like Infrastructure as Code, Helm, etc.
* What is Environment as Code?
* How can Environment as Code help resolve those challenges?

Adarsh Shah

February 06, 2023
Tweet

More Decks by Adarsh Shah

Other Decks in Technology

Transcript

  1. Environment as Code: Moving away
    from Imperative Pipelines
    Adarsh Shah
    Engineering Leader, Coach, Public Speaker
    Founder & CEO at CloudKnit
    @shahadarsh 

    https://cloudknit.io
    @shahadarsh
    https://bit.ly/eac-ghent
    Open Source Repo:

    https://github.com/cloudknit-io/cloudknit

    View Slide

  2. @shahadarsh
    https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit
    Some of the Current Tools
    AWS SDK

    View Slide

  3. @shahadarsh
    https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit
    CNCF Landscape

    View Slide

  4. @shahadarsh
    https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit
    CNCF Landscape

    View Slide

  5. @shahadarsh
    https://github.com/cloudknit-io/cloudknit
    https://bit.ly/eac-ghent

    View Slide

  6. @shahadarsh
    https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit
    Evolution of Environment Setup
    networking
    +
    eks
    +
    ….
    Monolith Deployment
    networking
    eks
    website
    postgres
    networking
    networking
    +
    eks
    +
    …. ec2 website
    eks postgres
    ec2
    Breaking into Separate & Smaller components
    +
    +

    View Slide

  7. @shahadarsh
    https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit
    Evolution of Environment Execution
    From Engineers
    machine
    Deployment Pipeline/GitOps execution
    from a Shared Environment

    View Slide

  8. @shahadarsh
    https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit
    • Option 1
    • Create a Monolith IaC
    • Option 2
    • Hand-roll Pipelines - Imperative
    • Manage Complex Dependencies
    Options for Entire Environment Provisioning

    View Slide

  9. @shahadarsh
    https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit
    Other Challenges scaling IaC
    Replicating Environments
    is a pain Not easy to visualize/understand
    Environments
    Drift Detection Not straightforward to 

    promote changes
    Dev QA Prod

    View Slide

  10. @shahadarsh
    https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit
    Terminologies
    Components: A logical grouping of 1 or more Infrastructure Resources or
    Applications that get provisioned together. For example, Networking is an
    Infrastructure Component with various Infrastructure resources like Virtual Private
    Cloud(VPC), Subnets, Internet Gateways, Route Tables, etc.
    Environment: A logical grouping of all the Components that are needed to run
    business applications. The grouping includes components like networking, eks,
    database, k8s apps, etc.

    View Slide

  11. @shahadarsh
    https://github.com/cloudknit-io/cloudknit
    https://bit.ly/eac-ghent
    What is Environment as
    Code?

    View Slide

  12. @shahadarsh
    https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit
    Environment as Code
    networking
    eks
    website
    postgres
    networking
    website
    eks postgres
    Declarative
    Environment as Code
    +

    View Slide

  13. @shahadarsh
    https://github.com/cloudknit-io/cloudknit
    https://bit.ly/eac-ghent
    Environment as Code

    View Slide

  14. @shahadarsh
    https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit
    IaC vs EaC
    Infrastructure as Code
    Automates various Lego Pieces 

    (i.e. Infrastructure Resources)
    Environment as Code
    Automates how those Lego Pieces 

    are connected to make up a Lego Toy 

    (i.e. Entire Environment)

    View Slide

  15. @shahadarsh
    https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit
    Environment as Code
    Environment as Code (EaC) is an abstraction over Cloud
    native tools that provides a declarative way of de
    fi
    ning an
    entire Environment. It has a Control Plane that manages the
    state of the environment, including relationships between
    various resources, Detects Drift as well enables Reconciliation.

    View Slide

  16. @shahadarsh
    https://github.com/cloudknit-io/cloudknit
    https://bit.ly/eac-ghent
    Imperative to Declarative
    HOW
    Pipelines EaC
    WHAT

    View Slide

  17. @shahadarsh
    https://github.com/cloudknit-io/cloudknit
    https://bit.ly/eac-ghent
    Provision Environment
    Environment as Code
    Environment
    State
    (Manages dependencies,
    status etc. )
    State
    networking
    eks
    State
    website
    postgres
    State
    networking
    website
    eks postgres
    Provision
    Reconcile
    Declarative
    Detect Drift
    Control Plane
    +

    View Slide

  18. @shahadarsh
    https://github.com/cloudknit-io/cloudknit
    https://bit.ly/eac-ghent
    networking
    website
    eks postgres
    Teardown Environment
    Environment as Code
    Environment
    State
    (Manages dependencies,
    status etc. )
    State
    networking
    State
    website
    eks postgres
    State
    Teardown
    Reconcile
    Declarative
    Detect Drift
    Control Plane
    +

    View Slide

  19. @shahadarsh
    https://github.com/cloudknit-io/cloudknit
    https://bit.ly/eac-ghent
    Attributes of Environment as
    Code

    View Slide

  20. @shahadarsh
    https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit
    Ability to de
    fi
    ne Entire Environment

    View Slide

  21. @shahadarsh
    https://github.com/cloudknit-io/cloudknit
    https://bit.ly/eac-ghent
    Environment as Code
    networking
    platform-k8s
    k8s-addons
    postgres
    networking
    eks
    website
    postgres
    Declarative
    Loosely Coupled

    View Slide

  22. @shahadarsh
    https://github.com/cloudknit-io/cloudknit
    https://bit.ly/eac-ghent
    Manage State for the entire Environment
    Example Environment State File

    View Slide

  23. @shahadarsh
    https://github.com/cloudknit-io/cloudknit
    https://bit.ly/eac-ghent
    Idempotent and Immutable for
    entire Environment

    View Slide

  24. @shahadarsh
    https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit
    Non-Idempotent
    Provision
    VM VM VM
    { }
    IaC
    3 VMs
    End State

    Expected = 3
    Actual = 6
    Idempotent
    End State

    Expected = 3
    Actual = 3
    Reapply
    VM VM VM
    Provision
    VM VM VM
    Reapply
    No change
    { }
    IaC
    3 VMs

    View Slide

  25. @shahadarsh
    https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit
    Provision
    v1 v1 v1
    User
    Mutable Infrastructure
    { }
    IaC
    Apply
    changes
    v2
    v2 v2
    Change
    { }
    IaC
    v1 v1 v1
    Provision
    User
    Provision
    v2 v2 v2
    User
    Immutable Infrastructure
    { }
    IaC
    Change
    { }
    IaC
    Deploys v2 to same Infrastructure Provisions new Infrastructure with v2

    View Slide

  26. @shahadarsh
    https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit
    Visualize and Understand Environments

    View Slide

  27. @shahadarsh
    https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit
    Drift Detection and Reconciliation
    Reconcile
    (Preferably with Approval step
    that shows the plan)
    Desired State Current State
    Control Loop
    Observe/Detect Drift

    View Slide

  28. @shahadarsh
    https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit
    Compare and Promote Changes between
    Environments
    Compare & promote changes
    across environments
    GitOps for Entire Environment

    View Slide

  29. @shahadarsh
    https://github.com/cloudknit-io/cloudknit
    https://bit.ly/eac-ghent
    Create Pull Request,
    EaC Validate
    Validate
    Approve & 

    Merge PR
    Control
    Plane
    Reconcile
    GitOps for Environment
    EaC
    Trunk
    networking
    platform-k8s
    k8s-addons
    postgres
    Observe
    Branch
    networking
    eks
    website
    postgres

    View Slide

  30. CloudKnit
    Open Source Repo: https://github.com/cloudknit-io/cloudknit

    View Slide

  31. Thank You
    @shahadarsh
    https://bit.ly/eac-ghent
    Adarsh Shah
    Engineering Leader, Coach, Public Speaker
    Founder & CEO at CloudKnit
    @shahadarsh 

    https://cloudknit.io
    Open Source Repo:

    https://github.com/cloudknit-io/cloudknit

    View Slide