Integrate This!

This talk summarises a section of the book Infrastructure as Code by Kief Morris.

It is highly recommended to read the full book:
https://learning.oreilly.com/library/view/infrastructure-as-code/9781098150341/

In our upcoming talk 'Integrate This!', we delve into the intricacies of infrastructure dependency through exploring three core patterns: resource matching, stack data lookup, and integration registry. These patterns form the backbone of how different stacks of infrastructure can interdependent seamlessly. With a clear dissection of the pros and cons of each pattern, we aim to equip attendees with a solid understanding to make informed decisions in their infrastructure setup. A spotlight will be cast on the integration registry pattern, showcasing real-world examples of its adoption, underscoring its practicality and efficacy. Join us, as we unravel the frameworks that can significantly enhance or impede your infrastructure interdependencies, steering you towards a more integrated and robust infrastructure ecosystem.

Join https://community.cncf.io/cloud-native-ho-chi-minh/ for future talks

Transcript

1. Integrate This! You're probably doing Infrastructure as code wrong Navigating Infrastructure Dependency Patterns
2. TOC Introduction to Infra as Code 3 Core IaC Practices Relation to Software Architecture Principles Define "Infrastructure Stack" Patterns to manage Stack Dependencies Deep Dive on Integration Registry pattern
3. Introduction to Infrastructure as Code The practice of provisioning and managing infrastructure using code. Writing code that can then be distributed and applied by automated systems. Opposite: Interactive Infrastructure Management. Optimize the process for making changes to IT systems
4. Path to Cloud pre-2010 Iron Age Making changes takes time, plan in advance, change in plan is considered a failure to plan Focus: Manage risk of change Solution: processes to slow down change 2010 Shadow Age The emergence of cloud, accelerates the ability to make changes. Digital disruptors have a "Move fast and break things" mentality. Cloud is used in the shadows, shielded from heavy handed processes 2016 Age of Sprawl Forced to adopt new rate of change or go bust. Cultural movements such as Agile and DevOps take a central role within organizations 2020 Age of Sprawl (continued) COVID and the new normal force even more traditional industries to adopt Cloud and the digital Economy 2023 Age of Sustainable Growth Post sprawl - due to complexity increase, while need to grow remains More careful where to invest, but also manage existing cost
5. Age of Sprawl - Evolve fast to survive Digital Economy - Haste in adoption resulted in numerous initiatives - Multiple (often disconnected) teams building platforms https://landscape.cncf.io
6. Infrastructure Automation & Change Myths 1 Infrastructure doesn't change very often - Upgrades (security / features) - Unknowns (feedback & learning) - Performance bottlenecks (Arch re-design) 2 Build first, Automate later - Automate to deliver faster - Easier to test automations - Very hard to automate existing system 3 Must trade quality for speed - Accelerate book: you're either good at both or bad at both, no trade-off - 4 key metrics
7. 4 Key Metrics 1 Delivery lead time The elapsed time it takes to implement, test, and deliver changes to the production system 2 Deployment frequency How often changes are deployed to production systems 3 Change fail percentage What percentage of changes either cause an impaired service or need immediate correction, such as a rollback or emergency fix 4 Mean Time to Restore How long it takes to restore service when there is an unplanned outage or impairment
8. Infrastructure as Code Objective Deliver changes continuously, quickly and reliably
9. Core Practice Define Everything as Code Define all your stuff "as code" is a core practice for making changes rapidly and reliably Why: - Reusability - Consistency - Visibility
10. Core Practice Continuously test and deliver all Work in Process Effective infrastructure teams are rigorous about testing. Use automation to deploy and test each component and integrate all the work everyone has in progress. Test as they work, rather than wait until finished According to Accelerate research, teams get better results when everyone integrates their work at least daily.
11. Core Practice Build small independent pieces that can change independently Large and tightly coupled systems are harder to change and easier to break. Each piece is easy to understand and has clearly defined interfaces. Each piece can be changed, deployed and tested in isolation.
12. IaC leverages: Software Architecture and Design Principles and guidelines for designing small and independent pieces that can change independently.
13. Rules for designing components (Part 1/2) 0 1 Avoid duplication Duplication forces people to make a change in multiple places Caveat: shared code creates tight coupling 0 2 Rule of composition It should be easy to replace one side of dependency relationship without disturbing the other 0 3 Single responsibility principle Any given component should have responsibility for one thing. Ensure cohesive content per component.
14. Rules for designing components (Part 2/2) 0 4 Domain concepts, not tech concepts A component focused on a tech concept rather than a domain concept may cause undesired coupling. 0 5 Principle of least knowledge Push for clear and simple interfaces between components. No dependencies on internal implementation details 0 6 No circular dependencies Provider components create resources used by consumer components. Relationships from providers to consumers should never result in a loop.
15. Concept: Infrastructure Stack Core deployable unit of infrastructure. Stacks are usually composed of smaller components or are used as components in other stacks.
16. Challenges with Infrastructure Stacks Large monolithic stacks are fragile. Common approach is to use composition. Using smaller reusable components in stack composition still suffers from problems: - May improve readability through module composition, but overall complexity remains - Shared modules cause coupling between stacks Ideally: making a large stack more manageable by breaking it into multiple smaller stacks, each of which can be provisioned, managed, and changed independently of others. Focus on managing dependencies between stacks
17. Managing Dependencies between stacks The specific challenge with stacks is implementing the integration between them without creating tight coupling. 3 core patterns for discovering dependencies: - Resource Matching - Stack Data lookup - Integration Registry
18. Pattern Resource Matching A consumer stack uses resource matching to discover a dependency by looking for infrastructure resources that match names, tags, or other identifying characteristics. diagram source: Infrastructure as Code - Chapter 17
19. Resource matching Requires a clear understanding of which resources should be used as dependencies. Consider switching to an alternative pattern if you experience issues with breaking dependencies between teams. Consequences As soon as a consumer stack implements resource matching to discover resources from another stack, the matching pattern becomes a contract. If someone changes the naming pattern, the dependency breaks
20. Pattern Stack Data Lookup Also known as: remote statefile lookup, stack reference lookup, or stack resource lookup. Stack data lookup finds provider resources using data structures maintained by the tool that manages the provider stack diagram source: Infrastructure as Code - Chapter 17
21. Stack Data Lookup Stack management tools provide stack data lookup features to integrate projects. Most tools force to explicitly define the resource to publish for consumption. This clarifies dependencies Consequences Tends to lock into a single stack management tool. Refactoring resources between stacks requires care for consumer references
22. Pattern Integration Registry Lookup A consumer stack can use integration registry lookup to discover a resource published by a provider stack. Both stacks refer to a registry, using a known location to store and read values. diagram source: Infrastructure as Code - Chapter 17
23. Integration Registry Stack management tools provide support for storing and retrieving values from configuration registries. This decouples the tools used from the stacks they manage. Teams are free to use their preferred tool and integrate based on configuration registry and naming conventions. Consequences Registry becomes a critical service.. You may not be able to provision or recover resources when the registry is unavailable. dependency
24. Integration Pattern comparison Table Resource Matching Stack Data Lookup Integration Registry Contract ⚠Implicit ✅ Explicit ✅Explicit Tool "lock-in" ✅No ⚠Yes ✅No Intra stack refactoring considerations ⚠ Maintain match attribute values ✅ Maintain state outputs ✅ Maintain registry entries Inter stack refactoring considerations ⚠ Significant impact Requires Configuration Service ✅No ✅No ⚠Yes
25. Deep Dive: Integration Registry
26. Warning: Code Examples ahead ⚠ Actual Practical code samples of integration registry pattern follow complexity increases along the way 1. Simple Terraform configuration 2. AWS CDK Construct 3. Complex project dependency graphs
27. Example: Terraform → SSM ParameterStore Write SSM ParameterStore entry Read SSM ParameterStore Entry
28. Example: CDK → IntegrationRegistry Construct Write SSM ParameterStore entry Read SSM ParameterStore Entry
29. Advanced example: Projen → CDKTF → TF Generate Projects and define their dependencies. See https://projen.io
30. Advanced example: Projen → CDKTF → TF Generate SSM Parameter Entries
31. Advanced example: Projen → CDKTF → TF Generate SSM Parameter Lookups
32. Advanced example: Projen → CDKTF → TF Write Read
33. Projen + NX graph @aws/pdk/monorepo
34. Conclusion - Use Infrastructure as Code to deliver changes consistently, quickly and reliably - Use Software Architecture and Design principles and guidelines - Understand ways to manage Stack complexities and manage dependencies - Consider 3 Stack dependency patterns - Review practical examples of the Integration Registry pattern
35. CNCF - HCMC Chapter Join us: https://community.cncf.io/cloud-native-ho-chi-minh
36. Thank you.