Infrastructure as Code (O'Reilly Live Training)

Originally presented at O'Reilly Live Training, 2021.

Whether you’re an infrastructure, software, security, test, or release engineer, you might find yourself building or using cloud or data center infrastructure. How can you update your system without disrupting your teammates, organization, and business-critical applications? You could use a command line or dashboard to manually add configuration to a resource, but the configuration might include insecure or conflicting parameters. At best, your infrastructure then has a variety of uniquely configured resources managed by multiple teammates. At worst, you accidentally bring down the system or allow a bad actor to exploit it. Over time, your system becomes difficult to reproduce, maintain, and operate. Enter infrastructure as code—an approach to automating your infrastructure. As more companies adopt public cloud and attempt DevOps, infrastructure as code becomes a necessary practice for team collaboration and system resiliency.

Transcript

1. Infrastructure as Code Principles & Practices for Building & Automating Infrastructure O’Reilly Live Training | Rosemary Wang
2. I spent four hours copying server names from a spreadsheet. @joatmon08 | 2
3. 😠 me on a weekend @joatmon08 | 3
4. I need to learn how to code. @joatmon08 | 4
5. 😭 me on a pairing with a senior developer @joatmon08 | 5
6. 🌱 Hi, I’m Rosemary • Developer Advocate at HashiCorp • Writes about Infrastructure as Code • Infrastructure Engineer & Consultant • Application & Security Dabbler Find me: @joatmon08 @joatmon08 | 6
7. Poll: What best describes your role? • Security • Infrastructure / Platform • Release • Test • Software • Other @joatmon08 | 7
8. By the end of this training, you’ll be able to.... • Define & express infrastructure as code • Write tests for infrastructure as code • Build a delivery pipeline for infrastructure as code • Evaluate the practices and patterns to apply in an organization @joatmon08 | 8
9. What is infrastructure as code? @joatmon08 | 9
10. Infrastructure Software, platform, or hardware that delivers or deploys applications to production. @joatmon08 | 10
11. @joatmon08 | 11 Write down steps for your infrastructure changes.
12. @joatmon08 | 12 Write down steps for your infrastructure changes. (Maybe) Test changes in development environment.
13. @joatmon08 | 13 Write down steps for your infrastructure changes. Wait for change approval. (Maybe) Test changes in development environment.
14. @joatmon08 | 14 Write down steps for your infrastructure changes. Wait for change approval. Apply the change to production system. (Maybe) Test changes in development environment.
15. 😌 best case: no one else is making changes and you didn’t break anything @joatmon08 | 15
16. 💥 worst case: your change conflicts and something broke everything @joatmon08 | 16
17. Infrastructure as Code (IaC) The process of automating infrastructure changes in a codified manner. It applies practices such as version control and continuous delivery. @joatmon08 | 17
18. @joatmon08 | 18 Write infrastructure changes in a codified manner.
19. @joatmon08 | 19 Write infrastructure changes in a codified manner. Commit changes to version control.
20. @joatmon08 | 20 Write infrastructure changes in a codified manner. Commit changes to version control. Test changes in development environment.
21. @joatmon08 | 21 Write infrastructure changes in a codified manner. Wait for code review. Commit changes to version control. Test changes in development environment.
22. @joatmon08 | 22 Write infrastructure changes in a codified manner. Wait for code review. Deploy the change to production system. Commit changes to version control. Test changes in development environment.
23. Use Cases • Provisioning • Configuration Management • Image Building @joatmon08 | 23
24. Which of the following is not infrastructure as code? A. Shell script to install a binary B. Documentation outlining steps in a UI C. Python code to add a VLAN to a network switch D. AWS CloudFormation stack @joatmon08 | 24
25. Which of the following is not infrastructure as code? A. Shell script to install a binary B. Documentation outlining steps in a UI C. Python code to add a VLAN to a network switch D. AWS CloudFormation stack E. Potentially all of the above @joatmon08 | 25
26. Principles (RICE) • Reproducible • Idempotent • Composable • Evolvable @joatmon08 | 26
27. How do I write infrastructure as code? @joatmon08 | 27
28. Imperative or Declarative? Imperative • How should you make the change? • Use Cases: ◦ Configuration Management Declarative • What change are you making? • Use Cases: ◦ Provisioning ◦ Image Building @joatmon08 | 28
29. Mutability or Immutability? Mutability • Change resource in-place • Less time to change Immutability • Change new resource, remove the old one • Less downtime due to (potential) failure @joatmon08 | 29
30. Code or Configuration? Code • Programming language • Declarative / imperative • Use client SDK for infrastructure • Less opinionated, more flexibility Domain-Specific Languages (DSLs) • Usually part of a tool • Declarative • Often considered configuration • More opinionated, less flexibility @joatmon08 | 30
31. Exercise: Create a network and a server on the network. Tools: HashiCorp Terraform, Google Cloud Platform github.com/joatmon08/olt-infrastructure-as-code @joatmon08 | 31
32. “Clean” Infrastructure as Code • Version control • Linting and formatting • Naming • Variables and constants • Parametrize dependencies • Keeping secrets @joatmon08 | 32
33. ❔ Q&A @joatmon08 | 33
34. How do I share infrastructure as code? @joatmon08 | 34
35. Modules and Dependencies Modules • Subsets of infrastructure resources • Grouped by business domain, function, or operational scope Dependencies • Relationship between a set of infrastructure resources • High-level resource depends on a low-level one. @joatmon08 | 35
36. Patterns for Modules • Singleton : all configuration in one place • Composite: build resources on each other • Factory : commonly created resources • Prototype : set of defaults for other modules • Builder : select resources to build @joatmon08 | 36
37. Principles for Dependencies @joatmon08 | 37 Inversion of Control High-level resource calls low-level resource Dependency Inversion Use a layer of abstraction to pass attributes Dependency Injection
38. Exercise: Refactor a configuration with patterns and principles Tools: HashiCorp Terraform, Google Cloud Platform github.com/joatmon08/olt-infrastructure-as-code @joatmon08 | 38
39. ❔ Q&A @joatmon08 | 39
40. How do I test infrastructure as code? @joatmon08 | 40
41. @joatmon08 | 41 end-to-end integration contract unit Testing Strategy Cost in time, effort, and resources increase as you go up the pyramid.
42. @joatmon08 | 42 end-to-end integration contract unit Testing Strategy Statically analyzes infrastructure configuration to verify values defined match expected changes.
43. @joatmon08 | 43 end-to-end integration contract unit Testing Strategy Validate the output of a low-level dependency matches input to high-level resource.
44. @joatmon08 | 44 end-to-end integration contract unit Testing Strategy Run against one or more resources to verify changes apply successfully.
45. @joatmon08 | 45 end-to-end integration contract unit Testing Strategy Run against entire system to check if it supports an expected workflow.
46. @joatmon08 | 46 end-to-end integration contract unit Testing Strategy manual Run unique tests, especially for devices that cannot be automated.
47. Exercise: Write a unit, contract, integration, and end-to-end tests Tools: HashiCorp Terraform, Google Cloud Platform, pytest github.com/joatmon08/olt-infrastructure-as-code @joatmon08 | 47
48. ❔ Q&A @joatmon08 | 48
49. ☕ Break @joatmon08 | 49
50. Where does it fit into my infrastructure development lifecycle? @joatmon08 | 50
51. Delivery Pipeline Expresses a workflow with a set of stages to build, test, deploy, and release infrastructure as code. Its stages can facilitate continuous delivery or deployment. @joatmon08 | 51
52. Exercise: Create a delivery pipeline to deploy to production Tools: GitHub Actions github.com/joatmon08/olt-infrastructure-as-code @joatmon08 | 52
53. Branching Models Feature-based New change? New branch. @joatmon08 | 53 Trunk-based New change? Push to main branch.
54. Put the steps in order for feature-based development. @joatmon08 | 54 Create a branch. Commit changes to branch. Rebase changes from main to branch. Merge changes from branch to main.
55. Trunk-based development @joatmon08 | 55 Commit changes to local main branch. Rebase changes from remote main to local main branch. Push changes to remote main branch.
56. Where should tests go in this feature-based delivery pipeline? @joatmon08 | 56 main branch create feature branch feature branch commit changes open pull request deploy to testing* merge pull request deploy to production unit tests integration tests integration tests end-to-end tests * Tests for each pull request may run in a separate environment or a shared testing environment.
57. Where should tests go in this trunk-based delivery pipeline? @joatmon08 | 57 main branch commit & push changes deploy to testing deploy to production integration tests unit tests integration tests end-to-end tests end-to-end tests
58. ❔ Q&A @joatmon08 | 58
59. How do I enforce its cost, compliance, and security? @joatmon08 | 59
60. Infrastructure as code can be... Secure • Least privilege • Protect sensitive information • Encrypt in-transit and at-rest • Access logging • Network access Industry best practices: https://ncp.nist.gov/repository @joatmon08 | 60 Compliant • Naming • Tags • Access control • Logging Organization (and regulatory) standards.
61. Exercise: Write a test to check infrastructure as code compliance. Tools: HashiCorp Terraform, Google Cloud Platform, pytest github.com/joatmon08/olt-infrastructure-as-code @joatmon08 | 61
62. Which is not a valid technique to help you reduce your infrastructure cost? A. Tag with an infrastructure expiration date B. Build a cost estimation test C. Remove infrastructure across regions D. Use smaller resource types E. Renegotiate with your infrastructure provider @joatmon08 | 62
63. Which is not a valid technique to help you reduce your infrastructure cost? A. Tag with an infrastructure expiration date B. Build a cost estimation test C. Remove infrastructure across regions D. Use smaller resource types E. Renegotiate with your infrastructure provider @joatmon08 | 63
64. ❔ Q&A @joatmon08 | 64
65. How do I rollback failed changes? @joatmon08 | 65
66. True or false? Infrastructure as code does not support rollback. A. True B. False @joatmon08 | 66
67. True or false? Infrastructure as code does not support rollback. A. True B. False C. Eh, it’s more like roll forward... @joatmon08 | 67
68. Failed Changes You want to minimize the blast radius of a failed change. This makes it easier to roll forward and localize fixes. @joatmon08 | 68
69. Rolling “Forward” @joatmon08 | 69 New machine image fails application. Find last working machine image. Update and commit last machine image. Unit test. Deploy to testing. Integration & end-to-end test. Deploy to production.
70. Canary Deployment @joatmon08 | 70 Copy production server configuration. Paste and rename to canary server. Set new image for canary server. Deploy canary server. Send 10% of requests to canary server. Debug errors and problems. Fix problem, roll out to production servers. Delete canary server. * Test or debug changes
71. Blue-Green Deployment @joatmon08 | 71 Copy blue network. Paste and rename to green network. Test green network. Deploy servers to green network. Direct traffic to green servers. Wait for regression test. Delete blue network. * Implement changes with large blast radius.
72. Feature Flags (AKA Feature Toggles) @joatmon08 | 72 Set variable to enable_new_network to false. Ready to create a new network. Set variable to enable_new_network to true. * Stage or hide changes during trunk-based development. Create a new network.
73. ❔ Q&A @joatmon08 | 73
74. Conclusion @joatmon08 | 74
75. Recap • Define & express infrastructure as code • Write tests for infrastructure as code • Build a delivery pipeline for infrastructure as code • Evaluate the practices and patterns to apply in an organization @joatmon08 | 75
76. github.com/joatmon08/olt -infrastructure-as-code @joatmon08 | 76
77. Rosemary Wang Additional Resources: gist.github.com/joatmon08/61aac109603af7a7d02e7712121f9b0e Other talks: joatmon08.github.io Find me: @joatmon08 @joatmon08 | 77