plan & apply. rinse & repeat.

Managing infrastructure and services using Terraform and Buildkite CI at Code Climate.

Code Climate addresses common challenges across engineering organizations with the adoption of tools and practices that bring automation and high visibility to changes in our infrastructure and services.

Transcript

1. plan & apply. rinse & repeat. Managing infrastructure and services using Terraform and Buildkite CI 1 Devon Blandin / Code Climate / USV DevOps Summit / July 10th, 2018
2. desired state: what are our goals? 1. Easily propose, review, and apply changes to infrastructure 2. Allow for automated releases to production 3. Infrastructure and services should be easily placed into & out of maintenance mode 4. Operators should have timely, helpful notifications of deploy activity 5. Operators should have controls to roll back changes 2
3. Easily propose, review, and apply changes to infrastructure goal #1 3
4. Easily propose, review, and apply changes to infrastructure 4
5. Easily propose, review, and apply changes to infrastructure 5
6. Easily propose, review, and apply changes to infrastructure 6
7. If you're using Terraform, every change you make is highly visible, remains in version control, and is discoverable within the git repository at any time. 7
8. Allow for automated releases to production goal #2 8
9. Allow for automated releases to production - We let CI deploy updates to services via terraform. - Operators execute any other changesets, also via terraform. 9
10. Allow for automated releases to production 10 - Use remote state to store your configuration - Use remote state locking to prevent collisions
11. Allow for automated releases to production 11
12. Allow for automated releases to production build all the artifacts test the artifacts report test coverage + trigger next pipeline all branches 12
13. Allow for automated releases to production + build rails assets + sync rails assets to S3 master 13
14. Allow for automated releases to production /velocity_prod/app_image_ref /velocity_prod/consumer_image_ref /velocity_prod/agent_image_ref /velocity_prod/log_forwarder_s3_bucket /velocity_prod/log_forwarder_s3_bucket codeclimate/velocity-app:b100 codeclimate/velocity-consumer:b100 codeclimate/velocity-agent:b100 com.codeclimate.velocity.assets log_forwarder/package_b100.zip master 14
15. Allow for automated releases to production validate plan all branches 15
16. Allow for automated releases to production validate plan master apply 16
17. Allow for automated releases to production 17
18. Allow for automated releases to production 18
19. Allow for automated releases to production 19
20. Allow for automated releases to production 20
21. Allow for automated releases to production AccessDeniedException: User: arn:aws:sts::208491357083:assumed-role/velocity-prod-terraform-ci/1529374107443555604 is not authorized to perform: kinesis:UpdateShardCount on resource: arn:aws:kinesis:us-east-1:208491357083:stream/velocity-prod-agent-high 21
22. The failing build encourages an operator to take control and run `terraform apply` locally if it’s a change they expect. 22
23. Infrastructure and services should be easily placed into & out of maintenance mode goal #3 23
24. Our infrastructure and services should be easy to put into maintenance mode /velocity_prod/maintenance true apply enable maintenance mode apply the change 24
25. Our infrastructure and services should be easy to put into maintenance mode 25
26. Our infrastructure and services should be easy to put into maintenance mode 26
27. Our infrastructure and services should be easy to put into maintenance mode 27
28. Our infrastructure and services should be easy to put into maintenance mode 28
29. Operators should have timely, helpful notifications of activity goal #4 29
30. Operators should have timely, helpful notifications of activity 30
31. Operators should have timely, helpful notifications of activity 31
32. Operators should have timely, helpful notifications of activity 32
33. Operators should have timely, helpful notifications of activity 33
34. Operators should have controls to roll back changes goal #5 34
35. Operators should have timely, helpful notifications of activity 35
36. Operators should have timely, helpful notifications of activity 36
37. Operators should have timely, helpful notifications of activity 37
38. Operators should have controls to roll back changes 38
39. Recap! 39
40. Easily propose, review, and apply changes to infrastructure - PRs for Terraform config changes - CI and local terraform invocations - Remote state and state locking - Parameter Store for config values 40
41. Allow for automated releases to production - One pipeline for artifact build & test - Separate pipeline for terraform ops - Limited IAM policy for CI - Allow CI to update existing services - Build error trigger to run locally 41
42. Infrastructure and services should be easily placed into & out of maintenance mode - Parameter Store param: true/false - Trickles down through TF config - Swaps image for web services - Scales down non-web services - Continue to merge if you want 42
43. Operators should have timely, helpful notifications of activity - Build success/failure - Terraform output 43
44. Operators should have controls to roll back changes - Rollback instructions - Parameter Store artifact references - Trigger new “ops” build 44
45. Devon Blandin / Code Climate / USV DevOps Summit / July 10th, 2018 Fin. Thanks! Some additional resources: https://segment.com/blog/rebuilding-our-infrastructure/ https://zwischenzugs.com/2017/02/21/terraform-and-dynamic-environments/ https://charity.wtf/tag/terraform/ https://blog.gruntwork.io/how-to-create-reusable-infrastructure-with-terraform-modules-25526d65f73d https://medium.com/@petey5000/petes-terraform-tips-694a3c4c5169 45