Upgrade to Pro — share decks privately, control downloads, hide ads and more …

TER301: Terraforming Safely and Collaboratively on AWS with Atlantis

TER301: Terraforming Safely and Collaboratively on AWS with Atlantis

Presented during AWS Summit 2019 Developer Lounge and AWS User Group Meetup May 2019

7bcb468888a82bfeb38a2818207e53c6?s=128

Steve Teo

April 10, 2019
Tweet

More Decks by Steve Teo

Other Decks in Technology

Transcript

  1. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T TER301: Terraforming AWS safely and collaboratively with Atlantis Steve Teo Director of Cloud Security Engineering Horangi Cyber Security www.linkedin.com/in/steveteo D e v e l o p e r L o u n g e
  2. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Steve ”Potay” Teo • CloudDevSecOps Fanatic • Currently building products at • Progression => UI/UX Designer / Developer => Full Stack Developer => Build & Release Engineer => Infrastructure & Tools Architect • AWS Areas of Interests: AWS Multi-Account Strategy, Cloud Security • Totally uncertified and proud :P
  3. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Communities I serve www.meetup.com/AWS-SG/ www.meetup.com/ Atlassian-User-Group-Singapore/
  4. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T AWS User Group Singapore - Monthly
  5. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T AWS User Group Singapore - Monthly
  6. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T AWS User Group Singapore - Monthly
  7. Before we start, a quick poll. How many of you

    know what Terraform is? How many of you are already using Terraform? How many of you are using Terraform in a team setting?
  8. None
  9. If you start using Terraform alone, it is definitely going

    to look like this.
  10. None
  11. However, if you are starting to use Terraform in a

    team setting, it is definitely going to look like this.
  12. None
  13. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T What this talk is about • Quick Introduction to Terraform • Challenges around Terraform State Management and Collaboration • How Atlantis solves these challenges safely and collaboratively • Demo What this talk is not about • Every other advanced Terraform topic • Terraform vs CloudFormation • Every Little Atlantis Feature! RTFM
  14. None
  15. None
  16. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Providers • Multi-Provider Orchestration • Abstracts APIs as resources • Context for resources
  17. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Providers • Multi-Provider Orchestration • Abstracts APIs as resources • Context for resources aws_instance aws_ec2_transit_gateway aws_db_instance
  18. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Providers • Multi-Provider Orchestration • Abstracts APIs as resources • Context for resources
  19. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Write Desired State • Think of how the world should be • Infrastructure as Code • Versionable • Specify dependencies
  20. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Terraform Plan • Preview changes before applying • Reduces mistakes & uncertainty • Map resource dependencies
  21. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Terraform Plan • Preview changes before applying • Reduces mistakes & uncertainty • Map resource dependencies
  22. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Terraform Apply • Reach the desired state • Live result log • Safe Orchestration
  23. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T One Simple Workflow
  24. None
  25. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Desired State vs Current State
  26. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Desired State vs Current State
  27. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T State File
  28. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T State file is Sacred! • Real world mapping of resources • Track metadata of resource and its dependency with respect to other resources • State is exposed - Protect your state file at all cost! • Teams collaborate via state files
  29. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Local Terraform Usage Pros • Simple Issues • No backup of state files • Not even possible to collaborate in a team setting • Different developers have different Terraform environments • Doesn’t integrate into a CI/CD workflow
  30. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Local Terraform Usage Pros • Simple Issues • No backup of state files • Not even possible to collaborate in a team setting • Different developers have different Terraform environments • Doesn’t integrate into a CI/CD workflow
  31. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T { Ops } { Dev } { Ops } { Dev } { Dev } Solo Development
  32. State files needs to be shared and backed up

  33. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Remote Backend: S3 • Durable Storage • Enable Bucket Versioning • DynamoDB Locking - State can be accessed safely
  34. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Remote Backend: S3 • Durable Storage • Enable Bucket Versioning • DynamoDB Locking - State can be accessed safely
  35. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Remote Backend Terraform Usage Cons • No backup of state files • Not even really possible to collaborate in a team setting • Different developers have different Terraform environments • Doesn’t integrate into a CI/CD workflow our
  36. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T { Ops } { Dev } { Ops } { Dev } { Dev } Now only Ops can perform change
  37. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T { Ops } { Dev } { Ops } { Dev } { Dev } Now only Ops can perform change
  38. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Workflow before Atlantis
  39. None
  40. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Atlantis • Started in HootSuite by Anubhav Mishra and currently maintained by Luke Kysow • Battle-tested at HootSuite with 100 developers working on 600 Terraform Repositories • Self-hosted • Open-source, supported by Hashicorp • Well written docs • Active maintainer
  41. Atlantis monitors git pull requests and centralizes Terraform Plans and

    Applies and enforces collaboration through pull requests
  42. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Monitors git pull requests
  43. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Monitors git pull requests
  44. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Centralizes Terraform Plans and Applies
  45. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Centralizes Terraform Plans and Applies
  46. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Centralizes Terraform Plans and Applies
  47. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Centralizes Terraform Plans and Applies • Issue: Pull requests can become really noisy through multiple plan iterations • Solution: Iterate locally against a Sandbox account or against read only state in s3 bucket
  48. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Enforces collaboration through pull requests • Pull Request Approval Check • Merge-ability Check
  49. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Enforces collaboration through pull requests • Pull Request Approval Check • Merge-ability Check
  50. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Enforces collaboration through pull requests
  51. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T
  52. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Standardized Workflow after Atlantis
  53. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T { Ops } { Dev } { Ops } { Dev } { Dev } Developers Writing Infrastructure Code
  54. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Developers Writing Infrastructure Code { Ops } { Dev } { Ops } { Dev } { Dev }
  55. None
  56. S U M M I T © 2019, Amazon Web

    Services, Inc. or its affiliates. All rights reserved.
  57. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Gotchas I Found - Approval / mergability checks doesn’t really work in solo based scenarios in a team based setting - Works in a very plan / apply centric model. No out of the box support for other terraform state manipulation commands, or even destroy. All this needs to be handled - Security models for working within Atlantis workflows are still being worked out, eg. https://github.com/runatlantis/atlantis/issues/308. Definitely evaluate this very carefully if you are using this for production system within a low-trust team model
  58. S U M M I T © 2019, Amazon Web

    Services, Inc. or its affiliates. All rights reserved.
  59. S U M M I T © 2019, Amazon Web

    Services, Inc. or its affiliates. All rights reserved.
  60. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T References • www.speakerdeck.com/anubhavmishra/taming-terraform-workflow- using-terraform-modules-and-github-1?slide=67 • www.speakerdeck.com/anubhavmishra/taming-infrastructure- workflow-at-scale-with-terraform • www.medium.com/runatlantis/introducing-atlantis-6570d6de7281 • www.medium.com/runatlantis/putting-the-dev-into-devops-why- your-developers-should-write-terraform-too-d3c079dfc6a8 • https://medium.com/runatlantis/moving-atlantis-to-runatlantis- atlantis-on-github-4efc025bb05f
  61. Thank you! S U M M I T © 2019,

    Amazon Web Services, Inc. or its affiliates. All rights reserved. Steve Teo Director of Cloud Security Engineering Horangi Cyber Security www.linkedin.com/in/steveteo
  62. S U M M I T © 2019, Amazon Web

    Services, Inc. or its affiliates. All rights reserved.