Taming Terraform Workflow Using Terraform Modules and Github

Abstract

Today’s cloud infrastructure is really complex. What if you could truly make this infrastructure a black box?
What if you could mutate this infrastructure safely and easily? See how to use Terraform efficiently across
hundreds of developers while leveraging a simple Github workflow and Terraform modules.

Additional info

Hootsuite was one of the early adopters of Terraform and faced some challenges early on with Terraform workflow.
* This session will cover how Hootsuite uses Terraform modules and leverages them to bridge the gap between devs and
operations.
* We will discuss how we came up with our own state management system that helps us safely mutate the state of our
infrastructure from one state to the other. How you can organize your Terraform states across multiple tiers of your
infrastructure. We will also talk about our own HTTP remote state store on top of S3 that enables us to edit states
and rollback.
* Most importantly, we came up with our own Terraform workflow that uses github pull requests to do approvals and
state locking so we don’t corrupt Terraform states for our infrastructure.

Proposal: https://github.com/cfgmgmtcamp/2017-pdx-cfp/blob/master/presentations/taming-terraform-workflow.md

Transcript

1. Senior Developer - Production Operations and Delivery
   @anubhavm
   Taming Terraform
   Workflow
   Anubhav Mishra
   Using Terraform Modules and Github
   

   View Slide

2. Anubhav Mishra
   @anubhavm
   

   View Slide

3. View Slide

4. View Slide

5. Talk
   ● Terraform at Hootsuite
   ● Use of Terraform Modules (Dev <--> Ops)
   ● Atlantis - Hootsuite’s Terraform Workflow
   ● Live Demo
   

   View Slide

6. Terraform at Hootsuite
   

   View Slide

7. Numbers
   ● 100+ developers
   ● 20+ microservices
   ● 2 cluster schedulers
   ● 1000+ servers
   ● 1000s of AWS resources
   

   View Slide

8. Timeline
   ● Mid 2015 - First Terraform Repo (v1)
   ● Starting 2016 - Terraform (v2)
   ● End of 2016 - Terraform “Advanced” Workflow
   

   View Slide

9. Before Terraform
   

   View Slide

10. View Slide

11. View Slide

12. View Slide

13. What is Terraform?
    ● Terraform allows you to manage, build and visualize
    your infrastructure as code
    

    View Slide

14. View Slide

15. View Slide

16. View Slide

17. First Attempt
    

    View Slide

18. First Attempt
    

    View Slide

19. First Attempt
    

    View Slide

20. First Attempt
    staging != production
    

    View Slide

21. Terraform State
    

    View Slide

22. First Attempt
    

    View Slide

23. View Slide

24. Problems with our first attempt
    ● Environment drift
    ● One massive state for every environment
    ● Hard to collaborate on infrastructure
    

    View Slide

25. Problems with our first attempt
    ● Environment drift
    ● One massive state for every environment
    ● Hard to collaborate on infrastructure
    

    View Slide

26. Terraform Version 2 (Second Attempt)
    

    View Slide

27. Terraform Version 2 (Second Attempt)
    

    View Slide

28. Terraform Version 2 (Second Attempt)
    

    View Slide

29. Problems with our first attempt
    ● Environment drift
    ● One massive state for every environment
    ● Hard to collaborate on infrastructure
    

    View Slide

30. Problems with our first attempt
    ● Environment drift
    ● One massive state for every environment
    ● Hard to collaborate on infrastructure
    

    View Slide

31. Terraform Version 2 (Second Attempt)
    terraform.tfstate
    {
    "version": 3,
    "terraform_version": "0.8.8",
    "serial": 175,
    "lineage":
    "2cb11085-2e4e-40ff-bf81-e4ddf8bc8d5d",
    "modules": [
    {
    "path": [
    "root"
    ],
    "outputs": {},
    "resources": {
    ....
    

    View Slide

32. Terraform Version 2 (Second Attempt)
    terraform.tfstate
    {
    "version": 3,
    "terraform_version": "0.8.8",
    "serial": 175,
    "lineage":
    "2cb11085-2e4e-40ff-bf81-e4ddf8bc8d5d",
    "modules": [
    {
    "path": [
    "root"
    ],
    "outputs": {},
    "resources": {
    ....
    

    View Slide

33. Terraform Version 2 (Second Attempt)
    terraform.tfstate
    

    View Slide

34. Terraform Version 2 (Second Attempt)
    

    View Slide

35. Terraform Version 2 (Second Attempt)
    

    View Slide

36. Terraform Version 2 (Second Attempt)
    

    View Slide

37. View Slide

38. terraform.tfstate
    {
    "version": 3,
    "terraform_version": "0.8.8",
    "serial": 1,
    "lineage": "9e42c0ac-9e73-4910-8859-504934f2b499",
    "backend": {
    "type": "s3",
    "config": {
    "bucket": "terraform",
    "key": "foo-bar",
    "region": "us-east-1"
    },
    ....
    "outputs": {},
    "resources": {
    "null_resource.hello": {
    "type": "null_resource",
    "depends_on": [],
    "primary": {
    "id": "792399369485920658",
    "attributes": {
    "id": "792399369485920658"
    },
    "meta": {},
    ....
    

    View Slide

39. stash
    

    View Slide

40. View Slide

41. View Slide

42. View Slide

43. Problems with our first attempt
    ● Environment drift
    ● One massive state for every environment
    ● Hard to collaborate on infrastructure
    

    View Slide

44. Problems with our first attempt
    ● Environment drift
    ● One massive state for every environment
    ● Hard to collaborate on infrastructure
    

    View Slide

45. View Slide

46. View Slide

47. View Slide

48. module.server
    Let me make a
    generic server
    module
    I need 2 servers.
    Now!
    Terraform Version 2 (Second Attempt)
    

    View Slide

49. View Slide

50. Terraform Version 2 (Second Attempt)
    

    View Slide

51. Terraform Version 2 (Second Attempt)
    

    View Slide

52. Inputs Output
    

    View Slide

53. Terraform Version 2 (Second Attempt)
    

    View Slide

54. Terraform Version 2 (Second Attempt)
    

    View Slide

55. Terraform Version 2 (Second Attempt)
    

    View Slide

56. Terraform Version 2 (Second Attempt)
    

    View Slide

57. Terraform Version 2 (Second Attempt)
    139 Repositories
    

    View Slide

58. ● Multi Environment Terraform Projects
    ● Isolated Terraform States for Services
    ○ Smaller Failure Domain
    ○ Rapid Iteration on Infrastructure
    ● Use of Terraform Modules
    ○ Environment Parity
    ○ Increased Collaboration Between Ops and Dev
    

    View Slide

59. Are we all done?
    

    View Slide

60. View Slide

61. View Slide

62. View Slide

63. How do we continue to move
    fast while we add people?
    

    View Slide

64. View Slide

65. atlantis
    A unified workflow for collaborating on Terraform through GitHub.
    https://github.com/hootsuite/atlantis
    

    View Slide

66. Demo
    

    View Slide

67. ● atlantis - https://github.com/hootsuite/atlantis
    ● cfgmgmtcamp-atlantis-demo -
    https://github.com/anubhavmishra/cfgmgmtcamp-atlantis-demo
    Links
    

    View Slide

68. Thank You!
    Senior Developer - Production Operations and Delivery
    @anubhavm
    Anubhav Mishra
    

    View Slide