Terraforming

My team heavily uses both Serverless and Terraform to build infrastructure, but as the number of projects we managed grew, it got harder to know how components were glued in different projects.
So we decided to try different ways to structure our projects in order
to :
– make better use terraform modules
– run terraform in a more secure manner
– share serverless components in a reliable way
– abstract complex infrastructure gluing
– faster project bootstrap times

Transcript

1. Terraforming David Przybilla dav009 dav009
2. ! ίϩϯϏΞ
3. first time at JAWS
4. Tokyo
5. until recently
 Data Engineering / NLP
6. Slowly get into
 Infrastructure
7. ..Ops + Golang + Python..
8. first week..started looking at projects..
9. first week..started looking at projects.. Github repos…
10. first week..started looking at projects.. Github repos… New project I had to work on…
13. code
14. code terraform apply
15. code aws infrastructure terraform apply
16. lots of small services
17. lots of infrastructure
18. ..lambda functions.. ..kinesis.. ..sqs.. ..ecs..
19. early adopters of terraform
20. many terraform versions
21. many different project structures
22. bootstrapping a project is also tough
23. where do I run terraform?
24. what role should run terraform? where do I run terraform?
25. what role should run terraform? where do I run terraform? ..credentials..
26. bootstrapping my project took longer than I wanted
27. ..complex project structure..
28. sample_old_project/
29. sample_old_project/ tf/
30. sample_old_project/ tf/ lambda_functions.tf
31. sample_old_project/ tf/ lambda_functions.tf iam.tf
32. sample_old_project/ tf/ lambda_functions.tf iam.tf apigateways.tf
33. sample_old_project/ tf/ lambda_functions.tf iam.tf apigateways.tf policies.tf ecs….
34. sample_old_project/ tf/ lambda_functions.tf iam.tf apigateways.tf policies.tf ecs….
36. api gateways
38. ..when it gets bigger tho..
42. many api gateways many lambdas many iam roles.. many everything..
43. many api gateways many lambdas many iam roles.. many everything.. even they are not exactly related to each other
44. I had to keep all pieces in my head at the same time
45. ..experiment a bit..
46. my_new_project/ tf
 /modules /stg /prd
47. my_new_project/ tf
 /modules /stg /prd
48. my_new_project/ tf
 /modules /stg /prd ..shared infrastructure.. between prd and stg
49. my_new_project/ tf
 /modules /stg .tfversion /prd .tfversion
50. my_new_project/ tf
 /modules /stg main.tf .tfversion /prd main.tf .tfversion …just import modules..
51. module: should be like a function
52. function/module: purpose: does only one thing
53. module: parameters outputs inside is invisible to importer
54. module: abstracts complexity
55. module: abstracts complexity don’t need to know how things are connected
56. module: abstracts complexity don’t need to know how things are connected don't need to see all pieces
57. database_synchornizer: main.tf vars.tf output.tf readme.md lambda.tf iam.tf … module connect infrastructure together input parameters output parameters
58. database_synchornizer: main.tf vars.tf output.tf lambda.tf iam.tf … module all related
59. database_synchornizer: main.tf vars.tf output.tf lambda.tf iam.tf … module all related less pieces
60. database_synchornizer: main.tf vars.tf output.tf lambda.tf iam.tf … module
61. database_synchornizer: main.tf vars.tf output.tf lambda.tf iam.tf … module api: main.tf vars.tf output.tf …. apigateways …
62. database_synchornizer: main.tf vars.tf output.tf lambda.tf iam.tf … module api: main.tf vars.tf output.tf …. apigateways … ecs-cluster: main.tf vars.tf output.tf ….
63. less to remember
64. less to remember other modules connect via output variables
65. we realise we could reuse modules
66. so we made our own GitHub repo only for modules, similar to: we realise we could share modules
67. github.com/terraform-community-modules github.com/segmentio/stack we realise we could reuse modules so we made our own GitHub repo only for modules, similar to:
68. ..looking at other projects..
69. ..looking at other projects.. ..how to run terraform..
70. what terraform version is this using?
71. what role should I assume to run terraform for this project?
72. ..Don’t forget to change environment..
73. project A: do this but don’t do that
74. project B: do this other thing but don’t do that other one
75. ..running terraform is scary…
76. ..take down service..
77. starting a new project?
78. starting a new project? difficult to bootstrap
79. bootstrapping should:
80. bootstrapping should: be fast
81. bootstrapping should: take little effort be fast
82. bootstrapping should: take little effort be fast be easy for a new team member
83. bootstrapping should: take little effort be fast be easy for a new team member complexity should be hidden
84. what to do?
85. ..a module.. .. a terraform automated pipeline that takes 10 min to set up..
86. ..a trigger..
90. codepipeline codecommit codebuild
92. how do we use it?
93. 1. import module: terraform_pipeline 2. follow project structure
94. module “terraform_pipeline" { source = “git::git@:…//terraform_pipeline?ref=vx.y” notifications_channel = “slack channel” terraform_role_arn = “some::arn::role” env = "stg" name = “my_project" } importing this module
95. module “terraform_pipeline" { source = “git::git@:…//terraform_pipeline?ref=vx.y” notifications_channel = “slack-channel” terraform_role_arn = “some::arn::role” env = "stg" name = “my_project" } importing this module
96. module “terraform_pipeline" { source = “git::git@:…//terraform_pipeline?ref=vx.y” notifications_channel = “slack-channel” terraform_role_arn = “some::arn::role” env = "stg" name = “my_project" } importing this module
97. module “terraform_pipeline" { source = “git::git@:…//terraform_pipeline?ref=vx.y” notifications_channel = “slack-channel” terraform_role_arn = “some::arn::role” env = "stg" name = “my_project" } no need to remember this role anymore! importing this module
98. module “terraform_pipeline" { source = “git::git@:…//terraform_pipeline?ref=vx.y” notifications_channel = “slack-channel” terraform_role_arn = “some::arn::role” env = "stg" name = “my_project" } importing this module
99. module “terraform_pipeline" { source = “git::git@:…//terraform_pipeline?ref=vx.y” notifications_channel = “slack-channel” terraform_role_arn = “some::arn::role” env = "stg" name = “my_project" } importing this module
100. .. details are hidden for importer ..
101. .. details are hidden for importer .. importer does not need to worry about :
102. importer does not need to worry about : how to send notifications to slack .. details are hidden for importer ..
103. .. details are hidden for importer .. importer does not need to worry about : how to run terraform how to send notifications to slack
104. .. details are hidden for importer .. importer does not need to worry about : how to run terraform module knows that (inside) how to send notifications to slack
105. .. details are hidden for importer .. importer does not need to worry about : how to run terraform module knows that (inside) how to send notifications to slack a new team member can import it quickly
106. my_new_project/ tf
 /modules /stg .tfversion /prd .tfversion
107. my_new_project/ tf
 /modules /stg .tfversion /prd .tfversion terraform ci module : 1. downloads terraform version specified in .tfversion 2. knows how to run a terraform binary 3. knows how to get git credentials to fetch to our private modules 4. knows how to send notifications to slack
108. my_new_project/ tf
 /modules /stg .tfversion /prd .tfversion want to run terraform for stg? go to stg folder
109. my_new_project/ tf
 /modules /stg .tfversion /prd .tfversion want to run terraform for stg? go to stg folder want to run terraform for prd? go to prd folder
114. reduced the complexity of bootstrapping a project
115. reduced the complexity of bootstrapping a project means a faster team
116. reduced the complexity of bootstrapping a project means a faster team also I am less scared to run terraform
117. reduced the complexity of bootstrapping a project means a faster team also I am less scared to run terraform all projects use the same pipeline
118. whats the next step to hide complexity?
119. whats the next step to hide complexity?
120. github.com/serverless/serverless
121. serverless serverless “applications”
122. serverless serverless “applications” code + glue + infrastructure i.e: serverless service to get a slack bot via FaaS
123. serverless serverless “applications” > serverless install --url <service-github-url> > sls deploy code + glue + infrastructure i.e: serverless service to get a slack bot via FaaS
124. please be aware a few things have changed:
125. - Terraform community modules - Serverless application repository (aws)
126. how are you running terraform?
127. how are you structuring your terraform projects ? how are you running terraform?
128. how are you structuring your terraform projects ? how are you running terraform? my team is using terraform + github.com/serverless/serverless
129. how are you structuring your terraform projects ? how are you running terraform? my team is using terraform + github.com/serverless/serverless feel free to talk to me! I would like to learn more
130. Thanks dav009 dav009 Q & A bit.ly/jd2018-sls