Maintaining an open source library of production-ready CloudFormation templates

Slide 1

Slide 1 text

https://github.com/ widdix/aws-cf-templates 1

Slide 2

Slide 2 text

Hello! I’m Michael Wittig

Slide 3

Slide 3 text

What do I do? 3 Independent consultant focusing on AWS & DevOps https://widdix.net Writer Amazon Web Services in Action (Manning, 2nd ed) & https://cloudonaut.io AWS

Slide 4

Slide 4 text

4 Maintaining an open source library of production-ready CloudFormation templates

Slide 5

Slide 5 text

Open source Apache License 2.0

Slide 6

Slide 6 text

6 Production-ready Reviewed by experts Pull Requests by default. Automated test suite verifies templates on every change and weekly. Secure Keep security groups as tight as possible, avoid * in IAM policies, bastion host, IAM SSH, keep AMIs up-to-date... Highly available No single point of failures by default or documented limitations. Scalable EC2 instances are auto scaled by default or documented limitations. Easy to deploy and update Everything is in CloudFormation. Built-in monitoring and logging Log files are shipped to CloudWatch Logs. Important metrics are monitored with CloudWatch Alarms.

Slide 7

Slide 7 text

CloudFormation Infrastructure as Code 7

Slide 8

Slide 8 text

How does it work? 8 Template CloudFormatio n Stack

Slide 9

Slide 9 text

1. Example Let’s start with an example. 9

Slide 10

Slide 10 text

Jenkins 10 VPC Alert Bastion Host Jenkins

Slide 11

Slide 11 text

11 VPC × VPC × Subnets × Internet Gateway × Route Tables × Network ACLs

Slide 12

Slide 12 text

12 Alert × SNS Topic × Topic Policy × Subscriptions

Slide 13

Slide 13 text

Demo AWS Management Console 13

Slide 14

Slide 14 text

14 Bastion Host × Elastic IP × SSH via IAM public SSH key × CloudWatch Logs × CloudWatch Alarms × Security Group × Auto Scaling Group (1:1:1)

Slide 15

Slide 15 text

15 Jenkins × Load Balancer × SSH via IAM public SSH key × CloudWatch Logs × CloudWatch Alarms × Security Group × EFS File System × Master Auto Scaling Group (1:1:1) × Agent Auto Scaling Group

Slide 16

Slide 16 text

2. Modularization Reuse templates and modularization. 16

Slide 17

Slide 17 text

How does it work? 17 Parameters “Parent Stack” Stack Exports

Slide 18

Slide 18 text

18 --- Parameters: ParentVPCStack: Type: String Resources: SecurityGroup: Type: 'AWS::EC2::SecurityGroup' Properties: VpcId: 'Fn::ImportValue': !Sub '${ParentVPCStack}-VPC' Outputs: SecurityGroup: Value: !Ref SecurityGroup Export: Name: !Sub '${ AWS::StackName}-SG' vpc-ssh-bastion.yaml

Slide 19

Slide 19 text

19 Learnings × Exports are super useful, thanks, AWS!

Slide 20

Slide 20 text

3. Verify templates How can we test if a template is working? 20

Slide 21

Slide 21 text

Automated tests × yamllint × aws cloudformation validate-template × deploy & test 21

Slide 22

Slide 22 text

22 Deploy & test @Test public void test() { KeyPair key = this.createKey("key"); this.createStack("vpc", "vpc/vpc-2azs.yaml", [...]); this.createStack("ssh-bastion", "vpc/vpc-ssh-bastion.yaml", [...]); String host = this.getStackOutputValue("ssh-bastion", "IPAddress"); this.probeSSH(host, key); this.deleteStack("ssh-bastion"); this.deleteStack("vpc"); this.deleteKey("key"); }

Slide 23

Slide 23 text

23 × CloudFront is super unstable during delete × Use unique names to enable parallel tests × Writing your test functions is hard × Expensive regarding AWS costs × Caught many bugs × RDS snapshots are created on deletion, breaking change! Learnings

Slide 24

Slide 24 text

4. Keep stacks up-to-date What’s the best way to keep stacks up-to-date when new template versions are released? 24

Slide 25

Slide 25 text

Sorry I don’t know yet. But I’m working on it. 25

Slide 26

Slide 26 text

26 Pipeline App A Stage A App B Stage B VPC Alert Bastion VPC Alert Bastion

Slide 27

Slide 27 text

Demo AWS Management Console 27

Slide 28

Slide 28 text

StackSets Create, update, or delete stacks across multiple accounts and regions with a single operation. 28

Slide 29

Slide 29 text

Sorry 29 No CloudFormation support yet.

Slide 30

Slide 30 text

5. Sustainable open source How to create a sustainable library? 30

Slide 31

Slide 31 text

31 Sustainable Users 620 stars 224 forks Core Contributors 2 people 1 company Community Contributions 11 people 52 PRs Sponsor a feature or bug fix Training and Consulting

Slide 32

Slide 32 text

6. Templates What templates are available today? 32

Slide 33

Slide 33 text

33 Templates https://github.com/widdix/aws-cf-templates VPC × Public/Private × Nat Gateway × Route53 Zone * × SSH Bastion Host × Flow Logs × VPC Endpoints ECS × Cluster × Service Security × Auth Proxy (GitHub) × CloudTrail × Config × Password Policy Operations × Alert × DynamoDB backup Applications × Jenkins × WordPress EC2 × Auto Recovery * https://github.com/widdix/aws-cf-templates/pull/122

Slide 34

Slide 34 text

Roadmap × Dashboards × Update stacks × Improve runtime of test suite × Improve Monitoring × Docs are not for newcomers × Load testing tool to fine tune scaling triggers × CLI tool 34

Slide 35

Slide 35 text

Save 40% 35 Second Edition Use code awstcd17 during checkout at www.manning.com × New chapters: Lambda, EFS, ElastiCache × YAML templates × Updated everything Code expires on October 8, 2017

Slide 36

Slide 36 text

https://github.com/ widdix/aws-cf-templates You can find me at: @hellomichibye [email protected] 36 And now? Try it!

Slide 37

Slide 37 text

Credits Special thanks to all the people who made and released these awesome resources for free: × Presentation template by SlidesCarnival × Photographs by Pexels and Unsplash × Watercolor textures by GraphicBurguer 37