Maintaining an open source library of production-ready CloudFormation templates

CloudFormation is the standard to provision AWS resources. But developing a template is a lot of work. Let’s speed up development and maintenance by working together on high-quality templates: Reviewed by certified experts, secure, highly available, scalable, easy to deploy and update, built-in monitoring and logging; As a maintainer of https://github.com/widdix/aws-cf-templates I will
share what I learned by answering the following questions: What is a
production-ready template? How to reuse templates? How to modularize
templates? How to keep stacks updated? How to assure that templates are
working? How to create a sustainable library?

Transcript

1. https://github.com/ widdix/aws-cf-templates 1
2. Hello! I’m Michael Wittig
3. What do I do? 3 Independent consultant focusing on AWS & DevOps https://widdix.net Writer Amazon Web Services in Action (Manning, 2nd ed) & https://cloudonaut.io AWS
4. 4 Maintaining an open source library of production-ready CloudFormation templates
5. Open source Apache License 2.0
6. 6 Production-ready Reviewed by experts Pull Requests by default. Automated test suite verifies templates on every change and weekly. Secure Keep security groups as tight as possible, avoid * in IAM policies, bastion host, IAM SSH, keep AMIs up-to-date... Highly available No single point of failures by default or documented limitations. Scalable EC2 instances are auto scaled by default or documented limitations. Easy to deploy and update Everything is in CloudFormation. Built-in monitoring and logging Log files are shipped to CloudWatch Logs. Important metrics are monitored with CloudWatch Alarms.
7. CloudFormation Infrastructure as Code 7
8. How does it work? 8 Template CloudFormatio n Stack
9. 1. Example Let’s start with an example. 9
10. Jenkins 10 VPC Alert Bastion Host Jenkins
11. 11 VPC × VPC × Subnets × Internet Gateway × Route Tables × Network ACLs
12. 12 Alert × SNS Topic × Topic Policy × Subscriptions
13. Demo AWS Management Console 13
14. 14 Bastion Host × Elastic IP × SSH via IAM public SSH key × CloudWatch Logs × CloudWatch Alarms × Security Group × Auto Scaling Group (1:1:1)
15. 15 Jenkins × Load Balancer × SSH via IAM public SSH key × CloudWatch Logs × CloudWatch Alarms × Security Group × EFS File System × Master Auto Scaling Group (1:1:1) × Agent Auto Scaling Group
16. 2. Modularization Reuse templates and modularization. 16
17. How does it work? 17 Parameters “Parent Stack” Stack Exports
18. 18 --- Parameters: ParentVPCStack: Type: String Resources: SecurityGroup: Type: 'AWS::EC2::SecurityGroup' Properties: VpcId: 'Fn::ImportValue': !Sub '${ParentVPCStack}-VPC' Outputs: SecurityGroup: Value: !Ref SecurityGroup Export: Name: !Sub '${ AWS::StackName}-SG' vpc-ssh-bastion.yaml
19. 19 Learnings × Exports are super useful, thanks, AWS!
20. 3. Verify templates How can we test if a template is working? 20
21. Automated tests × yamllint × aws cloudformation validate-template × deploy & test 21
22. 22 Deploy & test @Test public void test() { KeyPair key = this.createKey("key"); this.createStack("vpc", "vpc/vpc-2azs.yaml", [...]); this.createStack("ssh-bastion", "vpc/vpc-ssh-bastion.yaml", [...]); String host = this.getStackOutputValue("ssh-bastion", "IPAddress"); this.probeSSH(host, key); this.deleteStack("ssh-bastion"); this.deleteStack("vpc"); this.deleteKey("key"); }
23. 23 × CloudFront is super unstable during delete × Use unique names to enable parallel tests × Writing your test functions is hard × Expensive regarding AWS costs × Caught many bugs × RDS snapshots are created on deletion, breaking change! Learnings
24. 4. Keep stacks up-to-date What’s the best way to keep stacks up-to-date when new template versions are released? 24
25. Sorry I don’t know yet. But I’m working on it. 25
26. 26 Pipeline App A Stage A App B Stage B VPC Alert Bastion VPC Alert Bastion
27. Demo AWS Management Console 27
28. StackSets Create, update, or delete stacks across multiple accounts and regions with a single operation. 28
29. Sorry 29 No CloudFormation support yet.
30. 5. Sustainable open source How to create a sustainable library? 30
31. 31 Sustainable Users 620 stars 224 forks Core Contributors 2 people 1 company Community Contributions 11 people 52 PRs Sponsor a feature or bug fix Training and Consulting
32. 6. Templates What templates are available today? 32
33. 33 Templates https://github.com/widdix/aws-cf-templates VPC × Public/Private × Nat Gateway × Route53 Zone * × SSH Bastion Host × Flow Logs × VPC Endpoints ECS × Cluster × Service Security × Auth Proxy (GitHub) × CloudTrail × Config × Password Policy Operations × Alert × DynamoDB backup Applications × Jenkins × WordPress EC2 × Auto Recovery * https://github.com/widdix/aws-cf-templates/pull/122
34. Roadmap × Dashboards × Update stacks × Improve runtime of test suite × Improve Monitoring × Docs are not for newcomers × Load testing tool to fine tune scaling triggers × CLI tool 34
35. Save 40% 35 Second Edition Use code awstcd17 during checkout at www.manning.com × New chapters: Lambda, EFS, ElastiCache × YAML templates × Updated everything Code expires on October 8, 2017
36. https://github.com/ widdix/aws-cf-templates You can find me at: @hellomichibye michael@widdix.de 36 And now? Try it!
37. Credits Special thanks to all the people who made and released these awesome resources for free: × Presentation template by SlidesCarnival × Photographs by Pexels and Unsplash × Watercolor textures by GraphicBurguer 37