Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Maintaining an open source library of production-ready CloudFormation templates

E8f66870d1204779ecc45f2695faa73e?s=47 Michael Wittig
September 28, 2017

Maintaining an open source library of production-ready CloudFormation templates

CloudFormation is the standard to provision AWS resources. But developing a template is a lot of work. Let’s speed up development and maintenance by working together on high-quality templates: Reviewed by certified experts, secure, highly available, scalable, easy to deploy and update, built-in monitoring and logging; As a maintainer of https://github.com/widdix/aws-cf-templates I will
share what I learned by answering the following questions: What is a
production-ready template? How to reuse templates? How to modularize
templates? How to keep stacks updated? How to assure that templates are
working? How to create a sustainable library?

E8f66870d1204779ecc45f2695faa73e?s=128

Michael Wittig

September 28, 2017
Tweet

Transcript

  1. https://github.com/ widdix/aws-cf-templates 1

  2. Hello! I’m Michael Wittig

  3. What do I do? 3 Independent consultant focusing on AWS

    & DevOps https://widdix.net Writer Amazon Web Services in Action (Manning, 2nd ed) & https://cloudonaut.io AWS
  4. 4 Maintaining an open source library of production-ready CloudFormation templates

  5. Open source Apache License 2.0

  6. 6 Production-ready Reviewed by experts Pull Requests by default. Automated

    test suite verifies templates on every change and weekly. Secure Keep security groups as tight as possible, avoid * in IAM policies, bastion host, IAM SSH, keep AMIs up-to-date... Highly available No single point of failures by default or documented limitations. Scalable EC2 instances are auto scaled by default or documented limitations. Easy to deploy and update Everything is in CloudFormation. Built-in monitoring and logging Log files are shipped to CloudWatch Logs. Important metrics are monitored with CloudWatch Alarms.
  7. CloudFormation Infrastructure as Code 7

  8. How does it work? 8 Template CloudFormatio n Stack

  9. 1. Example Let’s start with an example. 9

  10. Jenkins 10 VPC Alert Bastion Host Jenkins

  11. 11 VPC × VPC × Subnets × Internet Gateway ×

    Route Tables × Network ACLs
  12. 12 Alert × SNS Topic × Topic Policy × Subscriptions

  13. Demo AWS Management Console 13

  14. 14 Bastion Host × Elastic IP × SSH via IAM

    public SSH key × CloudWatch Logs × CloudWatch Alarms × Security Group × Auto Scaling Group (1:1:1)
  15. 15 Jenkins × Load Balancer × SSH via IAM public

    SSH key × CloudWatch Logs × CloudWatch Alarms × Security Group × EFS File System × Master Auto Scaling Group (1:1:1) × Agent Auto Scaling Group
  16. 2. Modularization Reuse templates and modularization. 16

  17. How does it work? 17 Parameters “Parent Stack” Stack Exports

  18. 18 --- Parameters: ParentVPCStack: Type: String Resources: SecurityGroup: Type: 'AWS::EC2::SecurityGroup'

    Properties: VpcId: 'Fn::ImportValue': !Sub '${ParentVPCStack}-VPC' Outputs: SecurityGroup: Value: !Ref SecurityGroup Export: Name: !Sub '${ AWS::StackName}-SG' vpc-ssh-bastion.yaml
  19. 19 Learnings × Exports are super useful, thanks, AWS!

  20. 3. Verify templates How can we test if a template

    is working? 20
  21. Automated tests × yamllint × aws cloudformation validate-template × deploy

    & test 21
  22. 22 Deploy & test @Test public void test() { KeyPair

    key = this.createKey("key"); this.createStack("vpc", "vpc/vpc-2azs.yaml", [...]); this.createStack("ssh-bastion", "vpc/vpc-ssh-bastion.yaml", [...]); String host = this.getStackOutputValue("ssh-bastion", "IPAddress"); this.probeSSH(host, key); this.deleteStack("ssh-bastion"); this.deleteStack("vpc"); this.deleteKey("key"); }
  23. 23 × CloudFront is super unstable during delete × Use

    unique names to enable parallel tests × Writing your test functions is hard × Expensive regarding AWS costs × Caught many bugs × RDS snapshots are created on deletion, breaking change! Learnings
  24. 4. Keep stacks up-to-date What’s the best way to keep

    stacks up-to-date when new template versions are released? 24
  25. Sorry I don’t know yet. But I’m working on it.

    25
  26. 26 Pipeline App A Stage A App B Stage B

    VPC Alert Bastion VPC Alert Bastion
  27. Demo AWS Management Console 27

  28. StackSets Create, update, or delete stacks across multiple accounts and

    regions with a single operation. 28
  29. Sorry 29 No CloudFormation support yet.

  30. 5. Sustainable open source How to create a sustainable library?

    30
  31. 31 Sustainable Users 620 stars 224 forks Core Contributors 2

    people 1 company Community Contributions 11 people 52 PRs Sponsor a feature or bug fix Training and Consulting
  32. 6. Templates What templates are available today? 32

  33. 33 Templates https://github.com/widdix/aws-cf-templates VPC × Public/Private × Nat Gateway ×

    Route53 Zone * × SSH Bastion Host × Flow Logs × VPC Endpoints ECS × Cluster × Service Security × Auth Proxy (GitHub) × CloudTrail × Config × Password Policy Operations × Alert × DynamoDB backup Applications × Jenkins × WordPress EC2 × Auto Recovery * https://github.com/widdix/aws-cf-templates/pull/122
  34. Roadmap × Dashboards × Update stacks × Improve runtime of

    test suite × Improve Monitoring × Docs are not for newcomers × Load testing tool to fine tune scaling triggers × CLI tool 34
  35. Save 40% 35 Second Edition Use code awstcd17 during checkout

    at www.manning.com × New chapters: Lambda, EFS, ElastiCache × YAML templates × Updated everything Code expires on October 8, 2017
  36. https://github.com/ widdix/aws-cf-templates You can find me at: @hellomichibye michael@widdix.de 36

    And now? Try it!
  37. Credits Special thanks to all the people who made and

    released these awesome resources for free: × Presentation template by SlidesCarnival × Photographs by Pexels and Unsplash × Watercolor textures by GraphicBurguer 37