AWS CloudFormation

Amazing and powerful abilities of AWS CloudFormation that mostly pass unnoticed. How to decouple stacks using exported values, how to utilize `cfn-init` and update code on existing servers, how to create completely custom resources for CloudFormation to manage and more!

This was presented at the Devops in Israel meetup event https://www.meetup.com/devops-in-israel/events/254851565/

Transcript

1. AWS CloudFormation
2. Evgeny Zislis
3. { "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "EC2 Instance and a Security Group", "Parameters" : { "KeyName": { "Description" : "An existing EC2 KeyPair to enable SSH access", "Type": "AWS::EC2::KeyPair::KeyName" }, "InstanceType" : { "Description" : "EC2 instance type", "Type" : "String", "Default" : "t2.small", "AllowedValues" : [ "t2.small", "t2.medium", "t2.large"] }, "SSHLocation" : { "Description" : "CIDR from where SSH access is allowed", "Type": "String", "MinLength": "9", "MaxLength": "18", "Default": "0.0.0.0/0", "AllowedPattern": "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})" } ...
4. "Resources" : { "EC2Instance" : { "Type" : "AWS::EC2::Instance", "Properties" : { "InstanceType" : { "Ref" : "InstanceType" }, "SecurityGroups" : [ { "Ref" : "InstanceSecurityGroup" } ], "KeyName" : { "Ref" : "KeyName" }, "ImageId" : { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref" : "AWS::Region" }, { "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" }, "Arch" ] } ] } } }, "InstanceSecurityGroup" : { "Type" : "AWS::EC2::SecurityGroup", "Properties" : { "GroupDescription" : "Enable SSH access via port 22", "SecurityGroupIngress" : [ { "IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22", "CidrIp" : { "Ref" : "SSHLocation"} } ] } }
5. /prəˈviZHən/ supply with equipment the IT manager was responsible to provision servers for the production release pro·vi·sion
6. /ˈôdəˌmāt/ convert a process to reduce human intervention to a minimum jobs that used to be done by people are now able to be done through automation au·to·mate
7. /rəˈpēdəb(ə)l/ able to be done again repeatable experiments under laboratory conditions re·peat·a·ble
8. /prəˈdiktəb(ə)l/ behaving or occurring in a way that is expected. unsurprising. Murphy's unpredictable rule predictably hits you pre·dict·a·ble
9. CloudFormation is a tool for repeatable and predictable automation of cloud infrastructure provisioning
11. Create a VPC aws --region us-east-1 \ cloudformation create-stack \ --stack-name VPC \ --template-url https://aws-quickstart.s3.amazonaws.com/quickstart-aws-vpc/templates/aws-vpc.template \ --parameters \ ParameterKey=KeyPairName,ParameterValue=evgeny-ssh \ ParameterKey=AvailabilityZones,ParameterValue="'us-east-1a,us-east-1b'" { "StackId": "arn:aws:cloudformation:us-east-1:675783095585:stack/VPC/76271620-d0f8-11e8-b3a0-500c28b04cd1" } aws.amazon.com/quickstart/architecture/vpc
13. Nested Stacks CloudFormation
14. Using Nested Stacks Resources: NestedStack: Type: AWS::CloudFormation::Stack Properties: TemplateURL: !Ref NestedStackS3File InstanceType: !Ref InstanceType Outputs: NestedStackBucketName: Value: !GetAtt NestedStack.InstanceName
15. docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/troubleshooting.html
16. Never, ever, should you actually use Nested Stacks because Nested Stacks are Stuck in UPDATE_COMPLETE_CLEANUP_IN_PROGRESS, UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS, or UPDATE_ROLLBACK_IN_PROGRESS A nested stack failed to roll back. Because of potential resource dependencies between nested stacks, AWS CloudFormation doesn't start cleaning up nested stack resources until all nested stacks have been updated or have rolled back. When a nested stack fails to roll back, AWS CloudFormation cancels all operations, regardless of the state that the other nested stacks are in. A nested stack that completed updating or rolling back but did not receive a signal from AWS CloudFormation to start cleaning up because another nested failed to roll back is in an UPDATE_COMPLETE_CLEANUP_IN_PROGRESS or UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS state. A nested stack that failed to update but did not receive a signal to start rolling back is in an UPDATE_ROLLBACK_IN_PROGRESS state. A nested stack might fail to roll back because of changes that were made outside of AWS CloudFormation, when the stack template doesn't accurately reflect the state of the stack. A nested stack might also fail if an Auto Scaling group in a nested stack had an insufficient resource signal timeout period when the group was created or updated. To fix the stack, contact AWS customer support. WTF? docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/troubleshooting.html
18. “no one knows for sure who erected these monoliths and why”
19. Exporting & Importing CloudFormation
20. Stack B: Resources: EC2Instance: Type: AWS::EC2::Instance Properties: NetworkInterfaces: - AssociatePublicIpAddress: true DeviceIndex: 0 GroupSet: - Fn::ImportValue: Global-SecurityGroup Stack A: Outputs: SecurityGroup: Value: !Ref SecurityGroup Description: ID of security group Export: Name: Global-SecurityGroup Exported Output and Imported Value
21. Exported Value Dependencies
22. Custom Resources CloudFormation
23. Custom Resources - Definition and usage Resources: CustomCWEventsLambda: Type: AWS::Lambda::Function Properties: Runtime: nodejs8.10 Handler: index.handler Role: !GetAtt LambdaExecutionRole.Arn Code: ZipFile: > const Response = require('cfn-response'); exports.handler = function handler(event, context) { Response.send(event, context, Response.FAILED, {}); throw new Error('Not Implemented'); } CloudWatchEventsRule: Type: Custom::Events::Rule Parameters: ServiceToken: !GetAtt CustomCWEventsLambda.Arn Name: 'STRING_VALUE' - A name for this rule. *optional* Description: 'STRING_VALUE' - A description of the rule. State: 'ENABLED' - Indicates whether the rule is enabled or disabled. from github.com/devops-israel/aws-cf-custom-events
24. Custom Resources - Implementation const Response = require('cfn-response'); const log = { info(...args) { console.log(...args) }, error(...args) { console.error(...args) }, } exports.handler = function handler(event, context) { log.info(event, context); switch (event.RequestType) { case 'Create': return createResource(event, context); case 'Delete': return deleteResource(event, context); case 'Update': return updateResource(event, context); default: log.error(new Error(`ERROR: Unknown event.RequestType provided: ${event.RequestType}`)); Response.send(event, context, Response.FAILED, {}); } } from github.com/devops-israel/aws-cf-custom-events
25. cfn-init CloudFormation
26. Resources: AutoScalingGroup: Type: AWS::AutoScaling::AutoScalingGroup Properties: LaunchConfigurationName: !Ref LaunchConfiguration CreationPolicy: ResourceSignal: Timeout: PT15M LaunchConfiguration: Type: AWS::AutoScaling::LaunchConfiguration Properties: UserData: Fn::Base64: !Sub | #cloud-config packages: - aws-cfn-bootstrap bootcmd: - "while ! ping -i 10 -I eth0 -n -w 60 -W 2 -c 1 -q s3.amazonaws.com; do sleep 1; done" runcmd: - /opt/aws/bin/cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource LaunchConfiguration - /opt/aws/bin/cfn-signal -e $? --region ${AWS::Region} --stack ${AWS::StackName} --resource AutoScalingGrou Updating EC2 Instances in an Auto Scaling Group
27. LaunchConfiguration: Type: AWS::AutoScaling::LaunchConfiguration Properties: ... Metadata: AWS::CloudFormation::Init: config: commands: ecs_task_sysctl1: command: "sysctl -w net.ipv4.conf.all.route_localnet=1" ecs_task_sysctl2: command: "sysctl -w net.ipv4.ip_local_port_range='1024 65000'" ecs_task_roles1: command: "iptables -t nat -A PREROUTING -p tcp -d 169.254.170.2 --dport 80 -j DNAT --to-destination 127.0.0.1:51679 ecs_task_roles2: command: "iptables -t nat -A OUTPUT -d 169.254.170.2 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 51679" save_iptables: command: "service iptables save" yum: ecs-banner: [] ecs-init: [] Updating EC2 Instances in an Auto Scaling Group
28. LaunchConfiguration: Type: AWS::AutoScaling::LaunchConfiguration Properties: ... Metadata: AWS::CloudFormation::Init: config: ... files: "/etc/sysctl.d/10-route_localnet.conf": mode: "0444" owner: root group: root content: "net.ipv4.conf.all.route_localnet = 1" "/etc/sysctl.d/10-more_ports.conf": mode: "0444" owner: root group: root content: "net.ipv4.ip_local_port_range = 1024 65000" Updating EC2 Instances in an Auto Scaling Group
29. "/etc/ecs/ecs.config": mode: 000644 owner: root group: root # http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-config.html # and https://github.com/aws/amazon-ecs-agent#environment-variables content: !Sub | # http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-config.html ECS_CLUSTER=${Cluster} ECS_ENGINE_AUTH_TYPE=${ECSEngineAuthType} ECS_AVAILABLE_LOGGING_DRIVERS=["json-file","awslogs","syslog","journald","gelf","fluentd","splunk"] ECS_ENABLE_TASK_IAM_ROLE=true ECS_DISABLE_PRIVILEGED=true ECS_INSTANCE_ATTRIBUTES={"env":"${EnvironmentName}"} ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION=10m # http://docs.aws.amazon.com/AmazonECS/latest/developerguide/automated_image_cleanup.html ECS_IMAGE_CLEANUP_INTERVAL=10m ECS_IMAGE_MINIMUM_CLEANUP_AGE=15m ECS_NUM_IMAGES_DELETE_PER_CYCLE=10 AWS_DEFAULT_REGION=${AWS::Region} Updating EC2 Instances in an Auto Scaling Group
30. "/etc/cfn/cfn-hup.conf": mode: 000400 owner: root group: root content: !Sub | [main] stack=${AWS::StackId} region=${AWS::Region} "/etc/cfn/hooks.d/cfn-auto-reloader.conf": content: !Sub | [cfn-auto-reloader-hook] triggers=post.update path=Resources.LaunchConfiguration.Metadata.AWS::CloudFormation::Init action=/opt/aws/bin/cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource LaunchCon Updating EC2 Instances in an Auto Scaling Group The cfn-hup helper is a daemon that detects changes in resource metadata and runs user-specified actions when a change is detected. This allows you to make configuration updates on your running Amazon EC2 instances through the UpdateStack API action.
31. Updating EC2 Instances in an Auto Scaling Group services: upstart: ecs: enabled: true ensureRunning: true files: - /etc/init/ecs.conf - /etc/ecs/ecs.config sysvinit: cfn-hup: enabled: true ensureRunning: true files: - /etc/cfn/cfn-hup.conf - /etc/cfn/hooks.d/cfn-auto-reloader.conf
33. CloudFormation is a tool for repeatable and predictable automation of cloud infrastructure provisioning
34. We invite you to join Operations Israel Facebook group on on.fb.me/Ops-IL Thank you! www.prodops.io