The Power of IaC - AWS CloudFormation

This presentation shows how to use the concept of Infrastructure as a Code and AWS CloudFormation to model business problems using serverless components.

Transcript

1. The Power of IaC
2. Michał Górski Cloud Architect Northmill AB After the work: • Football • Fantasy Books • Cats • Tea • Traveling michal.gorski.pl@gmail.com michal-gorski
3. Agenda 1. What is the infrastructure 2. Serverless architecture 3. Infrastructure as a Code
4. Infrastructure
5. Invoicing process
6. Invoicing process
7. Invoicing process • Easy deployment • Slow process - low scalability • Server can be overwhelmed • Infrastructure needs to be managed to achieve high availability
8. Serverless FaaS
9. Serverless FaaS
10. Serverless BaaS
11. Serverless BaaS
12. Serverless BaaS
13. Serverless BaaS
14. Serverless BaaS
15. Serverless BaaS
16. Serverless BaaS
17. Serverless BaaS
18. Serverless BaaS
19. Serverless BaaS
20. Serverless • No server management • Flexible scaling • Pay for value • Automated high availability
21. Invoicing process
22. Invoicing process
23. Invoicing process
24. Invoicing process
25. Serverless architecture by hand
27. Serverless architecture by hand • How to deploy changes? • How to maintain more environments? • Can we track changes? • Can we automate component testing? • How can we check if the infrastructure is properly implemented? • How can we check security part of the infrastructure?
28. Infrastructure as a code - CloudFormation AWSTemplateFormatVersion: 2010-09-09 Transform: 'AWS::Serverless-2016-10-31' Description: Parameters: Mappings: Conditions: Resources: Outputs:
29. Invoicing process
30. Infrastructure as a code - Resources FileBucket: Type: AWS::S3::Bucket
31. Infrastructure as a code - Resources FileBucket: Type: AWS::S3::Bucket Properties: LifecycleConfiguration: Rules: - ExpirationInDays: 1 Status: Enabled Prefix: to-process/
32. Infrastructure as a code - Resources
33. Infrastructure as a code - Resources ParseFileLambda: Type: AWS::Serverless::Function Properties: Handler: Invoicing::Invoicing.Functions.ParseFileLambda::ExecuteAsync CodeUri: bin/Release/netcoreapp2.1/publish Runtime: dotnetcore2.1 MemorySize: 128 Timeout: 180 Environment: Variables: InvoiceQueueUrl: !Ref InvoiceQueue FileBucketName: !Sub file-bucket-${AWS::StackName}
34. Infrastructure as a code - Resources
35. Infrastructure as a code - Resources FileBucket: Type: AWS::S3::Bucket Properties: NotificationConfiguration: LambdaConfigurations: - Event: s3:ObjectCreated:* Filter: S3Key: Rules: - Name: prefix Value: to-process/ Function: !GetAtt ParseFileLambda.Arn
36. Function !GetAtt !GetAtt ParseFileLambda.Arn !GetAtt InvoiceQueue.QueueName !GetAtt MyELB.DNSName CheatSheet
37. Infrastructure as a code - Resources
38. Infrastructure as a code - Resources InvoiceQueue: Type: AWS::SQS::Queue Properties: VisibilityTimeout: 180
39. Infrastructure as a code - Resources
40. Infrastructure as a code - Resources HandleInvoiceLambda: Type: AWS::Serverless::Function Properties: Handler: Invoicing::Invoicing.Functions.HandleInvoiceLambda::ExecuteAsync CodeUri: bin/Release/netcoreapp2.1/publish Runtime: dotnetcore2.1 MemorySize: 128 Timeout: 180 Events: InvoiceQueueEvent: Type: SQS Properties: BatchSize: 1 Queue: !GetAtt InvoiceQueue.Arn
41. Infrastructure as a code - Resources
42. Infrastructure as a code - conclusions • Infrastructure in code repository ◦ Change tracking available ◦ Code review possible • Very friendly CI/CD process ◦ Deploy environment ◦ Automatic Component tests ◦ Automatic template validation ◦ Automatic security checks • Why every developer should know the infrastructure?
43. Infrastructure as a code - conclusions • Infrastructure in code repository ◦ Change tracking available ◦ Code review possible • Very friendly CI/CD process ◦ Deploy environment ◦ Automatic Component tests ◦ Automatic template validation ◦ Automatic security checks • Why every developer should know the infrastructure?
44. Invoicing process
45. Infrastructure as a code - Resources KMSKey: Type: AWS::KMS::Key Properties: Description: Invoicing KMS Key KeyPolicy: Version: 2012-10-17 Id: !Sub ${AWS::StackName}-kms-key Statement: - Sid: Allow all Effect: Allow Principal: AWS: !Sub arn:aws:iam::${AWS::AccountId}:root Action: - kms:* Resource: '*'
46. Function !Sub !Sub arn:aws:s3:::file-bucket-${AWS::StackName}/invoices/* !Sub ${FileBucket.Arn}/to-process/* !Sub arn:aws:iam::${AWS::AccountId}:root
47. Infrastructure as a code - Resources FileBucket: Type: AWS::S3::Bucket Properties: BucketEncryption: ServerSideEncryptionConfiguration: - ServerSideEncryptionByDefault: SSEAlgorithm: aws:kms KMSMasterKeyID: !Ref KMSKey
48. Function !Ref CheatSheet !Ref FileBucket file-bucket-invoicing-dev !Ref InvoiceQueue https://sqs.eu-west-1.amazonaws.com/111111111111/inv oicing-dev-InvoiceQueue-1CM6D9R2XVZQS !Ref KMSKey arn:aws:kms:eu-west-1:111111111111:key/0697add9-b385 -4393-b714-ae7b56282f57
49. More Functions !Join !Join [ ":", [ a, b, c ] ] a:b:c !Split !Split [ "|" , "a||c|" ] ["a", "", "c", ""] !Select !Select [ "1", [ "google", "aws", "azure"] ] aws
50. Invoicing process
51. Infrastructure as a code - Resources FileBucketPermission: Type: AWS::Lambda::Permission Properties: Action: lambda:InvokeFunction FunctionName: !GetAtt ParseFileLambda.Arn Principal: s3.amazonaws.com SourceArn: !GetAtt FileBucket.Arn
52. Invoicing process
53. Circular dependency problem
54. Circular dependency problem FileBucket: Type: AWS::S3::Bucket Properties: NotificationConfiguration: LambdaConfigurations: - Event: s3:ObjectCreated:* Filter: S3Key: Rules: - Name: prefix Value: to-process/ Function: !GetAtt ParseFileLambda.Arn
55. Circular dependency problem FileBucketPermission: Type: AWS::Lambda::Permission Properties: Action: lambda:InvokeFunction FunctionName: !GetAtt ParseFileLambda.Arn Principal: s3.amazonaws.com SourceArn: !GetAtt FileBucket.Arn
56. Circular dependency problem FileBucket: Type: AWS::S3::Bucket Properties: BucketName: !Sub file-bucket-${AWS::StackName}
57. Circular dependency problem FileBucketPermission: Type: AWS::Lambda::Permission Properties: Action: lambda:InvokeFunction FunctionName: !GetAtt ParseFileLambda.Arn Principal: s3.amazonaws.com SourceArn: !Sub arn:aws:s3:::file-bucket-${AWS::StackName}
58. Invoicing process
59. Infrastructure as a code - Resources ParseFileLambdaRole: Type: AWS::IAM::Role Properties: Policies: - PolicyName: allowGetObjectFromS3 PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - s3:GetObject Resource: !Sub arn:aws:s3:::file-bucket-${AWS::StackName}/to-process/*
60. Infrastructure as a code - Resources ParseFileLambdaRole: Type: AWS::IAM::Role Properties: Policies: - PolicyName: allowDecryptFiles PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - kms:Decrypt Resource: !GetAtt KMSKey.Arn
61. Infrastructure as a code - Resources ParseFileLambdaRole: Type: AWS::IAM::Role Properties: ManagedPolicyArns: - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - sts:AssumeRole Principal: Service: - lambda.amazonaws.com
62. Infrastructure as a code - Resources ParseFileLambda: Type: AWS::Serverless::Function Properties: ... Role: !GetAtt ParseFileLambdaRole.Arn ...
63. Invoicing process
64. Infrastructure as a code - Resources HandleInvoiceLambdaRole: Type: AWS::IAM::Role Properties: Policies: - PolicyName: allowPutObjectToS3 PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - s3:PutObject Resource: !Sub arn:aws:s3:::file-bucket-${AWS::StackName}/invoices/*
65. Infrastructure as a code - Resources HandleInvoiceLambdaRole: Type: AWS::IAM::Role Properties: Policies: - PolicyName: allowToUseSqs PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - sqs:ReceiveMessage - sqs:DeleteMessage - sqs:GetQueueAttributes Resource: !GetAtt InvoiceQueue.Arn
66. Infrastructure as a code - Resources HandleInvoiceLambda: Type: AWS::Serverless::Function Properties: ... Role: !GetAtt HandleInvoiceLambdaRole.Arn ...
67. Invoicing process
68. Infrastructure as a code - Resources InvoiceDeadLetterQueue: Type: AWS::SQS::Queue Properties: VisibilityTimeout: 300
69. Infrastructure as a code - Resources InvoiceQueue: Type: AWS::SQS::Queue Properties: VisibilityTimeout: 180 RedrivePolicy: deadLetterTargetArn: !GetAtt InvoiceDeadLetterQueue.Arn maxReceiveCount: 3
70. Invoicing process
71. Infrastructure as a code - Outputs Outputs: PutObjectsInBucketPolicy: Value: !Ref PutObjectsInBucketPolicy Export: Name: !Sub PutObjectsInFileBucketPolicy-${AWS::StackName}
72. Infrastructure as a code - Outputs PutObjectsInBucketPolicy: Type: AWS::IAM::ManagedPolicy Properties: Description: Policy for uploading files to file bucket PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: s3:PutObject Resource: !Sub ${FileBucket.Arn}/to-process/* - Effect: Allow Action: kms:GenerateDataKey* Resource: !GetAtt KMSKey.Arn
73. Invoicing process
74. Serverless architecture
75. Serverless architecture
76. Serverless architecture
78. Infrastructure as a code - AWS CLI dotnet lambda deploy-serverless -cfg deploy-config-dev.json
79. Infrastructure as a code - AWS CLI dotnet lambda deploy-serverless -cfg deploy-config-dev.json { "profile": "Default", "region": "eu-west-1", "configuration": "Release", "framework": "netcoreapp2.1", "s3-bucket": "sandbox-deploy-bucket", "s3-prefix": "Invoicing/", "template": "serverless.yaml", "stack-name": "invoicing-dev", "stack-wait": true }
80. Infrastructure as a code - conclusions • Infrastructure in code repository ◦ Change tracking available ◦ Code review possible • Very friendly CI/CD process ◦ Deploy environment ◦ Automatic Component tests - on demand environment ◦ Automatic template validation ◦ Automatic security checks • Why every developer should know the infrastructure?
81. Infrastructure as a code - AWS CLI dotnet lambda deploy-serverless -cfg deploy-config-dev.json { "profile": "Default", "region": "eu-west-1", "configuration": "Release", "framework": "netcoreapp2.1", "s3-bucket": "sandbox-deploy-bucket", "s3-prefix": "Invoicing/", "template": "serverless.yaml", "stack-name": "invoicing-dev", "stack-wait": true }
82. Infrastructure as a code - Resources KMSKey: Type: AWS::KMS::Key Properties: Description: Invoicing KMS Key KeyPolicy: Version: 2012-10-17 Id: !Sub ${AWS::StackName}-kms-key Statement: - Sid: Allow all Effect: Allow Principal: AWS: !Sub arn:aws:iam::${AWS::AccountId}:root Action: - kms:* Resource: '*'
83. Infrastructure as a code - drift
84. Infrastructure as a code - drift
85. Thank you!! Q&A