CI/CD on AWS

AWS Summit, London, May 8th, 2019

How can you accelerate the delivery of new, high-quality services? How can you be able to experiment and get feedback quickly from your customers? To get the most out of the agility afforded by serverless and containers, it is essential to build CI/CD pipelines that help teams iterate on code and quickly release features. In this talk, we demonstrate how developers can build effective CI/CD release workflows to manage their serverless or containerized deployments on AWS. We cover infrastructure-as-code (IaC) application models, such as AWS Serverless Application Model (AWS SAM) and new imperative IaC tools. We also demonstrate how to set up CI/CD release pipelines with AWS CodePipeline and AWS CodeBuild, and we show you how to automate safer deployments with AWS CodeDeploy.

Transcript

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T CI/CD on AWS Danilo Poccia Principal Evangelist, Serverless AWS @danilop Tonino Greco Head of Infrastructure and DevOps Dunelm @toninog
2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Listen Iterate Experiment Innovation Flywheel Experiments power the engine of rapid innovation
3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Pillars of releasing modern applications
4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Pillars of releasing modern applications Infrastructure as code
5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Infrastructure as code goals 1. Make infrastructure changes repeatable and predictable 2. Release infrastructure changes using the same tools as code changes 3. Replicate production environment in a staging environment to enable continuous testing
6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Release infrastructure-as-code with AWS CloudFormation “Master” branch Prepare template Create & execute change set Create & execute change set
7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Model function environments with AWS Serverless Application Model (SAM) • Open source framework for building serverless applications on AWS • Shorthand syntax to express functions, APIs, databases, and event source mappings • Transforms and expands SAM syntax into AWS CloudFormation syntax on deployment • Supports all AWS CloudFormation resource types https://aws.amazon.com/serverless/sam/
8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Model container environments with AWS Cloud Development Kit (CDK) • Open source framework to define cloud infrastructure in TypeScript, Java, C#, … • Provides library of higher-level resource types (“construct” classes) that have AWS best practices built in by default, packaged as npm modules • Provisions resources with CloudFormation • Supports all CloudFormation resource types AWS CDK https://awslabs.github.io/aws-cdk D eveloper Preview
9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T AWS Cloud Development Kit (CDK) npm install -g aws-cdk cdk init app --language typescript cdk synth cdk deploy cdk diff cdk destroy CodePipeline Use CloudFormation deployment actions with any synthesized CDK application Jenkins Use CDK CLI D eveloper Preview TypeScript C# F# Java Python …
10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T CDK template import ec2 = require('@aws-cdk/aws-ec2'); import ecs = require('@aws-cdk/aws-ecs'); import cdk = require('@aws-cdk/cdk'); class BonjourFargate extends cdk.Stack { constructor(parent: cdk.App, name: string, props?: cdk.StackProps) { super(parent, name, props); const vpc = new ec2.VpcNetwork(this, 'MyVpc', { maxAZs: 2 }); const cluster = new ecs.Cluster(this, 'Cluster', { vpc }); new ecs.LoadBalancedFargateService( this, "FargateService", { cluster, image: ecs.DockerHub.image("amazon/amazon-ecs-sample"), }); } } const app = new cdk.App(); new BonjourFargate(app, 'Bonjour'); app.run(); TypeScript
11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T import ec2 = require('@aws-cdk/aws-ec2'); import ecs = require('@aws-cdk/aws-ecs'); import cdk = require('@aws-cdk/cdk'); class BonjourFargate extends cdk.Stack { constructor(parent: cdk.App, name: string, props?: cdk.StackProps) { super(parent, name, props); const vpc = new ec2.VpcNetwork(this, 'MyVpc', { maxAZs: 2 }); const cluster = new ecs.Cluster(this, 'Cluster', { vpc }); new ecs.LoadBalancedFargateService( this, "FargateService", { cluster, image: ecs.DockerHub.image("amazon/amazon-ecs-sample"), }); } } const app = new cdk.App(); new BonjourFargate(app, 'Bonjour'); app.run(); CDK template TypeScript
12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T CDK template import ec2 = require('@aws-cdk/aws-ec2'); import ecs = require('@aws-cdk/aws-ecs'); import cdk = require('@aws-cdk/cdk'); class BonjourFargate extends cdk.Stack { constructor(parent: cdk.App, name: string, props?: cdk.StackProps) { super(parent, name, props); const vpc = new ec2.VpcNetwork(this, 'MyVpc', { maxAZs: 2 }); const cluster = new ecs.Cluster(this, 'Cluster', { vpc }); new ecs.LoadBalancedFargateService( this, "FargateService", { cluster, image: ecs.DockerHub.image("amazon/amazon-ecs-sample"), }); } } const app = new cdk.App(); new BonjourFargate(app, 'Bonjour'); app.run(); TypeScript
13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T CDK Lambda cron example export class LambdaCronStack extends cdk.Stack { constructor(app: cdk.App, id: string) { super(app, id); const lambdaFn = new lambda.Function(this, 'Singleton', { code: new lambda.InlineCode( fs.readFileSync('lambda-handler.py', { encoding: 'utf-8' })), handler: 'index.main’, timeout: 300, runtime: lambda.Runtime.Python37, }); const rule = new events.EventRule(this, 'Rule', { scheduleExpression: 'cron(0 18 ? * MON-FRI *)’, }); rule.addTarget(lambdaFn); } } Lambda function CloudWatch Events rule CloudFormation Stack Set the target TypeScript
14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Model pipelines with AWS CDK • Minimize copy-and-paste by using object-oriented language • Define microservice pipeline “shape” in one class, then re-use it across many pipelines • CDK includes many high-level constructs for modeling a CodePipeline pipeline, including automatically configuring IAM role policies
15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T CDK pipelines: Construct export class MyMicroservicePipeline extends cdk.Construct { constructor(parent: cdk.Construct, name: string, props: MyMicroservicePipelineProps) { super(parent, name); const pipeline = new codepipeline.Pipeline(this, 'Pipeline', { pipelineName: props.serviceName, }); const githubAccessToken = new cdk.SecretParameter(this, 'GitHubToken', { ssmParameter: 'GitHubToken' }); new codepipeline.GitHubSourceAction(this, 'GitHubSource', { stage: pipeline.addStage('Source'), owner: 'myorg', repo: props.serviceName, oauthToken: githubAccessToken.value }); …
16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T CDK pipelines: Stack import cdk = require('@aws-cdk/cdk'); import { MyMicroservicePipeline } from './pipeline'; class MyMicroservicePipelinesStack extends cdk.Stack { constructor(parent: cdk.App, name: string, props?: cdk.StackProps) { super(parent, name, props); new MyMicroservicePipeline(this, 'Pipeline1', { 'serviceName': 'Microservice1' }); new MyMicroservicePipeline(this, 'Pipeline2', { 'serviceName': 'Microservice2' }); new MyMicroservicePipeline(this, 'Pipeline3', { 'serviceName': 'Microservice3' }); new MyMicroservicePipeline(this, 'Pipeline4', { 'serviceName': 'Microservice4' }); } } const app = new cdk.App(); new MyMicroservicePipelinesStack(app, 'MyMicroservicePipelines'); app.run();
17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Pillars of releasing modern applications Infrastructure as code
18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Pillars of releasing modern applications
19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Pillars of releasing modern applications Continuous integration
20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Continuous integration goals Source Build Test Production
21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Continuous integration goals 1. Automatically kick off a new release when new code is checked in 2. Build and test code in a consistent, repeatable environment 3. Continually have an artifact ready for deployment 4. Continually close feedback loop when build fails
22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T AWS CodePipeline • Continuous delivery service for fast and reliable application updates • Model and visualize your software release process • Builds, tests, and deploys your code every time there is a code change • Integrates with third-party tools and AWS
23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T AWS CodePipeline: Supported sources Pick branch AWS CodeCommit GitHub Pick object or folder Amazon S3 Pick Docker tag Amazon ECR Automatically kick off release and pull latest source code
24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T AWS CodePipeline: ECR source action Source code: “master” branch ECR repository: “release” tag
25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T AWS CodePipeline: Supported triggers Automatically kick off release Amazon CloudWatch Events • Scheduled (nightly release) • AWS Health events (Fargate platform retirement) Available in CloudWatch Events console, API, SDK, CLI, and AWS CloudFormation Webhooks • DockerHub • Quay • Artifactory Available in CodePipeline API, SDK, CLI, and CloudFormation
26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T AWS CodeBuild • Fully managed build service that compiles source code, runs tests, and produces software packages • Scales continuously and processes multiple builds concurrently • No build servers to manage • Pay by the minute, only for the compute resources you use • Monitor builds through CloudWatch Events
27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T AWS CodeBuild • Each build runs in a new Docker container for a consistent, immutable environment • Docker and AWS CLI are installed in every official CodeBuild image • Provide custom build environments suited to your needs through the use of Docker images
28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T AWS CodeBuild: Lambda buildspec version: 0.2 phases: build: commands: - npm ci - npm test - > aws cloudformation package --template-file template.yaml --output-template packaged.yaml --s3-bucket $BUCKET artifacts: type: zip files: - packaged.yaml
29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T AWS CodeBuild: Lambda buildspec using SAM CLI version: 0.2 phases: install: commands: - pip install --upgrade awscli aws-sam-cli build: commands: - sam build - sam package --s3-bucket $BUCKET --output-template-file packaged.yaml artifacts: type: zip files: - packaged.yaml
30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T AWS CodeBuild: Docker buildspec version: 0.2 phases: build: commands: - $(aws ecr get-login --no-include-email) - docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG . - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $ECR_REPO:$IMAGE_TAG - docker push $ECR_REPO:$IMAGE_TAG
31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Pillars of releasing modern applications Continuous integration
32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Pillars of releasing modern applications
33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Pillars of releasing modern applications Continuous deployment
34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Continuous deployment goals Source Build Test Production
35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Continuous deployment goals 1. Automatically deploy new changes to staging environments for testing 2. Deploy to production safely without impacting customers 3. Deliver to customers faster: Increase deployment frequency, and reduce change lead time and change failure rate
36. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T AWS CodeDeploy • Automates code deployments to any instance and Lambda • Handles the complexity of updating your applications • Avoid downtime during application deployment • Roll back automatically if failure detected • Deploy to Amazon EC2, Lambda, ECS, or on- premises servers
37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T CodeDeploy – Lambda canary deployment API Gateway Lambda function weighted alias “live” v1 Lambda function code 100%
38. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T CodeDeploy – Lambda canary deployment API Gateway Lambda function weighted alias “live” v1 code 100% Run PreTraffic hook against v2 code before it receives traffic v2 code 0%
39. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T CodeDeploy – Lambda canary deployment API Gateway Lambda function weighted alias “live” v1 code 90% Wait for 10 minutes, roll back in case of alarm v2 code 10%
40. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T CodeDeploy – Lambda canary deployment API Gateway Lambda function weighted alias “live” v1 code 0% Run PostTraffic hook and complete deployment v2 code 100%
41. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T CodeDeploy – Lambda deployments in SAM templates Resources: GetFunction: Type: AWS::Serverless::Function Properties: AutoPublishAlias: live DeploymentPreference: Type: Canary10Percent10Minutes Alarms: - !Ref ErrorsAlarm - !Ref LatencyAlarm Hooks: PreTraffic: !Ref PreTrafficHookFunction PostTraffic: !Ref PostTrafficHookFunction Canary10Percent30Minutes Canary10Percent5Minutes Canary10Percent10Minutes Canary10Percent15Minutes Linear10PercentEvery10Minutes Linear10PercentEvery1Minute Linear10PercentEvery2Minutes Linear10PercentEvery3Minutes AllAtOnce
42. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T API Gateway canary stage API Gateway Production stage v1 code v2 code 99.5% 0.5% Canary stage
43. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T CodeDeploy-ECS blue-green deployments • Provisions “green” tasks, then flips traffic at the load balancer • Validation “hooks” enable testing at each stage of the deployment • Fast rollback to “blue” tasks in seconds if case of hook failure or CloudWatch alarms • Monitor deployment status and history via console, API, Amazon SNS notifications, and CloudWatch Events • Use “CodeDeploy-ECS” deploy action in CodePipeline or “aws ecs deploy” command in Jenkins
44. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T CodeDeploy-ECS appspec version: 1.0 Resources: - TargetService: Type: AWS::ECS::Service Properties: - TaskDefinition: "my_task_definition:8" LoadBalancerInfos: - ContainerName: "SampleApp" ContainerPort: 80 Hooks: - BeforeInstall: "LambdaFunctionToExecuteAnythingBeforeNewRevisionInstalltion" - AfterInstall: "LambdaFunctionToExecuteAnythingAfterNewRevisionInstallation" - AfterAllowTestTraffic: "LambdaFunctionToValidateAfterTestTrafficShift" - BeforeAllowTraffic: "LambdaFunctionToValidateBeforeTrafficShift" - AfterAllowTraffic: "LambdaFunctionToValidateAfterTrafficShift"
45. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T CodeDeploy-ECS blue-green deployment 100% Prod traffic
46. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T CodeDeploy-ECS blue-green deployment Target group 2 100% Prod traffic Test traffic listener (port 9000)
47. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T CodeDeploy-ECS blue-green deployment Green tasks: v2 code 100% Prod traffic Provision green tasks
48. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T CodeDeploy-ECS blue-green deployment 100% Prod traffic Run hook against test endpoint before green tasks receive prod traffic 0% Prod traffic
49. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T CodeDeploy-ECS blue-green deployment Flip traffic to green tasks, rollback in case of alarm 0% Prod traffic 100% Prod traffic
50. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T CodeDeploy-ECS blue-green deployment 100% Prod traffic Drain blue tasks
51. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T AWS Deployment Framework (ADF) https://github.com/awslabs/aws-deployment-framework
52. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Pillars of releasing modern applications Continuous deployment
53. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Pillars of releasing modern applications
54. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
56. CHALLENGES • Multiple independent tribes deploying to AWS • Complex Serverless deployments • Deploy rapid and often Our Answer • SAM CLI - Wrapped in Ansible • Part of a Pipeline library – functions for the developer • Jenkins Pipeline as code • Extensible, reproducible and simple • Slack integrations Dunelm Ltd
57. Git Commit Jenkins Webhook Automation Code Product Code Code Linting Prepare Worker Node Install Software Security Validation Build / compile Run Tests Deploy Install Software for building Vulnerability and Library scan Compile the software Run Unit and functional tests Deploy to AWS
58. THE BENEFITS ü Simple, reproducible and security conscious ü Scalable for performance – each pipeline has it’s own worker node (on Spot instances) ü Highly customisable for each tribe – self serve ü Centrally managed library of functions – easy to adapt to change ü Rapid onboarding of new developers – easy to use ü Rapid development of CI/CD pipelines ü Over 100 developers using the pipelines (across all Dunelm technology tribes) Dunelm Ltd
59. THE BENEFITS – THE STATS • Daily there are • >200 pipelines active • each deploying > 15 times / day • Having > 10 stages per pipeline • Productivity gains • > 95% of pipelines are successful • Allow tribes to be > 30% more effective (based on burnup charts and sprint reviews) • Allows us to test and security audit automatically Dunelm Ltd
60. ADDITIONAL BENEFITS • Slack integration with Jenkins • Status updates in Tribe channels • Kick off adhoc builds from Slack • Slack integration with AWS • Able to delete stacks (AWS CloudFormation) • Add parameters to AWS Systems Manager • Create Amazon Route53 DNS entries • AWS Lambda function to delete AWS CloudFormation stacks once they are removed from BitBucket Dunelm Ltd
61. Thank you! S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Danilo Poccia @danilop Tonino Greco @toninog