CI/CD for Modern Applications

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
O S L O
2 0 1 9 . 0 4 . 0 3
CI/CD for Modern Applications
Danilo Poccia
Principal Evangelist, Serverless
@danilop
M A D 1

View Slide

Development transformation at Amazon: 2001–2002
monolithic application
+ teams
2001
Lesson learned: decompose for agility
2002
microservices
+ 2 pizza teams

View Slide

Full ownership
Full accountability
“DevOps”
Focused innovation
Two-pizza teams

View Slide

Monolith development lifecycle
monitor
release
test
build
developers
delivery pipelines
services

View Slide

Microservice development lifecycle
???
developers
delivery pipelines
services

View Slide

Microservice development lifecycle
developers services
monitor
release
test
build
delivery pipelines
monitor
release
test
build
monitor
release
test
build
monitor
release
test
build
monitor
release
test
build
monitor
release
test
build

View Slide

Listen
Iterate
Experiment
Innovation
Flywheel
Experiments power the engine of rapid innovation

View Slide

Approaches to modern application development
• Simplify environment management
• Reduce the impact of code changes
• Automate operations
• Accelerate the delivery of new, high-quality services
• Gain insight across resources and applications
• Protect customers and the business

View Slide

• Simplify environment management with serverless technologies
• Reduce the impact of code changes with microservice architectures
• Automate operations by modeling applications & infrastructure as code
• Accelerate the delivery of new, high-quality services with CI/CD
• Gain insight across resources and applications by enabling observability
• Protect customers and the business with end-to-end security & compliance

View Slide

What is serverless?
No infrastructure provisioning,
no management
Automatic scaling
Pay for value Highly available and secure

View Slide

Serverless containers
Long-running
Abstracts the OS
Fully managed orchestration
Fully managed cluster scaling
Serverless functions
Event-driven
Many language runtimes
Data source integrations
No server management

View Slide

Comparison of operational responsibility
AWS Lambda
Serverless functions
AWS Fargate
Serverless containers
ECS/EKS
Container-management as a service
EC2
Infrastructure-as-a-Service
More opinionated
Less opinionated
AWS manages Customer manages
• Data source integrations
• Physical hardware, software, networking,
and facilities
• Provisioning
• Application code
• Container orchestration, provisioning
• Cluster scaling
• Physical hardware, host OS/kernel,
networking, and facilities
• Security config and updates, network config,
management tasks
• Container orchestration control plane
• Physical hardware software,
• Work clusters
firewall, management tasks
• Physical hardware software,
• Scaling
management tasks
• Provisioning, managing scaling and
patching of servers

View Slide

• Simplify environment management with serverless technologies
• Reduce the impact of code changes with microservice architectures
• Automate operations by modeling applications & infrastructure as code
• Accelerate the delivery of new, high-quality services with CI/CD
• Gain insight across resources and applications by enabling observability
• Protect customers and the business with end-to-end security & compliance

View Slide

Release process stages
Source Build Test Production

View Slide

Pillars of releasing modern applications

View Slide

Infrastructure
as code

View Slide

Infrastructure as code
Declarative
I tell you
what I need
I tell you
what to do
Imperative

View Slide

Infrastructure as code goals
1. Make infrastructure changes repeatable and predictable
2. Release infrastructure changes using the same tools as code changes
3. Replicate production environment in a staging environment to enable
continuous testing

View Slide

Release infrastructure-as-code
“Master”
branch
Prepare
template
Create & execute
change set
Create & execute
change set

View Slide

Model function environments with AWS
Serverless Application Model (SAM)
• Open source framework for building serverless
applications on AWS
• Shorthand syntax to express functions, APIs,
databases, and event source mappings
• Transforms and expands SAM syntax into AWS
CloudFormation syntax on deployment
• Supports all AWS CloudFormation resource types
https://aws.amazon.com/serverless/sam
O
pen
Source

View Slide

SAM template
AWSTemplateFormatVersion: '2010-09-09’
Transform: AWS::Serverless-2016-10-31
Resources:
GetFunction:
Type: AWS::Serverless::Function
Properties:
Handler: index.get
Runtime: nodejs8.10
CodeUri: src/
Policies:
- DynamoDBReadPolicy:
TableName: !Ref MyTable
Events:
GetResource:
Type: Api
Properties:
Path: /resource/{resourceId}
Method: get
MyTable:
Type: AWS::Serverless::SimpleTable
Just 20 lines to create:
• Lambda function
• IAM role
• API Gateway
• DynamoDB table

View Slide

Use SAM CLI to package and deploy SAM templates
sam init --name my-function --runtime python
cd my-function/
sam validate
sam local generate-event/invoke/start-api/start-lambda
sam build # Depending on the runtime
sam package --s3-bucket my-packages-bucket \
--output-template-file packaged.yaml
sam deploy --template-file packaged.yaml \
--stack-name my-stack-prod
sam logs -n MyFunction --stack-name my-stack-prod -t
sam publish # To the Serverless Application Repository
O
pen
Source

View Slide

TweetSource:
Type: AWS::Serverless::Application
Properties:
Location:
ApplicationId: arn:aws:serverlessrepo:...
SemanticVersion: 2.0.0
Parameters:
TweetProcessorFunctionName: !Ref MyFunction
SearchText: '#serverless -filter:nativeretweets'
Nested apps to simplify solving recurring problems
Standard
Component
Custom
Business
Logic
Polling schedule
(CloudWatch
Events rule)
trigger
TwitterProcessor
SearchCheckpoint
TwitterSearchPoller
Twitter
Search API

View Slide

Model container environments with AWS
Cloud Development Kit (CDK)
Developer
Preview
• Open source framework to define cloud
infrastructure in JavaScript, TypeScript, Java, C#,
Python, …
• Provides library of higher-level resource types
(“construct” classes) that have AWS best practices
built in by default, packaged as npm modules
• Provisions resources with CloudFormation
• Supports all CloudFormation resource types
AWS
CDK
https://awslabs.github.io/aws-cdk
O
pen
Source

View Slide

import cdk = require('@aws-cdk/cdk');
import ec2 = require('@aws-cdk/aws-ec2');
import ecs = require('@aws-cdk/aws-ecs');
class BonjourFargate extends cdk.Stack {
constructor(parent: cdk.App, name: string, props?: cdk.StackProps) {
super(parent, name, props);
const vpc = new ec2.VpcNetwork(this, 'MyVpc', { maxAZs: 2 });
const cluster = new ecs.Cluster(this, 'Cluster', { vpc });
new ecs.LoadBalancedFargateService(
this, "FargateService", {
cluster,
image: ecs.DockerHub.image("amazon/amazon-ecs-sample"),
});
}
}
const app = new cdk.App();
new BonjourFargate(app, 'Bonjour');
app.run();
CDK template

View Slide

CDK template
import ec2 = require('@aws-cdk/aws-ec2');
import ecs = require('@aws-cdk/aws-ecs');
class BonjourFargate extends cdk.Stack {
const vpc = new ec2.VpcNetwork(this, 'MyVpc', { maxAZs: 2 });
const cluster = new ecs.Cluster(this, 'Cluster', { vpc });
new ecs.LoadBalancedFargateService(
this, "FargateService", {
cluster,
image: ecs.DockerHub.image("amazon/amazon-ecs-sample"),
});
}
}
new BonjourFargate(app, 'Bonjour');
app.run();

View Slide

Model pipelines with AWS CDK
• Minimize copy-and-paste by using object-oriented language
• Define microservice pipeline “shape” in one class, then re-use it across
many pipelines
• CDK includes many high-level constructs for modeling a CodePipeline
pipeline, including automatically configuring IAM role policies

View Slide

CDK pipelines: Construct
export class MyMicroservicePipeline extends cdk.Construct {
constructor(parent: cdk.Construct, name: string, props:
MyMicroservicePipelineProps) {
super(parent, name);
const pipeline = new codepipeline.Pipeline(this, 'Pipeline', {
pipelineName: props.serviceName,
});
const githubAccessToken = new cdk.SecretParameter(this, 'GitHubToken’,
{ ssmParameter: 'GitHubToken' });
new codepipeline.GitHubSourceAction(this, 'GitHubSource', {
stage: pipeline.addStage('Source’),
owner: 'myorg’,
repo: props.serviceName,
oauthToken: githubAccessToken.value
});
…

View Slide

import { MyMicroservicePipeline } from './pipeline';
class MyMicroservicePipelinesStack extends cdk.Stack {
new MyMicroservicePipeline(this, 'Pipeline1', { 'serviceName': 'Microservice1' });
}
}
new MyMicroservicePipelinesStack(app, 'MyMicroservicePipelines');
app.run();
CDK pipelines: Stack

View Slide

Use CDK CLI to synthesize and deploy CDK templates
npm install -g aws-cdk
cdk init app --language typescript
cdk synth
cdk deploy
cdk diff
cdk destroy

View Slide

Infrastructure
as code

View Slide


View Slide

Continuous
integration

View Slide

Continuous integration goals

View Slide

1. Automatically kick off a new release when new code is checked in
2. Build and test code in a consistent, repeatable environment
3. Continually have an artifact ready for deployment
4. Continually close feedback loop when build fails

View Slide

AWS CodePipeline
• Continuous delivery service for fast and reliable
application updates
• Model and visualize your software release process
• Builds, tests, and deploys your code every time
there is a code change
• Integrates with third-party tools and AWS

View Slide

AWS CodePipeline: Supported sources
Pick branch
AWS CodeCommit
GitHub
Pick object or folder
Amazon S3
Pick Docker tag
Amazon ECR
Automatically kick off release and pull latest source code

View Slide

AWS CodePipeline: ECR source action
Source code:
“master” branch
ECR repository:
“release” tag

View Slide

AWS CodePipeline: Supported triggers
Automatically kick off release
Amazon CloudWatch Events
• Scheduled (nightly release)
• AWS Health events (Fargate
platform retirement)
Available in CloudWatch Events
console, API, SDK, CLI, and AWS
CloudFormation
Webhooks
• DockerHub
• Quay
• Artifactory
Available in CodePipeline API,
SDK, CLI, and CloudFormation

View Slide

AWS CodeBuild
• Fully managed build service that compiles source
code, runs tests, and produces software packages
• Scales continuously and processes multiple builds
concurrently
• No build servers to manage
• Pay by the minute, only for the compute
resources you use
• Monitor builds through CloudWatch Events

View Slide

AWS CodeBuild
• Each build runs in a new Docker container for a
consistent, immutable environment
• Docker and AWS CLI are installed in every official
CodeBuild image
• Provide custom build environments suited to
your needs through the use of Docker images

View Slide

AWS CodeBuild: Lambda buildspec
version: 0.2
phases:
build:
commands:
- npm ci
- npm test
- >
aws cloudformation package
--template-file template.yaml
--output-template packaged.yaml
--s3-bucket $BUCKET
artifacts:
type: zip
files:
- packaged.yaml

View Slide

AWS CodeBuild: Lambda buildspec using SAM CLI
version: 0.2
phases:
install:
commands:
- pip install --upgrade awscli aws-sam-cli
build:
commands:
- sam build
- sam package --s3-bucket $BUCKET --output-template-file packaged.yaml
artifacts:
type: zip
files:
- packaged.yaml

View Slide

AWS CodeBuild: Docker buildspec
version: 0.2
phases:
build:
commands:
- $(aws ecr get-login --no-include-email)
- docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .
- docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $ECR_REPO:$IMAGE_TAG
- docker push $ECR_REPO:$IMAGE_TAG

View Slide

1. Automatically kick off a new release when new code is checked in
2. Build and test code in a consistent, repeatable environment
3. Continually have an artifact ready for deployment
4. Continually close feedback loop when build fails

View Slide

Continuous
integration

View Slide


View Slide

Continuous
deployment

View Slide

Continuous deployment goals

View Slide

1. Automatically deploy new changes to staging environments for testing
2. Deploy to production safely without impacting customers
3. Deliver to customers faster: Increase deployment frequency,
and reduce change lead time and change failure rate

View Slide

AWS CodeDeploy
• Automates code deployments to any instance
and Lambda
• Handles the complexity of updating your
applications
• Avoid downtime during application deployment
• Roll back automatically if failure detected
• Deploy to Amazon EC2, containers, Lambda, or
on-premises servers

View Slide

CodeDeploy – Lambda deployments
Enable in your serverless application template
Resources:
GetFunction:
Type: AWS::Serverless::Function
Properties:
DeploymentPreference:
Type: Canary10Percent10Minutes
Alarms:
- !Ref ErrorsAlarm
- !Ref LatencyAlarm
Hooks:
PreTraffic: !Ref PreTrafficHookFunction
PostTraffic: !Ref PostTrafficHookFunction
Canary10Percent30Minutes
Linear10PercentEvery10Minutes
Linear10PercentEvery1Minute
AllAtOnce

View Slide

CodeDeploy – Lambda canary deployment
API
Gateway
Lambda
function
weighted
alias “live”
v1 Lambda
function
code
100%

View Slide

API
Gateway
Lambda
function
weighted
alias “live”
v1 code
100%
Run PreTraffic hook against v2 code before it receives traffic
v2 code
0%

View Slide

API
Gateway
Lambda
function
weighted
alias “live”
v1 code
90%
Wait for 10 minutes, roll back in case of alarm
v2 code
10%

View Slide

API
Gateway
Lambda
function
weighted
alias “live”
v1 code
0%
Run PostTraffic hook and complete deployment
v2 code
100%

View Slide

API Gateway canary stage
API
Gateway
Production
stage
v1 code
v2 code
99.5%
0.5%
Canary
stage

View Slide

CodeDeploy-ECS blue-green deployments
• Provisions “green” tasks, then flips traffic at the
load balancer
• Validation “hooks” enable testing at each stage of
the deployment
• Fast rollback to “blue” tasks in seconds if case of
hook failure or CloudWatch alarms
• Monitor deployment status and history via
console, API, Amazon SNS notifications, and
CloudWatch Events
• Use “CodeDeploy-ECS” deploy action in
CodePipeline or “aws ecs deploy” command in
Jenkins

View Slide

CodeDeploy-ECS appspec
version: 1.0
Resources:
- TargetService:
Type: AWS::ECS::Service
Properties:
- TaskDefinition: "my_task_definition:8"
LoadBalancerInfos:
- ContainerName: "SampleApp"
ContainerPort: 80
Hooks:
- BeforeInstall: "LambdaFunctionToExecuteAnythingBeforeNewRevisionInstalltion"
- AfterInstall: "LambdaFunctionToExecuteAnythingAfterNewRevisionInstallation"
- AfterAllowTestTraffic: "LambdaFunctionToValidateAfterTestTrafficShift"
- BeforeAllowTraffic: "LambdaFunctionToValidateBeforeTrafficShift"
- AfterAllowTraffic: "LambdaFunctionToValidateAfterTrafficShift"

View Slide

CodeDeploy-ECS blue-green deployment
100%
Prod
traffic

View Slide

Target
group 2
100%
Prod
traffic
Test traffic listener
(port 9000)

View Slide

Green tasks:
v2 code
100%
Prod
traffic
Provision green tasks

View Slide

100%
Test
traffic
100%
Prod
traffic
Run hook against test endpoint before green tasks receive prod traffic

View Slide

100%
Prod
traffic
Flip traffic to green tasks, rollback in case of alarm
0%
Prod
traffic

View Slide

100%
Prod
traffic
Drain blue tasks

View Slide

Container image tagging for deployments
• Docker tags are resolved when each container starts, not just during
deployments
• Deploying “latest” or “prod” can result in untested code in production
after a scale-out event
• Use unique “immutable” tags for deployments

View Slide


View Slide

Build pushes new “latest” image
Image: [email protected] (“latest”)

View Slide

Service scales up, launching new tasks
Image: [email protected] (“latest”)

View Slide

Deploy using immutable tags
{
"name": "sample-app",
"image": "amazon/amazon-ecs-
[email protected]:3e39d933b1d948c92309bb583b5a1f3d28f0119e1551ca1fe538ba414a41af48d"
}
{
"name": "sample-app",
"image": "amazon/amazon-ecs-sample:build-b2085490-359f-4eaf-8970-6d1e26c354f0"
}
SHA256 Digest
Build ID

View Slide

Compute immutable tags during build
SHA256 Digest
export IMAGE_URI=`docker inspect --format='{{index .RepoDigests 0}}' my_image:$IMAGE_TAG`
Example Result:
amazon/[email protected]:3e39d933b...
Build ID
export IMAGE_TAG=build-`echo $CODEBUILD_BUILD_ID | awk –F":" ‘{print $2}'`
Example Result:
build-b2085490-359f-4eaf-8970-6d1e26c354f0

View Slide


View Slide

Build pushes new image tagged with new build ID
Image: [email protected] (“build-22222”)

View Slide

Service scales up, launching new tasks

View Slide

Image: “build-22222” tag
Deployment updates service’s task definition, replacing tasks

View Slide

1. Automatically deploy new changes to staging environments for testing
2. Deploy to production safely without impacting customers
3. Deliver to customers faster: Increase deployment frequency,
and reduce change lead time and change failure rate

View Slide

Continuous
deployment

View Slide


View Slide

Capital One – Credit Offers API serverless architecture
Affiliates
www.capitalone.com/
credit-cards/prequalify
AWS Cloud
Capital One
API Gateway
VPC
Lambda
Function
Traces Logs
Production Support
Command Center
COAT
Credit Offers API Team
Lambda
Function
S3 Bucket
TTL
Third-Party
API

View Slide

Capital One – Credit Offers API CI/CD pipeline
Continuous Improvement, Continuous Delivery!
GitHub LGTM Bot Jenkins AWS SAM
S3 Bucket
(Versioning)
Lambda
Function
DeploymentType:
dev: AllAtOnce
qa: AllAtOnce
qaw: AllAtOnce
prod: Canary10Percent10Minutes
prodw: Canary10Percent10Minutes
canary5xxGetProductsAlarm:
Type: AWS::CloudFormation::Alarm
Properties:
AlarmActions:
- !FindInMap:
- params
- AdminSNSTopic
- !Ref Environment
AlarmDescription: 500 error from product
listing Lambda.
ComparisonOperator:
GreatherThanOrEqualTothreshold
Period: 300
Statistic: Sum
Threshold: 1
EvaluationPeriod: 1

View Slide

Capital One – Benefits from taking the API serverless
Performance gains
From the time the request
is received by lambda to
the time to send the
response back
70%
Cost savings
By removing EC2, ELB and
RDS from our solution
90%
Increase in team velocity
Reduce investment in team’s time
on DevOps and dedicate back to
feature development!
30%

View Slide


View Slide

Demo – Store & Reply
AWS Cloud
Region
https://github.com/danilop/store-and-reply

View Slide

Takeaways
1. Manage your infrastructure as code
2. Frequently build and integrate your code to get a first feedback
3. Continuously release in production using canary releases with
monitoring and automated rollbacks
4. Use canary releases to get both technical and business feedback

View Slide

Thank you!
Danilo Poccia
@danilop

View Slide

CI/CD for Modern Applications

CI/CD for Modern Applications

Danilo Poccia

More Decks by Danilo Poccia

Other Decks in Programming

Featured

Transcript