Save 37% off PRO during our Black Friday Sale! »

Building a CI/CD pipeline on AWS

Building a CI/CD pipeline on AWS

An overview of why and how to build a CI/CD pipeline using the Code-suite AWS tools

5f27d0b43fc28785db65c57ae7e81ec5?s=128

Milan Cermak

April 09, 2019
Tweet

Transcript

  1. BUILDING A CI/CD PIPELINE ON AWS https://milancermak.com @milancermak Photo by

    Robert Bock on Unsplash
  2. WHAT TO BUILD Holy grail A git-push-to-production pipeline, with automated

    rollbacks, with separated environments, with separated accounts, distributed in multiple regions Start simple Automate deployments for EVERYTHING Photo by Kobby Mendez on Unsplash
  3. WHY BOTHER? Speed Focus Peace of mind GOOD FOR BUSINESS

    Photo by Taha Mazandarani on Unsplash
  4. TOOLS OF THE TRADE CloudFormation CodeCommit & GitHub CodeBuild CodeDeploy

    CodePipeline Photo by Rick Mason on Unsplash
  5. CODEPIPELINE

  6. None
  7. ACTION CONFIGURATION Action configuration depends on the action category: source,

    build, test, deploy, approval, invoke
  8. CODEPIPELINE ARTIFACTS Artifact is a fancy word for a zip

    file Code from the repository Output of CodeBuild … Accessible to actions Stored in an S3 bucket
  9. None
  10. AN EXAMPLE Photo by Victor Garcia on Unsplash

  11. PULL CODE FROM REPO

  12. Name: FetchSource Actions: - Name: CodeCommit ActionTypeId: Category: Source Owner:

    AWS Provider: CodeCommit Version: '1' Configuration: RepositoryName: !GetAtt CodeRepository.Name BranchName: master OutputArtifacts: - Name: SourceOutput
  13. MAKE THE PIPELINE UPDATE ITSELF

  14. Name: UpdatePipeline Actions: - Name: PipelineStack ActionTypeId: Category: Deploy Owner:

    AWS Provider: CloudFormation Version: '1' Configuration: ActionMode: CREATE_UPDATE Capabilities: CAPABILITY_IAM RoleArn: !GetAtt CloudformationRole.Arn StackName: !Ref AWS::StackName TemplatePath: SourceOutput::infrastructure/ pipeline.yml ParameterOverrides: !Sub '{"Service": "$ {Service}"}' InputArtifacts: - Name: SourceOutput
  15. PREPARE THE DEPLOYMENT PACKAGE

  16. Name: Build Actions: - Name: Functions ActionTypeId: Category: Build Owner:

    AWS Provider: CodeBuild Version: '1' Configuration: ProjectName: !Ref CodeBuildProject InputArtifacts: - Name: SourceOutput OutputArtifacts: - Name: BuildOutput
  17. CODEBUILD PROJECT DECLARATION

  18. CodeBuildProject: Type: AWS::CodeBuild::Project Properties: Artifacts: Type: CODEPIPELINE Cache: Location: !Sub

    '${ArtifactsBucket}/buildcache/ functions' Type: S3 Environment: ComputeType: BUILD_GENERAL1_SMALL EnvironmentVariables: - Name: ARTIFACTS_BUCKET Value: !Ref ArtifactsBucket Type: PLAINTEXT Image: aws/codebuild/python:3.7.1 Type: LINUX_CONTAINER ServiceRole: !GetAtt CodeBuildRole.Arn Source: BuildSpec: infrastructure/buildspec.yml Type: CODEPIPELINE TimeoutInMinutes: 15
  19. CODEBUILD TASK DECLARATION

  20. BUILDSPEC.YML Collection of build commands in YAML Creates artifacts that

    are passed further along the pipeline Tips: Print useful progress messages, run verbose programs in quiet mode Use a cache Put “complex” multiline commands in a .sh file Update aws-cli if you need newish services support Debug your build process locally with Docker
  21. version: 0.2 env: variables: # ARTIFACTS_BUCKET is available from pipeline.yml

    PIP_CACHE: pip_cache phases: install: commands: - printenv - pip install -r requirements-dev.txt —cache-dir $PIP_CACHE pre_build: commands: - pytest -vrf tests build: commands: - aws cloudformation package --template-file infrastructure/functions.yml --output-template-file packaged_functions.yml --s3-bucket "${ARTIFACTS_BUCKET}" artifacts: files: - packaged_functions.yml cache: paths: - 'pip_cache/**/*'
  22. DEPLOY BETA STACK

  23. Name: Deploy Actions: - Name: Beta ActionTypeId: Category: Deploy Owner:

    AWS Provider: CloudFormation Version: '1' Configuration: ActionMode: CREATE_UPDATE Capabilities: CAPABILITY_IAM,CAPABILITY_AUTO_EXPAND RoleArn: !GetAtt CloudformationRole.Arn StackName: !Sub '${Service}-beta-functions' TemplatePath: BuildOutput::packaged_functions.yml ParameterOverrides: !Sub '{"Service": "${Service}", "Stage": "beta"}' InputArtifacts: - Name: BuildOutput
  24. DEPLOY PRODUCTION STACK

  25. Actions: - Name: Prod ActionTypeId: Category: Deploy Owner: AWS Provider:

    CloudFormation Version: '1' Configuration: ActionMode: CREATE_UPDATE Capabilities: CAPABILITY_IAM,CAPABILITY_AUTO_EXPAND RoleArn: !GetAtt CloudformationRole.Arn StackName: !Sub '${Service}-prod-functions' TemplatePath: BuildOutput::packaged_functions.yml ParameterOverrides: !Sub '{"Service": "${Service}", "Stage": “prod"}' InputArtifacts: - Name: BuildOutput
  26. LAMBDA CANARY DEPLOYS

  27. USING CODEDEPLOY Connect Lambda with CodeDeploy via a CloudWatch Alarm

    Typically monitor the error rate Easy with AWS SAM, but there’s too much magic Tips: Make sure the alarms works as expected! Skip gradual rollout for non-production stacks Deploy “smaller regions” (i.e. not us-east-1) first
  28. Parameters: Stage: Description: Environment stage (deployment phase) Type: String AllowedValues:

    - beta - prod Conditions: IsBeta: !Equals [!Ref Stage, beta] Resources: HelloWorld: Type: AWS::Serverless::Function Properties: # ... AutoPublishAlias: !Ref Stage DeploymentPreference: Type: !If [IsBeta, AllAtOnce, Canary10Percent5Minutes] Alarms: - !Ref HelloWorldErrorsAlarm
  29. HelloWorldErrorsAlarm: Type: AWS::CloudWatch::Alarm Properties: ComparisonOperator: GreaterThanThreshold Dimensions: - Name: Resource

    Value: !Sub '${HelloWorld}:${Stage}' - Name: FunctionName Value: !Ref HelloWorld - Name: ExecutedVersion Value: !GetAtt HelloWorld.Version.Version EvaluationPeriods: 2 MetricName: Errors Namespace: AWS/Lambda Period: 60 Statistic: Sum Threshold: 0
  30. DEMO

  31. WHAT’S NEXT? Dynamic feature pipelines Separate accounts per stage Multi-region

    deployments Pipeline performance metrics Deploying dependent services Photo by Samuel Zeller on Unsplash
  32. THANK YOU Photo by Joel & Jasmin Førestbird on Unsplash