CI/CD Pipelines w/ CDK Multi-Account/Region Deployments October 2021 

Intros AWS 2 Austin Loveless - Founder Full Stack/DevOps Engineer - Stuzo AWS Community Hero Kevin Tinn – Co-organizer Sr Cloud Application Architect – AWS AWS Community Builder 

Agenda AWS 3 • Overview of CI/CD • CDK Pipeline Components • Cross Account Setup • Regional CDK Bootstrapping • Initial Deployment of Pipeline • App Architecture • Organizing the Deployments • Transitions • Cleanup • Q&A 

Overview of CI/CD Pipelines AWS 4 • CI/CD: Continuous Integration/Continuous Deployment • Pipeline: Automated tooling that defines and automates execution of steps to ensure code is of sufficient quality and is deployed in a repeatable, consistent manner • CI/CD bridges the gaps between development and operation activities and teams by enforcing automation in building, testing and deployment of applications. - https://en.wikipedia.org/wiki/CI/CD • Upon update to the code in source control, the pipeline is executed and software is deployed without manual action 

CDK Pipeline Concepts AWS 5 • Source: Fetches source code and initiates the pipeline • Build: Compiles code and creates CloudFormation used in subsequent steps • UpdatePipeline: Modifies the pipeline, if necessary, to accommodate added/removed stages/waves in the pipeline • PuplishAssets: Preps and publishes files (To S3) and containers (To ECR) to all target deployment regions 

CDK Pipeline Components AWS 6 • Pipeline: A CDK Stack that defines the steps of the CI/CD automation and deployment targets • Stage: Single deployment unit of your application (e.g. networking, databases, lambda, api gateway) • Wave: Grouping of Stages that defines what Stages can be deployed in parallel • Actions: Actions between Stages that can be used to define additional pipeline activities and workflow (e.g. Automated testing and deployment approvals) 

Cross Account Setup AWS 7 The following depicts the organization of accounts we use: 

Cross Account Setup - Console AWS 8 • In target account create role, configure it to trust pipeline account, and attach policy • OrganizationAccountAccessRole is what I used 

Cross Account Setup - Console AWS 9 • Not necessary for pipeline, but useful • In pipeline account create policy and attach to role you use that allows account, and attach policy: • Repeat for all accounts • Use Switch Role link in menu drop- down to switch between accounts • Menu keeps history (Very useful) 

Regional CDK Bootstrapping AWS 1 0 • Each account/region combo that is deployed to must be bootstrapped. Since it is a cross-account deployment, a trust must be established during this process. • Deploying AWS CDK apps into an AWS environment may require that you provision resources the AWS CDK needs to perform the deployment. These resources include an Amazon S3 bucket for storing files and IAM roles that grant permissions needed to perform deployments. The process of provisioning these initial resources is called bootstrapping. 

Regional CDK Bootstrapping AWS 1 1 • For each target account/region run the following CLI command (Must be ran as user with appropriate privs in the target account): cdk bootstrap --trust --cloudformation-execution- policies arn:aws:iam::aws:policy/AdministratorAccess aws:/// • Given we are deploying to 2 regions in 3 different accounts, we must run this command 6 times 

Initial Deployment of Pipeline – Pipeline Overview AWS 1 2 • Repo: https://github.com/kevasync/awsmug-serverless- graphql-api • ./cdk/bin/pipeline.ts: Creation of CDK App for Pipeline • ./cdk/lib/cdk-pipeline-stack.ts: Definition of stacks to deploy, as well as environments to deploy to • Must be deployed on-time manually 

Initial Deployment of Pipeline – GitHub Access AWS 1 3 • In pipeline account/region, create Secret Manager secret to store access token for GitHub repo • Create at https://github.com/settings/tokens/ • Store token as plaintext secret in Secret Manager w/ name github-token 

Initial Deployment of Pipeline AWS 1 4 • Clone the repo and run command: cdk deploy CdkPipelineStack • When prompted to create security groups/deploy, accept • Navigate to CodePipeline in Console and cancel the initial build that • Set Env Vars in the Build step of the pipeline 

Initial Deployment of Pipeline – Env Config AWS 1 5 

Configure Pipeline Role Privs AWS 1 6 • In pipeline account, create a Policy for each target account to allow Pipeline role to assume Roles created during bootstrap process • Get CDK prefix from a target account: • Create policy for all 3 accounts: 

Configure Pipeline Role Privs AWS 1 7 Attach all 3 policies to the Pipeline Build role (Very similar to what we did for cross account access in console): 

Initial Deployment of Pipeline – Run Pipeline AWS 1 8 • Navigate to Pipeline and Release Changes 

App Architecture AWS 1 9 • 3 Stacks: • VPC: Core networking • RDS: RDS Postgres Instance (or read replica in multi-region deployment) • API: Lambda w/ VPC attachment and API gateway • Let’s take a look at some diagrams and some code 

App Architecture AWS 2 0 

Multi-Account/Region Gotchas AWS 2 1 • When deploying robust, fault-tolerant infra across accounts and regions be aware of secrets and encryption • To use Secret Manager secrets across accounts, specific policies on the secret must be set • This is cumbersome • To use secrets across regions in the same account, Secret Manager secret replication is ideal • When encrypting in multiple accounts/regions, KMS keys must be managed. • Each account/region has it’s own master keys, which is used in many instances as the default KMS key to encrypt with 

Organizing the Deployments AWS 2 2 • Deployment of 6 different RDS instances takes a long time • Waves can be used to deploy multiple stages in parallel • In this example: • DEV and QA • PRD and STG Primary • PRD and STG Secondary 

Actions AWS 2 3 • Robust CI/CD pipelines also require testing and deployment approval steps • Can run unit tests in pipeline constructor’s synth step • Can run functional/e2e tests in a wave/stage’s post steps • Can add approval steps to a stage/wave option’s pre array: 

Cleanup AWS 2 4 • Each Stack (VPC, RDS, API) is deployed independently to each account/region • This allows each to be updated separately • You will need to go CloudFormation in each account/region and delete the stacks when you want to clean up the resources • Explore nested stacks if this behavior is not acceptable 

Thanks for joining! AWS 2 5 • Q&A • Please connect on LinkedIn • Upcoming Meetups: • November 10th: From MVP to Production w/ Serverless Chris Gradwohl Full Stack Engineer and AWS Serverless Architect at Courier • December: CI/CD using Trunk Based Development Austin Loveless Full Stack/DevOps Engineer at Stuzo AWS Community Hero