No content

Architecting around Multiple AWS Accounts Things you really need to know 

Self Introduction • Steve Teo • Dev -> Build & Release -> Cloud DevOps • 7 years working in various engineering teams • 3 years working on AWS • https://www.linkedin.com/in/steve-teo-b7988541/ 

Community https://www.meetup.com/AWS-SG/ https://www.meetup.com/ Atlassian-User-Group-Singapore/ 

What is this talk about? 

Question •How many of you are responsible for your company’s AWS Account(s)? •How many accounts does your company / team have? 

Purpose For cloud architects & engineers to be aware of the benefits and complexities of an AWS Multi-Account Architecture 

Goals •Understand the various motivations for wanting to separate AWS Accounts •Understand the immediate and non-immediate decisions you will be likely to face •Understand gotchas, best practices 

So why this talk? 

Background ? ? ? ? ? ? ? ? ? ? ? ? ? ? 10.0.0.0/16 10.0.0.0/16 

Background • Previous Company • 2 Legacy Accounts with VPCs with overlapping CIDRs • Non-Production and Production workloads sitting in the same Account YIKES! • Eventually migrated department workloads to 40+ AWS Accounts • March 2017: https://speakerdeck.com/stevepotayteo/a-multi-aws- account-story • Current Company • Worked on AWS Account & VPC Strategy based on enterprise requirements • Thinking of how to scale AWS Multi-Accounts and VPCs is one of my weird hobbies 

Lack of AWS Account or VPC Strategy -> Massive Technical Debt Annoying Unusable 

What really needs to be done? •Define an AWS Account Strategy •Which should be deliberate, tailored to your organization’s current needs and allowed to evolve to future needs •Removes doubt on where to place workloads 

Motivations for Separating AWS Accounts 

Recap – AWS Account What is an AWS Account? 1. Financial Responsibility 1. Billing and Financial 2. Reserved Instances 2. Resource Containment 1. Resources Boundary 2. Limits 3. Security Boundary 1. AWS User Access Security 2. Data Ref: https://www.slideshare.net/AmazonWebServices/arc325managing-multiple-aws-accounts-at-scale - Slide 14 

Recap – VPC What is a VPC? 1. A virtual private cloud (VPC) is a virtual network dedicated to your AWS account. It is logically isolated from other virtual networks in the AWS Cloud. 2. VPCs = network containment != AWS User Access Security! 

No content

Separate by Business / Dev Team • "Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.” – Melvin Conway • Eg. Business – Profit Center • Eg. Dev Team – Cost Center • Usually done because of a need for showback / chargeback or isolation among workloads • Easily broken when prone to organization changes 

Separate by Platform / Service / System / Application •Wide-grained – Platform / Service •Fine-grained – System / Application •Splitting it too fine-grained might not make sense at all if it makes workloads too small • Eg. 1 AWS account just for 1 EC2? • Container optimization? 

Separate by Environment •By default you get •network / data containment •user access security •Orthogonal to other ways of separation •Eg. Sandbox / Non-Prod / Prod / DR •Eg. DEV / SIT / QA / STG / PROD / DR 

Other Ways •Service Tiering (eg. Tier 1, Tier 2 services) •PCI / HIPAA (Regulated vs Non-regulated) •AWS Service Limits / API Rate Limits •Limit visibility of workloads 

Special Accounts • Shared / Management Services (eg. Tools, DNS, AD) • Landing Zone (Bastion) account • Direct Connect (For provisioning of DX) • Logging Account • Security Account • Sec Logs Account • Transit Account (Transit VPC for hybrid connectivity) • Backup Vault (for DR) • Organisation Master Billing Account 

No content

No content

Drive towards Clean Evolving Architecture • “The purpose of a good architecture is to defer decisions, delay decisions.” – Uncle Bob Martin •Decisions are driven by concerns, which can be immediate or non-immediate depending on scale and requirements • Usually any concerns around networking foundation are immediate 

Immediate Concerns 

Recap – AWS Account & VPC Separate AWS accounts by definition means separate VPCs! You got to deal with scaling VPCs as well! 

VPC Strategy •You need a VPC strategy •Reference VPC Architectures •Proper IP Address allocation to prevent overlapping VPC CIDRs •Maintain a proper inventory! •VPC Peering and limitations per VPC •Soft: 50, Hard: 125 

Direct Connect Strategy •Significantly more complex when you need to have DX to your on-prem environment •Pre-mature buying of DX and related services without understanding • Limitations of the various DX options • AWS Account and VPC Strategy • Interactivity between on-prem and AWS workloads • Security and Infrastructure policies •Engage a proper partner who can advise you 

Decision Point: Direct? 

Decision Point: Transit VPC? Ref: https://theithollow.com/2018/07/16/should-i-use-a-transit-vpc-in-aws/ 

Decision Point: Shared Services VPC https://aws.amazon.com/answers/networking/aws-multiple-vpc-vpn-connection-sharing/ 

Centralizing Internal DNS https://aws.amazon.com/blogs/security/how-to-centralize-dns-management-in-a-multi-account-environment/ Authorizations that you can create so you can associate VPCs that were created by one account with a hosted zone that was created by another account = 100 

Centralizing Internal DNS https://aws.amazon.com/blogs/security/how-to-centralize-dns-management-in-a-multi-account-environment/ 

Cost Attribution •Especially if your AWS bill goes to different cost centers, like in an Enterprise •You need to pre-empt and explain to your finance team how to handle a AWS bill and do cost attribution before they get really upset •Better for cost transparency than cost tag allocation 

AWS Account Security • Preventive Controls • Use AWS Organisations – Service Control Policies to enforce global policies, eg. Prevent disabling of Cloudtrail across all accounts • All Root Accounts and Admin accounts to be 2FA enabled • Apply baseline IAM policies through Cloudformation / Terraform and CI/CD Pipelines • Detective Controls • Enable AWS GuardDuty 

Resource Security • Stick to established Architectural Principles and Reference VPCs Patterns • Build up a security model around individual resources to understand how to secure it • Detective & Remediative Controls • Automated AWS Config Rules, eg. Disable global unrestricted access to Port 22 • Enable AWS GuardDuty 

Centralize Communication Channels • Forward all emails to all the root account emails into a single inbox to make sure no email is missed • Incidents / Deprecations / Outages • Setup and standardize alternate contacts 

Non-Immediate Concerns For smaller AWS Account footprints 

Reduce Entropy through Infra as Code •Are your AWS Accounts Pets or Cattle? •Automate everything – Terraform / CloudFormation through CI / CD pipeline •Prevent / avoid manual changes as much as possible to eliminate configuration draft 

Logs & Alerts Management • Infrastructure Monitoring • Cloudwatch can’t scale beyond 1 Account. Use tools like Datadog • Centralize Logs to Logging Accounts / Logging Infrastructure • Cloudtrail • Cloudwatch Logs • Config • S3 Logs • ELB Logs • Infrastructure Events • CloudFront • VPC Flow Logs • Leverage on ChatOps – pump useful alerts to Slack for monitoring discussion 

Golden Image Preparation • Need to prepare and distribute to one multiple accounts • Use • AWS Systems Manager Automation • Packer • And CI / CD Pipelines • Document images manifest and inventory 

Backup & Restore Use proper tools like Cloud Protection Manager (CPM) to provide single pane of glass and orchestrate across accounts eg. DR scenario However: Inflexible licensing around multiple AWS Accounts 

Limit Unauthorized Resource Sprawl • “You Can’t See What You Don't Know” • Preventive Controls • Restrict services via AWS Organisation SCP • Restrict regions via IAM, i.e. aws:RequestedRegion • Detective Controls • Enable Cloudtrail / Config 

IAM User Access Management • Insane to maintain individual IAM users on every account, eg. 100 accounts, 10 users each • Options • Use a landing zone account, and STS AssumeRole to other account’s IAM roles • What worked well for me in the past • AD -> Okta -> SAML Federation to every AWS Account • Okta was good for us to manage complexity with multiple services 

No content

Gotchas 

Gotchas • Reserved Instances and Capacity Reservation • RIs in a central account vs RIs in separate accounts • Capacity Reservation only holds if specific account and specific AZs • Resource & Charges Duplication • Same resources provisioned across multiple accounts • Might want to consolidate if you have the option • Eg. Multiple NAT Gateways across AWS Accounts - Considering using Proxy Instances • AWS POC Credits • Do not join a POC account to an existing Organisation, as credits can only be applied on the master billing account! Instead let the POC account be standalone for now so that you can apply the credits directly on it 

Gotchas • Certain services do not support single pane of glass yet • Eg. Cloudwatch (use something like Datadog to mitigate this) • Eg. Trusted Advisor • Might have to roll in your own and use third party • Some commercial tools can’t support or have punitive licenses for Multiple AWS Accounts 

Previous Gotchas (No longer Issues) • Consolidated Billing and Reserved Instances • Wanted selection of reserved instances specifically not against certain accounts • https://aws.amazon.com/about-aws/whats-new/2017/11/customize-your- organizations-aws-credit-and-reserved-instance-ri-discount-sharing-using- new-billing-preferences/ 

Tips & Best Practices • There is no perfect design. It’s all about meeting your requirements as best • Evolving Architecture • Make the most important decisions first, eg. AWS Account Structure, VPC Segmentation • AWS is a very complex beast. Understand the fundamentals well • Be hands-on. Experiment and validate assumptions with hands-on • Be aware of soft and hard limits which can impact the overall architecture, eg. VPC Peering Limits, DX VIF • Enforce a naming strategy, document or keep an inventory of • AWS Accounts, VPCs 

Tips & Best Practices • Enable CloudTrail / Config by default • Secure globally using AWS Organisations SCP where possible • Automate AWS Account Configuration - Terraform / CloudFormation • Centralize Everything eg. AWS Systems Manager • Use SaaS/tools that can scale with multiple AWS Accounts (eg. Licensing, Automation) • Know the nature of AWS resources (eg. Account-Specific, AZ-Specific, Region-Specific, VPC-Specific) • Disable unused services / regions • Do not create any workloads on the organisation master account 

Learning Resources • From One to Many: Evolving VPC • https://www.youtube.com/watch?v=jjk_zZRLXXw 8:31 • https://www.youtube.com/watch?v=3Gv47NASmU4 20:35 • https://aws.amazon.com/answers/account-management/aws-multi- account-security-strategy/ • https://www.slideshare.net/AmazonWebServices/arc325managing- multiple-aws-accounts-at-scale • http://www.glomex.com/blog/multi-account-handling/ • https://aws.amazon.com/answers/aws-landing-zone/ 

Q & A? 

Thank you