Share
Embed
Architecting around Multiple AWS Accounts
Presented at AWS APAC Community Day 2018 @ Ho Chin Minh City, Vietnam
Steve Teo
September 17, 2018
Transcript
55 slides
-
Self Introduction • Steve Teo • Dev -> Build &
Release -> Cloud DevOps • 7 years working in various engineering teams • 3 years working on AWS • https://www.linkedin.com/in/steve-teo-b7988541/ -
Question •How many of you are responsible for your company’s
AWS Account(s)? •How many accounts does your company / team have? -
Purpose For cloud architects & engineers to be aware of
the benefits and complexities of an AWS Multi-Account Architecture -
Goals •Understand the various motivations for wanting to separate AWS
Accounts •Understand the immediate and non-immediate decisions you will be likely to face •Understand gotchas, best practices -
Background ? ? ? ? ? ? ? ? ?
? ? ? ? ? 10.0.0.0/16 10.0.0.0/16 -
Background • Previous Company • 2 Legacy Accounts with VPCs
with overlapping CIDRs • Non-Production and Production workloads sitting in the same Account YIKES! • Eventually migrated department workloads to 40+ AWS Accounts • March 2017: https://speakerdeck.com/stevepotayteo/a-multi-aws- account-story • Current Company • Worked on AWS Account & VPC Strategy based on enterprise requirements • Thinking of how to scale AWS Multi-Accounts and VPCs is one of my weird hobbies -
Lack of AWS Account or VPC Strategy -> Massive Technical
Debt Annoying Unusable -
What really needs to be done? •Define an AWS Account
Strategy •Which should be deliberate, tailored to your organization’s current needs and allowed to evolve to future needs •Removes doubt on where to place workloads -
Recap – AWS Account What is an AWS Account? 1.
Financial Responsibility 1. Billing and Financial 2. Reserved Instances 2. Resource Containment 1. Resources Boundary 2. Limits 3. Security Boundary 1. AWS User Access Security 2. Data Ref: https://www.slideshare.net/AmazonWebServices/arc325managing-multiple-aws-accounts-at-scale - Slide 14 -
Recap – VPC What is a VPC? 1. A virtual
private cloud (VPC) is a virtual network dedicated to your AWS account. It is logically isolated from other virtual networks in the AWS Cloud. 2. VPCs = network containment != AWS User Access Security! -
Separate by Business / Dev Team • "Any organization that
designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.” – Melvin Conway • Eg. Business – Profit Center • Eg. Dev Team – Cost Center • Usually done because of a need for showback / chargeback or isolation among workloads • Easily broken when prone to organization changes -
Separate by Platform / Service / System / Application •Wide-grained
– Platform / Service •Fine-grained – System / Application •Splitting it too fine-grained might not make sense at all if it makes workloads too small • Eg. 1 AWS account just for 1 EC2? • Container optimization? -
Separate by Environment •By default you get •network / data
containment •user access security •Orthogonal to other ways of separation •Eg. Sandbox / Non-Prod / Prod / DR •Eg. DEV / SIT / QA / STG / PROD / DR -
Other Ways •Service Tiering (eg. Tier 1, Tier 2 services)
•PCI / HIPAA (Regulated vs Non-regulated) •AWS Service Limits / API Rate Limits •Limit visibility of workloads -
Special Accounts • Shared / Management Services (eg. Tools, DNS,
AD) • Landing Zone (Bastion) account • Direct Connect (For provisioning of DX) • Logging Account • Security Account • Sec Logs Account • Transit Account (Transit VPC for hybrid connectivity) • Backup Vault (for DR) • Organisation Master Billing Account -
Drive towards Clean Evolving Architecture • “The purpose of a
good architecture is to defer decisions, delay decisions.” – Uncle Bob Martin •Decisions are driven by concerns, which can be immediate or non-immediate depending on scale and requirements • Usually any concerns around networking foundation are immediate -
Recap – AWS Account & VPC Separate AWS accounts by
definition means separate VPCs! You got to deal with scaling VPCs as well! -
VPC Strategy •You need a VPC strategy •Reference VPC Architectures
•Proper IP Address allocation to prevent overlapping VPC CIDRs •Maintain a proper inventory! •VPC Peering and limitations per VPC •Soft: 50, Hard: 125 -
Direct Connect Strategy •Significantly more complex when you need to
have DX to your on-prem environment •Pre-mature buying of DX and related services without understanding • Limitations of the various DX options • AWS Account and VPC Strategy • Interactivity between on-prem and AWS workloads • Security and Infrastructure policies •Engage a proper partner who can advise you -
Centralizing Internal DNS https://aws.amazon.com/blogs/security/how-to-centralize-dns-management-in-a-multi-account-environment/ Authorizations that you can create so
you can associate VPCs that were created by one account with a hosted zone that was created by another account = 100 -
Cost Attribution •Especially if your AWS bill goes to different
cost centers, like in an Enterprise •You need to pre-empt and explain to your finance team how to handle a AWS bill and do cost attribution before they get really upset •Better for cost transparency than cost tag allocation -
AWS Account Security • Preventive Controls • Use AWS Organisations
– Service Control Policies to enforce global policies, eg. Prevent disabling of Cloudtrail across all accounts • All Root Accounts and Admin accounts to be 2FA enabled • Apply baseline IAM policies through Cloudformation / Terraform and CI/CD Pipelines • Detective Controls • Enable AWS GuardDuty -
Resource Security • Stick to established Architectural Principles and Reference
VPCs Patterns • Build up a security model around individual resources to understand how to secure it • Detective & Remediative Controls • Automated AWS Config Rules, eg. Disable global unrestricted access to Port 22 • Enable AWS GuardDuty -
Centralize Communication Channels • Forward all emails to all the
root account emails into a single inbox to make sure no email is missed • Incidents / Deprecations / Outages • Setup and standardize alternate contacts -
Reduce Entropy through Infra as Code •Are your AWS Accounts
Pets or Cattle? •Automate everything – Terraform / CloudFormation through CI / CD pipeline •Prevent / avoid manual changes as much as possible to eliminate configuration draft -
Logs & Alerts Management • Infrastructure Monitoring • Cloudwatch can’t
scale beyond 1 Account. Use tools like Datadog • Centralize Logs to Logging Accounts / Logging Infrastructure • Cloudtrail • Cloudwatch Logs • Config • S3 Logs • ELB Logs • Infrastructure Events • CloudFront • VPC Flow Logs • Leverage on ChatOps – pump useful alerts to Slack for monitoring discussion -
Golden Image Preparation • Need to prepare and distribute to
one multiple accounts • Use • AWS Systems Manager Automation • Packer • And CI / CD Pipelines • Document images manifest and inventory -
Backup & Restore Use proper tools like Cloud Protection Manager
(CPM) to provide single pane of glass and orchestrate across accounts eg. DR scenario However: Inflexible licensing around multiple AWS Accounts -
Limit Unauthorized Resource Sprawl • “You Can’t See What You
Don't Know” • Preventive Controls • Restrict services via AWS Organisation SCP • Restrict regions via IAM, i.e. aws:RequestedRegion • Detective Controls • Enable Cloudtrail / Config -
IAM User Access Management • Insane to maintain individual IAM
users on every account, eg. 100 accounts, 10 users each • Options • Use a landing zone account, and STS AssumeRole to other account’s IAM roles • What worked well for me in the past • AD -> Okta -> SAML Federation to every AWS Account • Okta was good for us to manage complexity with multiple services -
Gotchas • Reserved Instances and Capacity Reservation • RIs in
a central account vs RIs in separate accounts • Capacity Reservation only holds if specific account and specific AZs • Resource & Charges Duplication • Same resources provisioned across multiple accounts • Might want to consolidate if you have the option • Eg. Multiple NAT Gateways across AWS Accounts - Considering using Proxy Instances • AWS POC Credits • Do not join a POC account to an existing Organisation, as credits can only be applied on the master billing account! Instead let the POC account be standalone for now so that you can apply the credits directly on it -
Gotchas • Certain services do not support single pane of
glass yet • Eg. Cloudwatch (use something like Datadog to mitigate this) • Eg. Trusted Advisor • Might have to roll in your own and use third party • Some commercial tools can’t support or have punitive licenses for Multiple AWS Accounts -
Previous Gotchas (No longer Issues) • Consolidated Billing and Reserved
Instances • Wanted selection of reserved instances specifically not against certain accounts • https://aws.amazon.com/about-aws/whats-new/2017/11/customize-your- organizations-aws-credit-and-reserved-instance-ri-discount-sharing-using- new-billing-preferences/ -
Tips & Best Practices • There is no perfect design.
It’s all about meeting your requirements as best • Evolving Architecture • Make the most important decisions first, eg. AWS Account Structure, VPC Segmentation • AWS is a very complex beast. Understand the fundamentals well • Be hands-on. Experiment and validate assumptions with hands-on • Be aware of soft and hard limits which can impact the overall architecture, eg. VPC Peering Limits, DX VIF • Enforce a naming strategy, document or keep an inventory of • AWS Accounts, VPCs -
Tips & Best Practices • Enable CloudTrail / Config by
default • Secure globally using AWS Organisations SCP where possible • Automate AWS Account Configuration - Terraform / CloudFormation • Centralize Everything eg. AWS Systems Manager • Use SaaS/tools that can scale with multiple AWS Accounts (eg. Licensing, Automation) • Know the nature of AWS resources (eg. Account-Specific, AZ-Specific, Region-Specific, VPC-Specific) • Disable unused services / regions • Do not create any workloads on the organisation master account -
Learning Resources • From One to Many: Evolving VPC •
https://www.youtube.com/watch?v=jjk_zZRLXXw 8:31 • https://www.youtube.com/watch?v=3Gv47NASmU4 20:35 • https://aws.amazon.com/answers/account-management/aws-multi- account-security-strategy/ • https://www.slideshare.net/AmazonWebServices/arc325managing- multiple-aws-accounts-at-scale • http://www.glomex.com/blog/multi-account-handling/ • https://aws.amazon.com/answers/aws-landing-zone/