Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A Multi AWS Account Story

A Multi AWS Account Story

It was presented during the AWS Singapore March Meetup


Steve Teo

March 14, 2017

More Decks by Steve Teo

Other Decks in Technology


  1. A Multi AWS Account Story Steve Teo

  2. Self-Introduction • Been working on AWS for almost 2 years

    • Developer -> Release Engineer -> Cloud DevOps • Organiser of the Singapore Atlassian User Group https://www.meetup.com/Atlassian-User-Group-Singapore/ • LinkedIn: https://www.linkedin.com/in/steve-teo-b7988541/
  3. Background • September 2015 • Dual Roles - Public Cloud

    Engineering SME & Product Development Team • Product Engineering Team ◦ On-Prem moving to the Cloud ◦ Mixed Windows & Linux Workloads, Manual Deployment Process ◦ Charge-back was a very important consideration ◦ “Oh btw, you are the Infra Lead now”
  4. None
  5. None
  6. State of the AWS Accounts • Good: Consolidated Billing •

    Good: No need to link anything back to on-prem • Bad: Multiple VPCs with conflicting or too big CIDRs. E.g. • Bad: No clear purpose of various AWS accounts • Bad: Undocumented AWS Accounts • Bad: Unclear chargeback / workload purpose • Bad: Abandoned Workloads • Can’t be helped: Vendors had AWS Management IAM Access • Definite Pain Point: Conflicting or too big CIDRs
  7. Research • From One to Many: Evolving VPC ◦ https://www.youtube.com/watch?v=jjk_zZRLXXw

    8:31 ◦ Latest Version: https://www.youtube.com/watch?v=3Gv47NASmU4 20:35 • VPC Peering • Blast Radius • Read passing mention of Intuit having > 1000 AWS Accounts • Action Plan ◦ VPC Segmentation ◦ AWS Account Segregation (Security / Charge-back)
  8. None
  9. None
  10. How We Separated AWS Accounts • Separate by Projects ◦

    Purpose & Charge-back • Then Environments (Prod , Non-Prod) ◦ Security • Shared Services (Management), eg. Tools, AD • Consolidated Billing Master Account
  11. Other Ways/Motivations to Separate AWS Accounts • Direct Connect Account

    • Backup Vault Account (For DR) • Logging Account • Security Account • Nature of Apps (Regulated vs Non-Regulated) • Different Departments / Teams / Chargeback • AWS Account Limit / API Rate Limits • Limit Visibility of Workloads • Organisational Changes
  12. None
  13. New Challenges to Solve • Good: Easy to reason about

    our infrastructure and sleep at night • Day 2 Concerns ◦ Creating and Securing Each AWS Account ◦ VPC Connectivity ◦ Visibility of resources ◦ All Others ▪ DNS ▪ Logs, Monitoring, Compliance & Audit ▪ Backup, Patch Management • Good to invest engineering effort to strategize and automate this!
  14. Creation of AWS Accounts • Every AWS Account requires a

    new email address (painful process for us) • Creation was a manual process. Painful and boring • Snooped around ◦ https://github.com/intuit/aws_account_utils ◦ AWS Internal API to automate AWS Account Creation • Have a naming convention to your account • If I had to do it again ◦ Use AWS Organisations ◦ Use Email Addresses with the + trick, eg. master+account1@gmail.com
  15. Securing AWS Root Account • Root Account ◦ Multiple Google

    Authenticators with Authy - to prevent SPOF • If I had to do it again ◦ Use AWS Organisations ◦ “An new account that is created by using AWS Organizations does not initially have a password assigned to the root user.”
  16. Securing AWS IAM Accounts • What we saw out there

    ◦ STS AssumeRole from Jump Host Account ◦ AWS Managed AD was not existent yet • Our Approach ◦ AD -> Okta -> SAML Federation to every AWS Account ◦ AD was good for us because of Windows workloads ◦ Okta was good for us to manage complexity with multiple services
  17. None
  18. VPC Connectivity

  19. DNS • More of a challenge associated with VPC Peering

    as a result of AWS Accounts • We ended up using AD DNS to manage internal DNS across all VPCs, with Route53 Zones for public DNS • Considerations ◦ Route53 doesn’t integrate neatly with resources in different AWS accounts
  20. Logs, Monitoring, Compliance & Audit • Infrastructure Monitoring ◦ Datadog.

    Cloudwatch can’t scale beyond 1 Account • Centralize Logs ◦ Cloudtrail ◦ Config ◦ S3 Logs ◦ ELB Logs ◦ Infrastructure Events ◦ CloudFront ◦ VPC Flow Logs • Leverage on ChatOps - Real-Time monitoring and discussion
  21. Backup, Patch Management • Backup ◦ Use Cloud Protection Management

    (CPM) ◦ However: Inflexible licensing around multiple AWS Accounts • Patch Management ◦ Consider using AWS EC2 Systems Manager ◦ Or leverage on Configuration Management with Packer / Shared AMIs
  22. Other Complexities Found Along the Way • Consolidated Billing and

    Reserved Instances ◦ Wanted selection of reserved instances specifically not against certain accounts • Reserved Instances and Capacity Reservation ◦ RIs in a central account vs RIs in separate accounts ◦ Capacity Reservation only holds if specific account and specific AZs • Resource Duplication ◦ Eg. Multiple NAT Instances - Considering using Proxy Instances • Support for Multiple AWS Accounts? - We were on Enterprise Support, so this did not bite us
  23. Recommended Best Practices • Evolving Architecture ◦ Make the most

    important decision first, eg. VPC Segmentation ◦ “A good architecture allows major decisions to be deferred” • Document or CMDB the following ◦ AWS Accounts ◦ VPCs • Naming Strategy for at least ◦ AWS Accounts • Automate AWS Account Configuration - Terraform / CloudFormation • Centralize Everything • Secure Everything ◦ Use AWS Organisations Service Control Policies
  24. Recommended Best Practices • Use SaaS/Tools that can scale to

    multiple AWS Accounts (eg. Licensing, Automation) • Be aware of the nature of AWS resources (eg. Account-Specific, AZ-Specific, Region-Specific, VPC-Specific) • Be aware of limits which can impact the overall architecture ◦ VPC Peering Limits ◦ Software Licenses • Look at AWS Organisations
  25. Summary • Think about using a multi Account Strategy in

    any AWS Journey; every organisation can benefit from this • Research far and wide, think of scenarios and do POCs • Good to invest engineering effort to strategize and automate the associated challenges
  26. None
  27. Learning Resources • From One to Many: Evolving VPC ◦

    https://www.youtube.com/watch?v=jjk_zZRLXXw 8:31 ◦ Latest Version: https://www.youtube.com/watch?v=3Gv47NASmU4 20:35 • https://aws.amazon.com/answers/account-management/aws-multi-account-se curity-strategy/ • http://www.glomex.com/blog/multi-account-handling/
  28. Q & A?

  29. Thank you