A Multi AWS Account Story

It was presented during the AWS Singapore March Meetup

Transcript

1. A Multi AWS Account Story Steve Teo
2. Self-Introduction • Been working on AWS for almost 2 years • Developer -> Release Engineer -> Cloud DevOps • Organiser of the Singapore Atlassian User Group https://www.meetup.com/Atlassian-User-Group-Singapore/ • LinkedIn: https://www.linkedin.com/in/steve-teo-b7988541/
3. Background • September 2015 • Dual Roles - Public Cloud Engineering SME & Product Development Team • Product Engineering Team ◦ On-Prem moving to the Cloud ◦ Mixed Windows & Linux Workloads, Manual Deployment Process ◦ Charge-back was a very important consideration ◦ “Oh btw, you are the Infra Lead now”
6. State of the AWS Accounts • Good: Consolidated Billing • Good: No need to link anything back to on-prem • Bad: Multiple VPCs with conflicting or too big CIDRs. E.g. 10.0.0.0/16 • Bad: No clear purpose of various AWS accounts • Bad: Undocumented AWS Accounts • Bad: Unclear chargeback / workload purpose • Bad: Abandoned Workloads • Can’t be helped: Vendors had AWS Management IAM Access • Definite Pain Point: Conflicting or too big CIDRs
7. Research • From One to Many: Evolving VPC ◦ https://www.youtube.com/watch?v=jjk_zZRLXXw 8:31 ◦ Latest Version: https://www.youtube.com/watch?v=3Gv47NASmU4 20:35 • VPC Peering • Blast Radius • Read passing mention of Intuit having > 1000 AWS Accounts • Action Plan ◦ VPC Segmentation ◦ AWS Account Segregation (Security / Charge-back)
10. How We Separated AWS Accounts • Separate by Projects ◦ Purpose & Charge-back • Then Environments (Prod , Non-Prod) ◦ Security • Shared Services (Management), eg. Tools, AD • Consolidated Billing Master Account
11. Other Ways/Motivations to Separate AWS Accounts • Direct Connect Account • Backup Vault Account (For DR) • Logging Account • Security Account • Nature of Apps (Regulated vs Non-Regulated) • Different Departments / Teams / Chargeback • AWS Account Limit / API Rate Limits • Limit Visibility of Workloads • Organisational Changes
13. New Challenges to Solve • Good: Easy to reason about our infrastructure and sleep at night • Day 2 Concerns ◦ Creating and Securing Each AWS Account ◦ VPC Connectivity ◦ Visibility of resources ◦ All Others ▪ DNS ▪ Logs, Monitoring, Compliance & Audit ▪ Backup, Patch Management • Good to invest engineering effort to strategize and automate this!
14. Creation of AWS Accounts • Every AWS Account requires a new email address (painful process for us) • Creation was a manual process. Painful and boring • Snooped around ◦ https://github.com/intuit/aws_account_utils ◦ AWS Internal API to automate AWS Account Creation • Have a naming convention to your account • If I had to do it again ◦ Use AWS Organisations ◦ Use Email Addresses with the + trick, eg. master+account1@gmail.com
15. Securing AWS Root Account • Root Account ◦ Multiple Google Authenticators with Authy - to prevent SPOF • If I had to do it again ◦ Use AWS Organisations ◦ “An new account that is created by using AWS Organizations does not initially have a password assigned to the root user.”
16. Securing AWS IAM Accounts • What we saw out there ◦ STS AssumeRole from Jump Host Account ◦ AWS Managed AD was not existent yet • Our Approach ◦ AD -> Okta -> SAML Federation to every AWS Account ◦ AD was good for us because of Windows workloads ◦ Okta was good for us to manage complexity with multiple services
18. VPC Connectivity
19. DNS • More of a challenge associated with VPC Peering as a result of AWS Accounts • We ended up using AD DNS to manage internal DNS across all VPCs, with Route53 Zones for public DNS • Considerations ◦ Route53 doesn’t integrate neatly with resources in different AWS accounts
20. Logs, Monitoring, Compliance & Audit • Infrastructure Monitoring ◦ Datadog. Cloudwatch can’t scale beyond 1 Account • Centralize Logs ◦ Cloudtrail ◦ Config ◦ S3 Logs ◦ ELB Logs ◦ Infrastructure Events ◦ CloudFront ◦ VPC Flow Logs • Leverage on ChatOps - Real-Time monitoring and discussion
21. Backup, Patch Management • Backup ◦ Use Cloud Protection Management (CPM) ◦ However: Inflexible licensing around multiple AWS Accounts • Patch Management ◦ Consider using AWS EC2 Systems Manager ◦ Or leverage on Configuration Management with Packer / Shared AMIs
22. Other Complexities Found Along the Way • Consolidated Billing and Reserved Instances ◦ Wanted selection of reserved instances specifically not against certain accounts • Reserved Instances and Capacity Reservation ◦ RIs in a central account vs RIs in separate accounts ◦ Capacity Reservation only holds if specific account and specific AZs • Resource Duplication ◦ Eg. Multiple NAT Instances - Considering using Proxy Instances • Support for Multiple AWS Accounts? - We were on Enterprise Support, so this did not bite us
23. Recommended Best Practices • Evolving Architecture ◦ Make the most important decision first, eg. VPC Segmentation ◦ “A good architecture allows major decisions to be deferred” • Document or CMDB the following ◦ AWS Accounts ◦ VPCs • Naming Strategy for at least ◦ AWS Accounts • Automate AWS Account Configuration - Terraform / CloudFormation • Centralize Everything • Secure Everything ◦ Use AWS Organisations Service Control Policies
24. Recommended Best Practices • Use SaaS/Tools that can scale to multiple AWS Accounts (eg. Licensing, Automation) • Be aware of the nature of AWS resources (eg. Account-Specific, AZ-Specific, Region-Specific, VPC-Specific) • Be aware of limits which can impact the overall architecture ◦ VPC Peering Limits ◦ Software Licenses • Look at AWS Organisations
25. Summary • Think about using a multi Account Strategy in any AWS Journey; every organisation can benefit from this • Research far and wide, think of scenarios and do POCs • Good to invest engineering effort to strategize and automate the associated challenges
27. Learning Resources • From One to Many: Evolving VPC ◦ https://www.youtube.com/watch?v=jjk_zZRLXXw 8:31 ◦ Latest Version: https://www.youtube.com/watch?v=3Gv47NASmU4 20:35 • https://aws.amazon.com/answers/account-management/aws-multi-account-se curity-strategy/ • http://www.glomex.com/blog/multi-account-handling/
28. Q & A?
29. Thank you