Continuous Delivery Techniques for Managing AWS Accounts

Transcript

1. Rob Godfrey @robgodfr Continuous Delivery Techniques for Managing AWS Accounts
2. Usage of AWS at the FT • Started pilots in 2013 • Began migration of workloads in early 2014 • Today 600+ workloads are running in 50+ accounts @robgodfr
3. Problem: Large Shared Accounts 700+ EC2 Instances 250+ TB S3 Storage 1 PB Data Transfer / Mth 300+ Access Keys 300+ Individuals @robgodfr
4. Migrate to more but smaller accounts @robgodfr
5. Motivations for Multiple Accounts • Least privilege access to data and resources • Reduced blast radius in event of a security breach • Clearer resource ownership, better cost attribution and improved housekeeping @robgodfr
6. Challenges • Manual processes for provisioning & updating accounts • Significant inconsistencies across accounts • Limited investment in automation or testing 
 @robgodfr
7. Guiding Principles @robgodfr • Keep everything in version control • Create a repeatable, reliable process for releasing software • Automate almost everything • Build quality in
8. Where are we today? @robgodfr • All baseline infrastructure managed in source control • Resource definitions packaged into reusable modules • Consistent infrastructure across accounts • Automated pipelines for release and deployment • Infrastructure tests ensure a low deployment failure rate
9. @robgodfr 1. Reusable Modules for Infrastructure 2. Building Accounts using Modules 3. Management of many AWS Accounts What will we cover?
10. 1. Reusable Modules for Infrastructure
11. Baseline Account Capabilities @robgodfr Federated SSO Auditing Cost Visualisation Tool VPC Networking TLS Certificates Intrusion Detection EC2 Patch Inventory Governance
12. Modules @robgodfr Federated SSO Auditing Cost Visualisation Tool VPC Networking TLS Certificates Intrusion Detection EC2 Patch Inventory Governance
13. Many Modules, Many Accounts @robgodfr
14. Anatomy of a Module @robgodfr Resource Definitions Lambda Source Code Module Test Assets
15. Module Guidelines @robgodfr • Focused on a single capability • Ideally standalone • Should be deployable to any account & region • Should avoid resource name collisions
16. Avoiding name collisions LogsBucket: Type: "AWS::S3::Bucket" Properties: BucketName: !Sub logs-${AWS::AccountId}-${AWS::Region} ... @robgodfr
17. Module Pipelines Validate @robgodfr Test Publish S3 Module Version Repository Module Pipeline Module Source Repository
18. Cloudformation validate-template @robgodfr • Checks: ◦ JSON / YAML syntax ◦ Resource types ◦ Parameters to intrinsic functions • Doesn’t check: ◦ Property names ◦ Types of property values
19. Cfn-Lint - Better Template Validation @robgodfr Cfn-lint - https://github.com/martysweet/cfn-lint • Detects invalid property names • Detects invalid property values • Detects missing required properties • Detects invalid references • and more
20. Testing Modules @robgodfr • Deploy module to a sandbox account • describe-stack and verify stack properties e.g. ◦ status ◦ resources ◦ outputs Sandbox Account Deploy Verify
21. Stack Matchers @robgodfr @pytest.fixture def stack(): return get_stack("stack-name") def test_expected_resource(stack): assert_that(stack, has_resource("AWS::S3::Bucket", "ExpectedBucketName")) def test_expected_output(stack): assert_that(stack, has_output('OutputKeyName', 'ExpectedOutputValue')) https://github.com/financial-times/stack-matchers
22. InSpec 2.0 @robgodfr describe aws_cloudtrail_trail('trail') do
 it { should exist }
 it { should be_encrypted }
 its('s3_bucket_name') { should eq ”trail-bucket” }
 end
23. InSpec 2.0 - Supported Resources @robgodfr • Cloudtrail • Config • EC2 instances & ELBs • Flow Logs • IAM Users, Roles, Policies • KMS • RDS instances • Security Groups • S3 Buckets • SNS Topics • VPCs, Subnets, Route Tables
24. Publishing Modules @robgodfr • An immutable version of a module is copied up to s3 • Simple object prefix convention to distinguish versions e.g. / module / module version / module contents / federated-access / 1.2.98 / ...
25. Module Pipelines Validate @robgodfr Test Publish S3 Module Version Repository Module Pipeline Module Source Repository
26. 2. Building Accounts using Modules
27. Module Deployment Topology @robgodfr eu-west-1 us-east-1 singleton / global
28. Stack Definitions @robgodfr Module Reference Module Config Region Stack Definition
29. Stacks Definition Example --- template_path: modules/vpc/standard-vpc.yaml parameters: VpcName: "FT-Content-Platform-Prod-VPC" VpcCidr: "10.161.40.0/21" PrivateSubnetAZaCidr: "10.161.40.0/24" PrivateSubnetAZbCidr: "10.161.41.0/24" PrivateSubnetAZcCidr: "10.161.42.0/24" ... @robgodfr
30. Anatomy of an Account @robgodfr Stack Definitions Module Manifest Account Def Mf
31. Stack Definitions for Accounts ├── vpc │ ├── eu-west-1 │ │ └── standard-vpc-stack.yaml │ └── us-east-1 │ └── standard-vpc-stack.yaml │ └── vpc-connectivity ├── eu-west-1 │ ├── customer-gateway-iw-stack.yaml │ ├── customer-gateway-pr-stack.yaml │ └── virtual-private-gateway-stack.yaml └── ... @robgodfr
32. Module Manifest modules: - module: auditing version: 1.2.83 - module: certificates version: 1.2.8 - module: cloudhealth version: 1.2.107 - module: federated-access version: 1.2.99 ... @robgodfr
33. Account Pipelines Retrieve Modules @robgodfr Validate Stack Definitions Deploy Stacks S3 Module Version Repository Target Account Account Pipeline Account Source Repository
34. Cross Account Stack Deployment @robgodfr GoCD Server Launch Stacks Cloudformation Stacks Target Account Deployer Account
35. Cross Account Stack Deployment @robgodfr CD Server Deployment Role Instance Profile Launch Stacks Cloudformation Stacks Assume Deployment Role Target Account Deployer Account
36. Multi Stack Deployment • Cloudformation Stack Sets • Sceptre - https://sceptre.cloudreach.com @robgodfr
37. $ sceptre launch-env ├── vpc │ ├── eu-west-1 │ │ └── standard-vpc-stack.yaml │ └── us-east-1 │ └── standard-vpc-stack.yaml │ └── vpc-connectivity ├── eu-west-1 │ ├── customer-gateway-iw-stack.yaml │ ├── customer-gateway-pr-stack.yaml │ └── virtual-private-gateway-stack.yaml └── us-east-1 ├── ... @robgodfr
38. Useful Sceptre Features • Separates a stack’s template and its configuration • Handles switching to deployment role within target account • Resolves dependencies between stacks, passing stack outputs into parameters of dependent stacks • Support for running code as hooks before/after stack executions @robgodfr
39. Account Pipelines Retrieve Modules @robgodfr Validate Stack Definitions Deploy Stacks S3 Module Version Repository Target Account Account Pipeline Account Source Repository
40. 3. Management of Many AWS Accounts
41. What modules are deployed where? @robgodfr
42. Rolling out a new module version @robgodfr governance-1.2.192 eu-west-1 us-east-1 eu-west-1 us-east-1
43. Bump Version in Module Manifest modules: - module: auditing version: 1.2.83 - module: certificates version: 1.2.8 - module: governance version: 1.2.191 -> 1.2.192 @robgodfr
44. Updating many accounts /composer deploy <account(s)> <module> <version> @robgodfr
45. Updating many accounts /composer deploy <account(s)> <module> <version> Deploy v1.2.92 of the auditing module to all accounts: /composer deploy all auditing 1.2.92 @robgodfr
46. Summary
47. @robgodfr Cloudformation Templates Lambda Source Code Test Assets Stack Definitions Module Manifest Accounts Modules Def Mf
48. Module Pipelines Validate @robgodfr Test Publish S3 Module Version Repository Module Pipeline Module Source Repository
49. Account Pipelines Retrieve Modules @robgodfr Validate Stack Definitions Deploy Stacks S3 Module Version Repository Target Account Account Pipeline Account Source Repository
50. Thank You. @robgodfr We're hiring: https://github.com/Financial-Times/careers