And you thought you knew EC2

EC2 features, best practices, and pro tips. Presented at AWS Community Day 2017 in San Francisco.

1. Ben Whaley | @iAmTheWhaley … And You Thought You Knew EC2
2. Instance Types Family Intended use C5 Compute-optimized with Intel Skylake* R4 Memory-optimized, up to 488GiB** I3 High I/O - Watch for sharp corners F1 FPGA P2 GPU - 2x perf of G2 at 1.5x the cost D2 Dense Storage, up to 48TiB HDD & 10GiB network * Coming soon? ** Only 244GiB currently
3. 4 EBS Volume Types Type Description I/O Throughput Cost io1 Provisioned IOPS SSD Highest High Highest gp2 General purpose SSD High Low(ish) Medium st1 Throughput-optimized HDD Low Highest Low sc1 Non-optimized HDD Low Medium Lowest
4. Miscellaneous improvements & features • IPv6! • VPC endpoints for DynamoDB, S3 • Lightsail - EC2 quick start • Elastic GPUs • New regions • 2016 - Ohio, Canada, London, Mumbia, Seoul • Ningxia, Paris, Stockholm
5. EC2 Systems Manager Superpowers for EC2 instances and on-premises systems. • Remote command execution with Run Commands • Controlled secrets and configuration data with the Parameter Store • Periodic tasks with the State Manager and Maintenance Windows • Stepwise Automation workflows for initializing nodes • Collect and query Inventory and Patch status
6. Systems Manager Essentials • The SSM agent • Open source (Golang) executable for Linux and Windows • Available for cloud and on-premises systems • Assign IAM role with permissions to interface with SSM API • Install at boot or on existing systems • Polls for commands to execute • All actions recorded in CloudTrail (e.g. immutable audit trail) • Trigger SNS, Lambda from Systems Manager events • Store command history and output to S3 • Fine-grained access control to Run Commands • Integration with Config to track changes over time
7. Systems Manager Documents JSON Schema describing actions for the systems manager { "schemaVersion":"2.0", "description":"Run a script", "parameters":{ "commands":{ "type":"String", “description”:"Commands to run" } }, "mainSteps":[ { "action":"aws:runShellScript", "name":"runShellScript", "inputs":{ "runCommand":"{{ commands }}" } } ] }
8. Quick & dirty SSM demo: 1. Scale up ASG of managed instances 2. Run a command on remote agents 3. Store and retrieve a parameter Virtual Private Cloud Private subnet EC2 Instances Systems Manager ssm-agent Workstations
9. Five Ways to Provision Instances Userdata 1
10. Five Ways to Provision Instances Bake an AMI 2 (Alternative: half-baked)
11. Five Ways to Provision Instances Configuration management runs at boot, registers with server, converges a configuration 3
12. Five Ways to Provision Instances Autoscaling lifecycle hook —> CloudWatch event —> Run Command —> execute provisioning documents 4 (Alternative: CloudWatch event —> Lambda)
13. Five Ways to Provision Instances AWS::CloudFormation::Init 5
14. via fbrnc.net
15. Application Load Balancer • Almost as good as HAProxy or NGINX. Almost. • Host- and path-based routing • Additional metrics (# active connections, total traffic) • Improved health checks • Websockets • HTTP/2 • Integration with X-Ray (adds X-Amzn-Trace-Id header) • Integration with ECS • Integration with WAF
16. One solution: route53 health checks with autoscaling lifecycle events Problems: Rotating IP addresses Traffic surges ELBs
17. ELB IP 1 ELB IP 2 Clients Clients Clients Clients Moar clients? Feeling… weak… must… scale up. ELB IP 1 ELB IP 2 Clients Clients Couple of clients? Come at me, bro.
18. ELB IP 3 ELB IP 4 Clients Clients I sense a disturbance in the force… as if hundreds of clients suddenly cried out in pain Clients Clients ELB IP 1 ELB IP 2 Bueller? Bueller? … moments later
19. Clients Lookup A record for myservice.example.com Route53 IP 1 IP 2 Clients Client Connect to IP 1 Node with IP 1 Client Connect to IP 2 Node with IP 2 health checks ASG
20. Clients Connect to IP 1 Clients Connect to IP 2 ASG Node with IP 1 Node with IP 2 Clients Clients Clients Clients Clients Clients 1. Autoscale via custom CloudWatch metric 2. New node boots with IP 3 3. Autoscaling lifecycle hook adds IP 3 to myservice.example.com A record Moar clients!
21. Clients Connect to IP 1 Clients Connect to IP 2 ASG Node with IP 1 Node with IP 2 Clients Clients Clients Clients Clients Clients Clients Connect to IP 3 Node with IP 3 Clients Clients Clients
22. Pros: Solves problems Cons: Causes problems
23. EC2 Container Service Updates • ECR - best registry to use for AWS container workloads • Support for volumes • CloudWatch metrics for CPU and memory utilization across the cluster (set alarms for autoscaling) • IAM roles for ECS tasks • Blox allows custom schedulers (github.com/blox/blox) • 3rd party tooling (Convox, Empire) • Integration with ALB • Run tasks on a schedule • Execute tasks in response to CloudWatch events
24. Curated Pro Tips
25. Autoscaling groups of size one for self-healing and resilience.
26. Many EC2 IAM actions do not support resource-level permissions. Exercise caution.
27. Use the BurstBalance CloudWatch metric to monitor I/O credit balance for gp2, st1, sc1 EBS volumes.
28. Network throughput increases substantially with instance type. (Don't forget to enable enhanced networking)
29. Explicitly request SSD ephemeral disks when desired. Otherwise you may not get them.
30. Require MFA for SSH access.
31. Improve your SSH experience with ControlPersist.
32. Use ssh -D and the SwitchyOmega Chrome extension for convenient access to services in a private network.
33. Running multiple apps per instance? Use AssumeRole to assign granular permissions to each app.
34. Protect the EC2 metadata and userdata.
35. Enable Fail2ban to block IPs with failed login attempts.
36. Use Linux >= 4.4 for best results.
37. Capture userdata output to a file. #!/bin/bash -x exec > /var/log/userdata.log 2>&1
38. Coming this August!
39. Thanks! Ben Whaley @iAmTheWhaley