Upgrade to Pro — share decks privately, control downloads, hide ads and more …

And you thought you knew EC2

And you thought you knew EC2

EC2 features, best practices, and pro tips. Presented at AWS Community Day 2017 in San Francisco.


Ben Whaley

June 15, 2017

More Decks by Ben Whaley

Other Decks in Technology


  1. Ben Whaley | @iAmTheWhaley … And You Thought You Knew

  2. Instance Types Family Intended use C5 Compute-optimized with Intel Skylake*

    R4 Memory-optimized, up to 488GiB** I3 High I/O - Watch for sharp corners F1 FPGA P2 GPU - 2x perf of G2 at 1.5x the cost D2 Dense Storage, up to 48TiB HDD & 10GiB network * Coming soon? ** Only 244GiB currently
  3. 4 EBS Volume Types Type Description I/O Throughput Cost io1

    Provisioned IOPS SSD Highest High Highest gp2 General purpose SSD High Low(ish) Medium st1 Throughput-optimized HDD Low Highest Low sc1 Non-optimized HDD Low Medium Lowest
  4. Miscellaneous improvements & features • IPv6! • VPC endpoints for

    DynamoDB, S3 • Lightsail - EC2 quick start • Elastic GPUs • New regions • 2016 - Ohio, Canada, London, Mumbia, Seoul • Ningxia, Paris, Stockholm
  5. EC2 Systems Manager Superpowers for EC2 instances and on-premises systems.

    • Remote command execution with Run Commands • Controlled secrets and configuration data with the Parameter Store • Periodic tasks with the State Manager and Maintenance Windows • Stepwise Automation workflows for initializing nodes • Collect and query Inventory and Patch status
  6. Systems Manager Essentials • The SSM agent • Open source

    (Golang) executable for Linux and Windows • Available for cloud and on-premises systems • Assign IAM role with permissions to interface with SSM API • Install at boot or on existing systems • Polls for commands to execute • All actions recorded in CloudTrail (e.g. immutable audit trail) • Trigger SNS, Lambda from Systems Manager events • Store command history and output to S3 • Fine-grained access control to Run Commands • Integration with Config to track changes over time
  7. Systems Manager Documents JSON Schema describing actions for the systems

    manager { "schemaVersion":"2.0", "description":"Run a script", "parameters":{ "commands":{ "type":"String", “description”:"Commands to run" } }, "mainSteps":[ { "action":"aws:runShellScript", "name":"runShellScript", "inputs":{ "runCommand":"{{ commands }}" } } ] }
  8. Quick & dirty SSM demo: 1. Scale up ASG of

    managed instances 2. Run a command on remote agents 3. Store and retrieve a parameter Virtual Private Cloud Private subnet EC2 Instances Systems Manager ssm-agent Workstations
  9. Five Ways to Provision Instances Userdata 1

  10. Five Ways to Provision Instances Bake an AMI 2 (Alternative:

  11. Five Ways to Provision Instances Configuration management runs at boot,

    registers with server, converges a configuration 3
  12. Five Ways to Provision Instances Autoscaling lifecycle hook —> CloudWatch

    event —> Run Command —> execute provisioning documents 4 (Alternative: CloudWatch event —> Lambda)
  13. Five Ways to Provision Instances AWS::CloudFormation::Init 5

  14. via fbrnc.net

  15. Application Load Balancer • Almost as good as HAProxy or

    NGINX. Almost. • Host- and path-based routing • Additional metrics (# active connections, total traffic) • Improved health checks • Websockets • HTTP/2 • Integration with X-Ray (adds X-Amzn-Trace-Id header) • Integration with ECS • Integration with WAF
  16. One solution: route53 health checks with autoscaling lifecycle events Problems:

    Rotating IP addresses Traffic surges ELBs
  17. ELB IP 1 ELB IP 2 Clients Clients Clients Clients

    Moar clients? Feeling… weak… must… scale up. ELB IP 1 ELB IP 2 Clients Clients Couple of clients? Come at me, bro.
  18. ELB IP 3 ELB IP 4 Clients Clients I sense

    a disturbance in the force… as if hundreds of clients suddenly cried out in pain Clients Clients ELB IP 1 ELB IP 2 Bueller? Bueller? … moments later
  19. Clients Lookup A record for myservice.example.com Route53 IP 1 IP

    2 Clients Client Connect to IP 1 Node with IP 1 Client Connect to IP 2 Node with IP 2 health checks ASG
  20. Clients Connect to IP 1 Clients Connect to IP 2

    ASG Node with IP 1 Node with IP 2 Clients Clients Clients Clients Clients Clients 1. Autoscale via custom CloudWatch metric 2. New node boots with IP 3 3. Autoscaling lifecycle hook adds IP 3 to myservice.example.com A record Moar clients!
  21. Clients Connect to IP 1 Clients Connect to IP 2

    ASG Node with IP 1 Node with IP 2 Clients Clients Clients Clients Clients Clients Clients Connect to IP 3 Node with IP 3 Clients Clients Clients
  22. Pros: Solves problems Cons: Causes problems

  23. EC2 Container Service Updates • ECR - best registry to

    use for AWS container workloads • Support for volumes • CloudWatch metrics for CPU and memory utilization across the cluster (set alarms for autoscaling) • IAM roles for ECS tasks • Blox allows custom schedulers (github.com/blox/blox) • 3rd party tooling (Convox, Empire) • Integration with ALB • Run tasks on a schedule • Execute tasks in response to CloudWatch events
  24. Curated Pro Tips

  25. Autoscaling groups of size one for self-healing and resilience.

  26. Many EC2 IAM actions do not support resource-level permissions. Exercise

  27. Use the BurstBalance CloudWatch metric to monitor I/O credit balance

    for gp2, st1, sc1 EBS volumes.
  28. Network throughput increases substantially with instance type. (Don't forget to

    enable enhanced networking)
  29. Explicitly request SSD ephemeral disks when desired. Otherwise you may

    not get them.
  30. Require MFA for SSH access.

  31. Improve your SSH experience with ControlPersist.

  32. Use ssh -D and the SwitchyOmega Chrome extension for convenient

    access to services in a private network.
  33. Running multiple apps per instance? Use AssumeRole to assign granular

    permissions to each app.
  34. Protect the EC2 metadata and userdata.

  35. Enable Fail2ban to block IPs with failed login attempts.

  36. Use Linux >= 4.4 for best results.

  37. Capture userdata output to a file. #!/bin/bash -x exec >

    /var/log/userdata.log 2>&1
  38. Coming this August!

  39. Thanks! Ben Whaley @iAmTheWhaley