Operating ECS in production

Slide 1

Slide 1 text

https://github.com/widdix/aws-cf-templates Operating ECS in production

Slide 2

Slide 2 text

Hello! I am Michael Wittig AWS in Action (2nd ed) cloudonaut.io AWS Community Hero Independent AWS Consultant Twitter @hellomichibye 2

Slide 3

Slide 3 text

ECS Orchestrates Docker containers for you Manages Network and per Task Security 3

Slide 4

Slide 4 text

ECS Cluster 4 ECS Cluster ECS Instance = EC2 Instance running ecs-agent ECS optimized AMI ECS Instance ECS Instance

Slide 5

Slide 5 text

Task Definition 5 ECS Cluster aws ecs run-task \ --count 2 Task Definition Image ... ECS Task 1..N Container ECS Task 1..N Container

Slide 6

Slide 6 text

ECS Scheduling 6 ECS Cluster ECS Instance ECS Instance ECS Instance ECS Task 1..N Container ECS Task 1..N Container Placement constraints and strategies

Slide 7

Slide 7 text

ECS Networking ◎ Public/Private Load Balancer ◎ Elastic Network Interface (ENI) per task ○ Public IP ○ Private IP ○ Per Task Security Group 7

Slide 8

Slide 8 text

ECS Service 8 ECS Cluster Task Definition ECS Task ECS Task ECS Service ◎ Observer ◎ ENI ◎ Load Balancer ◎ Deployment

Slide 9

Slide 9 text

Operating ECS Challenges 9

Slide 10

Slide 10 text

1. Spinning up a cluster 10 Demo

Slide 11

Slide 11 text

Fault Tolerant Auto Scaling Group, Availability Zones 11 Demo

Slide 12

Slide 12 text

2. Updating a cluster New ECS optimized AMIs are released frequently! 12

Slide 13

Slide 13 text

Rolling Update CloudFormation replaces EC2 instances in Auto Scaling Groups in small batches. 13 Demo

Slide 14

Slide 14 text

But what about inflight requests? 14

Slide 15

Slide 15 text

Instance Draining Move all tasks from ECS instance before Instance is terminated. 15 Demo

Slide 16

Slide 16 text

Implementing Instance Draining ◎ Auto Scaling Lifecycle hook ○ Drain Instance ○ Wait until drained ○ Complete Lifecycle hook 16

Slide 17

Slide 17 text

Tasks are not rescheduled once places! Your last batch of ECS instances will end up with 0 tasks! 17

Slide 18

Slide 18 text

3. Scaling a cluster Or adding/removing EC2 instances. 18 Demo

Slide 19

Slide 19 text

We don’t know how many tasks we can schedule! 19

Slide 20

Slide 20 text

20 ECS Cluster Available: CPU 100 Memory 200 Available: CPU 100 Memory 200 Available: CPU 100 Memory 200 Available: CPU 300 Memory 600 Task CPU 200 Memory 200

Slide 21

Slide 21 text

Schedulable Containers 1. Define largest possible task (CPU/memory) 2. For each instance: a. Calculate how many largest possible tasks would fit b. Report to CloudWatch 3. Scale based on the sum of this metric 21 Credits http://garbe.io/blog/2017/04/12/a-better-solution-to-ecs-autoscaling/

Slide 22

Slide 22 text

22 ECS Cluster Available: CPU 100 Memory 200 Schedulable 0 Available: CPU 100 Memory 200 Schedulable 0 Available: CPU 100 Memory 200 Schedulable 0 Available: CPU 300 Memory 600 Schedulable 0 Largest Task CPU 200 Memory 200

Slide 23

Slide 23 text

No CloudWatch Events emitted when task launch failed due to capacity shortage. 23

Slide 24

Slide 24 text

4. Public load balancing 24

Slide 25

Slide 25 text

Public load balancing 25 ECS Cluster ECS Instance ECS Instance ECS Task 1..N Container ECS Task 1..N Container ECS Task 1..N Container Load Balancer (ALB) ◎ Path based ◎ Host based DNS

Slide 26

Slide 26 text

5. Internal service discovery / load balancing 26

Slide 27

Slide 27 text

Internal load balancing 27 ECS Task Frontend LB (internet-facing) ECS Task ECS Task Catalog (internal) ECS Task ECS Task Shopping card (internal) ECS Task ECS Task DNS DNS DNS

Slide 28

Slide 28 text

Catalog Catalog Internal Route 53 (with per task ENI) 28 ECS Task Frontend LB (internet-facing) ECS Task ECS Task ECS Task ECS Task ECS Task ECS Task DNS DNS DNS

Slide 29

Slide 29 text

6. Logging & Monitoring CloudWatch. 29

Slide 30

Slide 30 text

Credits Special thanks to all the people who made and released these awesome resources for free: ◎ Presentation template by SlidesCarnival ◎ Photographs by Pexels 30

Slide 31

Slide 31 text

Thanks! http://bit.ly/amazon-web-services-in-action-2nd-edition https://github.com/widdix/aws-cf-templates https://cloudonaut.io Twitter @hellomichibye Mail [email protected] 31