M****services on AWS ECS

Presentation from AWS User Group (2015.09.24) in Warsaw by Maciej Winnicki from X-Team.

Transcript

1. SAY MICROSERVICES ONE MORE GODDAMN TIME
2. ON M****SERVICES AWS ECS
3. Maciej Winnicki mthenw
4. What is AWS EC2 Container Service? How does it work? Service discovery Configuration Deployment Scaling What is missing?
5. What is AWS EC2 Container Service?
6. … is a highly scalable, fast, container management service that makes it easy to run, stop, and manage Docker containers on a cluster of Amazon EC2 instances. http://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html “
7. Amazon Docker As A Service https://www.expeditedssl.com/aws-in-plain-english
8. How does it work?
9. AZ AZ EC2 EC2 EC2 EC2
10. Cluster EC2 EC2 EC2 EC2
11. Cluster EC2 EC2 EC2 EC2 ECS Agent ECS Agent ECS Agent ECS Agent
12. Cluster EC2 EC2 EC2 EC2 ECS Agent ECS Agent ECS Agent ECS Agent Scheduler
13. Cluster EC2 EC2 EC2 EC2 ECS Agent ECS Agent ECS Agent ECS Agent Scheduler Task
14. Cluster EC2 EC2 EC2 EC2 ECS Agent ECS Agent ECS Agent ECS Agent Scheduler Task Task
15. Cluster EC2 EC2 EC2 EC2 ECS Agent ECS Agent ECS Agent ECS Agent Scheduler Task Task Task
16. Cluster EC2 EC2 EC2 EC2 ECS Agent ECS Agent ECS Agent ECS Agent Scheduler Task Task Task
17. Cluster EC2 EC2 EC2 EC2 ECS Agent ECS Agent ECS Agent ECS Agent Scheduler Task Task Task Docker container Docker container
18. Cluster EC2 EC2 EC2 EC2 ECS Agent ECS Agent ECS Agent ECS Agent Scheduler Task Task Task Docker container Docker container Task Definition {“containerDefinitions": [ { "name": "web", "image": "nginx", "cpu": 99, "memory": 100, "portMappings": [ { "containerPort": 80, "hostPort": 80 } ] } ]}
19. Cluster EC2 EC2 EC2 EC2 ECS Agent ECS Agent ECS Agent ECS Agent Scheduler Task
20. Cluster EC2 EC2 EC2 EC2 ECS Agent ECS Agent ECS Agent ECS Agent Scheduler Task Service 1 Task Service 1 Task Service 1
21. Cluster EC2 EC2 EC2 EC2 ECS Agent ECS Agent ECS Agent ECS Agent Scheduler Task Service 1 Task Service 1 Task Service 1 ELB
22. Cluster EC2 EC2 EC2 EC2 ECS Agent ECS Agent ECS Agent ECS Agent Scheduler Task Service 1 Task Service 1 Task Service 1 ELB :80 :8001 :8001 :8001
23. Cluster EC2 EC2 EC2 EC2 ECS Agent ECS Agent ECS Agent ECS Agent AWS ECS Task Service 1 Task Service 1 Task Service 1 ELB Key/Value Store Cluster Management Engine Agent Communication Service API http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html
24. Microservices
25. ECS Cluster AZ EC2 ECS Agent EC2 ECS Agent AZ EC2 ECS Agent EC2 ECS Agent
26. ECS Cluster (c4.xlarge) AZ EC2 ECS Agent EC2 ECS Agent AZ EC2 ECS Agent EC2 ECS Agent Stateful cluster (non-ECS, m4.xlarge) EC2 w/ EBS EC2 w/ EBS EC2 w/ EBS MongoDB MongoDB ElasticSearch ElasticSearch
27. Service discovery Configuration https://www.consul.io
28. Service discovery via DNS or HTTP Key/Value Storage Failure Detection https://www.consul.io
29. Service discovery Configuration https://www.consul.io
30. ECS Cluster AZ EC2 ECS Agent Consul EC2 ECS Agent Consul AZ EC2 ECS Agent Consul EC2 ECS Agent Consul Stateful cluster EC2 w/ EBS EC2 w/ EBS EC2 w/ EBS MongoDB MongoDB ElasticSearch ElasticSearch Consul Consul Consul
31. ECS Cluster AZ EC2 ECS Agent Consul (Server) EC2 ECS Agent Consul (Server) AZ EC2 ECS Agent Consul (Server) EC2 ECS Agent Consul Stateful cluster EC2 w/ EBS EC2 w/ EBS EC2 w/ EBS MongoDB MongoDB ElasticSearch ElasticSearch Consul Consul Consul Docker container Docker container Consul DNS
32. Service discovery via DNS Consul as a default DNS for every container
33. ECS Cluster AZ EC2 ECS Agent Consul (Server) EC2 ECS Agent Consul (Server) AZ EC2 ECS Agent Consul (Server) EC2 ECS Agent Consul Stateful cluster EC2 w/ EBS EC2 w/ EBS EC2 w/ EBS MongoDB MongoDB ElasticSearch ElasticSearch Consul Consul Consul Registrator New task Consul Docker (socket) Register new task/service
35. Service discovery Configuration https://www.consul.io
36. JSON file for local development Consul Key/Value for any other environment Dynamic feature flags stored in Consul K/V
37. Deployment
38. build image commit click! run ecs deploy deploy ECS
39. Small docker images! (e.g. mhart/alpine-node: 35MB) ECS strategy (add before remove)
40. Scaling
42. Cluster EC2 EC2 EC2 EC2 ECS Agent ECS Agent ECS Agent ECS Agent Task (Service 1) Task (Service 1) Task (Service 1) ELB HAProxy
43. What is still missing?
44. Exactly one task per instance Dynamic port allocation on ELB Stateful services