Stick Shift or Automatic? How to drive a Data Center (MesosCon Asia 2016)

Transcript

1. © 2016 Mesosphere, Inc. All Rights Reserved. 1 Stick Shift or Automatic? How to drive a Data Center Artem Harutyunyan Anand Mazumdar Joris Van Remoortere
2. © 2016 Mesosphere, Inc. All Rights Reserved. 2 • Keep machines healthy • Make sure there are some free resources • Avoid 3AM wakeup calls when hardware fails • Provide top-level services (application deployment platforms, CI, databases) • Operators don’t care about individual workloads and applications. That’s for the developers to worry about. What do datacenter operators do?
3. © 2016 Mesosphere, Inc. All Rights Reserved. 3 History: Mainframe Era
4. © 2016 Mesosphere, Inc. All Rights Reserved. 4 Recent times: Commodity Computing and Virtual Machines
5. © 2016 Mesosphere, Inc. All Rights Reserved. 5 Recent times: Commodity Computing and Virtual Machines
6. © 2016 Mesosphere, Inc. All Rights Reserved. 6 • Keep utilization high • Plan for varying capacity requirements • Deal with new types of failures • Run and maintain distributed applications • Network partitions • Each application does things differently • Coordinating operations between applications Datacenter Operator Problems X X X X X X X X X X X X X X X X X X X X
7. © 2016 Mesosphere, Inc. All Rights Reserved. 7 The Fundamental Theorem of Software Engineering “All problems in computer science can be solved by another level of indirection, except of course for the problem of too many indirections.” David J. Wheeler
8. © 2015 Mesosphere, Inc. All Rights Reserved. master agents coordinator coordinator coordinator 8 MESOS 2-LEVEL SCHEDULING
9. © 2015 Mesosphere, Inc. All Rights Reserved. 9 MESOS LEVEL OF INDIRECTION Mesos (agents) coordinator Mesos (master) coordinator responsible for allocation (and reallocation) of resources
10. © 2016 Mesosphere, Inc. All Rights Reserved. 10 Mesos: Treat machines as cattle not pets
11. © 2016 Mesosphere, Inc. All Rights Reserved. 11 Mesos: Automate With Software Not Humans
12. © 2016 Mesosphere, Inc. All Rights Reserved. 12 A New Form-Factor: The Datacenter Computer
13. © 2016 Mesosphere, Inc. All Rights Reserved. 13 • Deployment • Monitoring, Logging and Debugging • Resource Homogeneity & Limits • Planned downtime/maintenance • Failures • Capacity planning (Quotas, Reservations) • Stateful workloads Datacenter Operator Problems (Revisited)
14. © 2016 Mesosphere, Inc. All Rights Reserved. 14 • There is a common misconception that VM’s solve challenges for cluster management too. • Still need to use a tool like Puppet, Chef etc. for deploying Mesos • Use a meta scheduler (Marathon, Aurora etc.) to deploy the actual services Deployment
15. © 2016 Mesosphere, Inc. All Rights Reserved. 15 • For dependency injection for your application itself, use Docker, Appc • Mesos 0.28 introduced Unified Containerizer Runtime allowing you to run Docker, AppC images (present) and OCI images in the future natively Deployment
16. © 2016 Mesosphere, Inc. All Rights Reserved. 16 • Traditional monitoring solutions relying on host based monitoring won’t work in this new world. Instead, need to monitor based on applications • Need to adopt new mentality of “aggregate” monitoring for entire cluster Monitoring
17. © 2016 Mesosphere, Inc. All Rights Reserved. 17 • Mesos 1.0.0 introduced experimental support for event streaming in V1 Operator API • A subscriber can subscribe to events. Currently supported events are: • TASK_ADDED/TASK_UPDATED • AGENT_ADDED/AGENT_REMOVED • Events are streamed on a persistent connection • There is a talk by Zhitao Li from Uber tomorrow around this! (Extending Mesos for large scale clusters with multiple frameworks) Monitoring
18. © 2016 Mesosphere, Inc. All Rights Reserved. 18 • Remains the same, Aggregate logs per application from various nodes • In addition to application logs, operators have an insight into the overall health of their cluster • Mesos by default stores the stdout/stderr of containers in the task sandbox • No log rotation by default! Logging
19. © 2016 Mesosphere, Inc. All Rights Reserved. 19 • To address concerns around the default logging behavior, Mesos introduced custom logging support in 0.27.0 • Module can be used to change how Mesos redirects stdout/stderr of containers Logging
20. © 2016 Mesosphere, Inc. All Rights Reserved. 20 • Much harder to debug processes in containers and to set up proper tooling around it • Mesos 1.2.0 would introduce debugging capabilities allowing you to remotely “attach” to your running container and also launch new processes in it • Equivalent of docker attach/exec Debugging
21. © 2016 Mesosphere, Inc. All Rights Reserved. 21 • Debugging support would be added to the Agent API via these calls: • LAUNCH_NESTED_CONTAINER_SESSION • ATTACH_CONTAINER_INPUT • ATTACH_CONTAINER_OUTPUT • These would soon be part of the Mesos CLI too • We are also redesigning the Mesos CLI to make it functionally equivalent to the Docker CLI Debugging
22. © 2016 Mesosphere, Inc. All Rights Reserved. 22 • In a static partitioning world, almost all nodes running a service are of the same type • This is not true on shared clusters i.e., 1 CPU != 1 CPU on some other machine. Same is true for memory! • Tag your resources with labels to pass this information to the scheduler Resource Homogeneity
23. © 2016 Mesosphere, Inc. All Rights Reserved. 23 • Running an application on a virtual machine allows the application to grow as much as needed • Containers need to be sized appropriately beforehand. They would be killed if they exceed their share • This is a mindset change and needs to be communicated to application developers Resource Limits
24. © 2015 Mesosphere, Inc. All Rights Reserved. 24 PLANNING FOR FAILURE CHALLENGES [1] BenH Mesos NYC Meetup
25. © 2016 Mesosphere, Inc. All Rights Reserved. 25 • Failures are the norm rather than the exception especially when using commodity hardware • If you are getting paged due to host/rack failure, inherently you are doing something wrong like pinning services to instances • Work with service owners to ensure they have proper load balancer settings • Have some spare capacity to get around these failures Host/Rack Failures
26. © 2016 Mesosphere, Inc. All Rights Reserved. 26 • Use --agent_removal_rate_limit to guard against accidentally losing all agents on your cluster in the event of a large networking event Network Failures
27. © 2016 Mesosphere, Inc. All Rights Reserved. 27 • Historically, Mesos has defined a fixed policy for dealing with network partitions i.e., the framework receives TASK_LOST for all tasks running on the agent. • Frameworks have no way of determining when a task is definitely not running. Network Failures
28. © 2016 Mesosphere, Inc. All Rights Reserved. 28 • Frameworks can opt into this via the PARTITION_AWARE capability • It would then receive new task statuses instead of the default TASK_LOST previously • TASK_UNREACHABLE • TASK_DROPPED (Terminal) • TASK_GONE (Terminal) • TASK_GONE_BY_OPERATOR • TASK_UNKNOWN Network Failures
29. © 2016 Mesosphere, Inc. All Rights Reserved. 29 • Frameworks require visibility into the planned downtime in order to meet SLA’s or to ensure uninterrupted services for their end users • An operator can schedule maintenance for nodes in advance. This would result in the schedulers being notified about the unavailability via Inverse Offers Planned Downtime/Maintenance
30. © 2016 Mesosphere, Inc. All Rights Reserved. 30 • Quota • A mechanism for guaranteeing that a role will receive a certain minimum amount of resources • Reservations • Similar to Quota, but tied to a particular agent Capacity Planning
31. © 2016 Mesosphere, Inc. All Rights Reserved. 31 • Persistent Volumes • Mesos provides a mechanism to create a persistent volume from disk resources • When the task exits the persistent volume is offered back to the framework Stateful Workloads
32. © 2016 Mesosphere, Inc. All Rights Reserved. 32 • Availability • Reliability • Safety • Maintainability Principles of Fault Tolerance Distributed Systems Principles and Paradigms Tanenbaum et al
33. © 2016 Mesosphere, Inc. All Rights Reserved. 33 • Treat machines as cattle not pets • No machine should be special, we should be able to use data center resources interchangeably • Automate with software not with humans • Let the software schedule software, handle failures, improve utilization and handle maintenance Summary