machines healthy • Make sure there are some free resources • Avoid 3AM wakeup calls when hardware fails • Provide top-level services (application deployment platforms, CI, databases) • Operators don’t care about individual workloads and applications. That’s for the developers to worry about. What do datacenter operators do?
utilization high • Plan for varying capacity requirements • Deal with new types of failures • Run and maintain distributed applications • Network partitions • Each application does things differently • Coordinating operations between applications Datacenter Operator Problems X X X X X X X X X X X X X X X X X X X X
Theorem of Software Engineering “All problems in computer science can be solved by another level of indirection, except of course for the problem of too many indirections.” David J. Wheeler
is a common misconception that VM’s solve challenges for cluster management too. • Still need to use a tool like Puppet, Chef etc. for deploying Mesos • Use a meta scheduler (Marathon, Aurora etc.) to deploy the actual services Deployment
dependency injection for your application itself, use Docker, Appc • Mesos 0.28 introduced Unified Containerizer Runtime allowing you to run Docker, AppC images (present) and OCI images in the future natively Deployment
monitoring solutions relying on host based monitoring won’t work in this new world. Instead, need to monitor based on applications • Need to adopt new mentality of “aggregate” monitoring for entire cluster Monitoring
1.0.0 introduced experimental support for event streaming in V1 Operator API • A subscriber can subscribe to events. Currently supported events are: • TASK_ADDED/TASK_UPDATED • AGENT_ADDED/AGENT_REMOVED • Events are streamed on a persistent connection • There is a talk by Zhitao Li from Uber tomorrow around this! (Extending Mesos for large scale clusters with multiple frameworks) Monitoring
the same, Aggregate logs per application from various nodes • In addition to application logs, operators have an insight into the overall health of their cluster • Mesos by default stores the stdout/stderr of containers in the task sandbox • No log rotation by default! Logging
address concerns around the default logging behavior, Mesos introduced custom logging support in 0.27.0 • Module can be used to change how Mesos redirects stdout/stderr of containers Logging
harder to debug processes in containers and to set up proper tooling around it • Mesos 1.2.0 would introduce debugging capabilities allowing you to remotely “attach” to your running container and also launch new processes in it • Equivalent of docker attach/exec Debugging
support would be added to the Agent API via these calls: • LAUNCH_NESTED_CONTAINER_SESSION • ATTACH_CONTAINER_INPUT • ATTACH_CONTAINER_OUTPUT • These would soon be part of the Mesos CLI too • We are also redesigning the Mesos CLI to make it functionally equivalent to the Docker CLI Debugging
a static partitioning world, almost all nodes running a service are of the same type • This is not true on shared clusters i.e., 1 CPU != 1 CPU on some other machine. Same is true for memory! • Tag your resources with labels to pass this information to the scheduler Resource Homogeneity
an application on a virtual machine allows the application to grow as much as needed • Containers need to be sized appropriately beforehand. They would be killed if they exceed their share • This is a mindset change and needs to be communicated to application developers Resource Limits
are the norm rather than the exception especially when using commodity hardware • If you are getting paged due to host/rack failure, inherently you are doing something wrong like pinning services to instances • Work with service owners to ensure they have proper load balancer settings • Have some spare capacity to get around these failures Host/Rack Failures
Mesos has defined a fixed policy for dealing with network partitions i.e., the framework receives TASK_LOST for all tasks running on the agent. • Frameworks have no way of determining when a task is definitely not running. Network Failures
can opt into this via the PARTITION_AWARE capability • It would then receive new task statuses instead of the default TASK_LOST previously • TASK_UNREACHABLE • TASK_DROPPED (Terminal) • TASK_GONE (Terminal) • TASK_GONE_BY_OPERATOR • TASK_UNKNOWN Network Failures
require visibility into the planned downtime in order to meet SLA’s or to ensure uninterrupted services for their end users • An operator can schedule maintenance for nodes in advance. This would result in the schedulers being notified about the unavailability via Inverse Offers Planned Downtime/Maintenance
• A mechanism for guaranteeing that a role will receive a certain minimum amount of resources • Reservations • Similar to Quota, but tied to a particular agent Capacity Planning
Volumes • Mesos provides a mechanism to create a persistent volume from disk resources • When the task exits the persistent volume is offered back to the framework Stateful Workloads
machines as cattle not pets • No machine should be special, we should be able to use data center resources interchangeably • Automate with software not with humans • Let the software schedule software, handle failures, improve utilization and handle maintenance Summary
hatred
cscengineer
mayamasaki68
nwiizo
lehupa
news_it_enj
layerx
10xinc
databricksjapan
jacopen
yanzm
yukukotani
kinosuke01
konakalab
garrettdimon
jonyablonski
soudai
lemiorhan
mclloyd
aleyda
tanoku
katiecoart
samanthasiow
erikaheidi
axbom
