Nested Containers & Task Groups

Keynote - MesosCon Asia 2016

Transcript

1. Nested Containers & Task Groups MesosCon Asia 2016
2. About Us Vinod Kone • Apache Mesos Committer (since ‘12) • Engineer Manager @ Mesosphere • Previously: Tech Lead @ Twitter • PhD @ UCSB Jie Yu • Apache Mesos Committer (since ‘14) • Tech Lead @ Mesosphere • Previously: Sr. Engineer @ Twitter • PhD @ UMich
3. History of Containers in Mesos 0.10.0 0.18.0 Cgroups support (cpu, mem) Pluggable Isolators 0.20.0 Docker containers + Network monitoring 0.28.0 Docker containers without Docker daemon 1.0.0 CNI 1.1.0 Nested containers
4. Architecture Scheduler A Master Agent Agent Master Master Scheduler B Agent Agent
5. Primitives: Executors and Tasks • Schedulers launch tasks ◦ Tasks specify executor • Executors run tasks ◦ Multiple tasks per executor ◦ Executor per container Agent Executor Executor Task Task Task Task Container Container
6. Limitations • Managing a group of co-located tasks is hard ◦ All or nothing semantics • One image per container ◦ Tasks cannot have a different image than executor • No resource isolation between tasks within executor
7. A new primitive: A Mesos “Pod” • Ability to manage co-located and co-scheduled tasks as a single unit • Tasks should be able to share some but not all resources ◦ e.g., Sharing: Network namespace, Volumes ◦ e.g., Exclusive: Container Image • Dynamically updatable ◦ e.g., Resources can be added or removed at run time • Hierarchical isolation
8. Use case #1: Side-car container memcached log saver
9. Use case #2: Transient container backup
10. Use case #3: Hierarchical container kublet k8s pod
11. Task Groups & Nested Containers
12. New Primitive: Task Group • Schedulers can launch a group of tasks together ◦ ‘TaskGroupInfo’ object ◦ ‘LAUNCH_GROUP’ offer operation • Executors atomically received task group ◦ ‘LAUNCH_GROUP’ event
13. New Primitive: Nested Container • Support arbitrary levels of nesting • Re-use all existing isolators • Allow dynamically creation of nested containers Agent Container Executor Container Container Task Task
14. New Agent API for Nested Containers message agent::Call { enum Type { // Calls for managing nested containers // under an executor's container. LAUNCH_NESTED_CONTAINER = 14; WAIT_NESTED_CONTAINER = 15; KILL_NESTED_CONTAINER = 16; } }
15. Launch a Nested Container Container Executor Mesos Agent Containerizer LAUNCH Nginx
16. Wait for a Nested Container Container Executor Mesos Agent Containerizer WAIT Nginx Exit Status = 0
17. Destroy a Nested Container Container Executor Mesos Agent Containerizer KILL Nginx
18. Arbitrary Levels of Nesting Container Executor Nginx Mesos Agent Containerizer LAUNCH Debug
19. Nested containerization semantics • Controlled by the isolators configured by the operator ◦ Flexible ◦ Customizable • Current default behavior ◦ NET and UTS namespaces are shared ◦ MNT namespace is not shared ◦ Cgroup is shared
20. New executor: Default executor • Handles task groups • Uses v1 HTTP API • Will eventually replace command executor • Current restart policy ◦ Suicide if any container terminates with non zero exit status ◦ Will support a more sophisticated restart policy later
21. Demo
22. Thanks
23. Scheduler API
24. Executor API