• Using standard OS features (e.g., cgroups, namespaces) • Pluggable architecture allowing customization and extension Currently supported containerizers
support, Persistent volumes, etc. • Code duplication → bugs, maintenance issue Coordination needed between two containerizers • Global resources like GPU, net_cls handles, etc. 5 Maintaining multiple containerizers is hard
Depended on Docker Daemon • Stability concern • Extra dependency Support other image formats • Appc, CVMFS, ... Why don’t we base on Docker containerizer
Launcher creates Subprocess Container Process execve() Script = Isolator::prepare() * Create a cgroup for the container in memory cgroup hierarchy: /sys/fs/cgroup/memory/mesos/… * Start listening for OOM event Isolator::isolate(pid) Block on pipe Move ‘pid’ to the memory cgroup just created Invoke ‘Script’ Exec the executor Signal the Child to continue
Container Process Isolator::update() Change cgroup control: memory.limit_in_bytes Sending a new Task to Executor, ‘resources’ of the Executor changes Send Task to Executor
Container Process Isolator::cleanup() Remove the memory cgroup associated with the container Shutdown Executor or kill Task Destroy container Container terminated
on Mesos without a Docker daemon installed! • One less dependency in your stack • Agent restart handled gracefully, task not affected • Compose well with all existing isolators • Easier to add extensions Container image support Unified containerizer
layers • Backend: assemble rootfs from image layers • E.g., copy, overlayfs, bind, aufs Store can be extended • Currently supported: Docker, Appc • Plan to support: CVMFS (join the MesosCon talk!) Provisioner Unified containerizer
a container network spec proposed by CoreOS • Simpler and less dependencies than Docker CNM • K8s supported it • Rich plugins from network vendors Main advantages: • Clear separation between container and network management • IPAM has its own pluggable interface Container network support Unified containerizer
as root have restricted set of capabilities • Containers running as non-root can have certain capabilities User namespace • Full privileges inside the user namespace (e.g., uid=0) • Normal unprivileged user ID in the host user namespace Container security Unified containerizer
sub-containers • Sub-containers have container images (e.g., Docker) • When executor dies, sub-containers will be destroyed Use cases: • K8s on Mesos • Jenkins on Mesos • Native POD support Nested container Future work
More secure containers • VM workload • OpenStack integration Possible implementations • A new containerizer? • A plugin to unified containerizer? Goal: launching Mesos tasks/executors in VMs
images • Cached in different places Pluggable fetcher • A fetcher for each URI scheme • Allow URIs with custom scheme • Fetcher is modularized and can be extended (e.g., p2p) Unified caching • All artifacts are cached the same way • Content addressable storage • Garbage collection • Pre-fetching support Unified fetching and caching Future work
between isolators • Sharing information between isolators is not possible • Upgrading isolators in a backward compatible way is hard Potential solutions • Explicit isolator dependency, both data and control • Isolator versioning, and version checkpointing • Isolator registry? Better abstraction for isolators Future work