XP Days Ukraine 2019 - Kubernetes vs DC/OS. Let the fight begin!

Kubernetes is the most popular orchestrator for large microservices system. It provides possibility for automating application deployment, scaling and management. But does it really a magic wand, which may solve all your problems, or it would be better to use another solutions? We will compare two giant players in the orchestration and clusters building area: DC/OS (aka. Mesosphere) and Kubernetes from the developer point of view. Can you predict who will win? So, let the fight begin!

The authors of this “battle” have come a long way from DC/OS to Kubernetes and will try to convey their experience to the audience

Transcript

1. DC/OS Kubernetes Let the fight begin!
2. About us Software Engineer at Luxoft, Ukraine Software Engineer at Luxoft, Ukraine
3. The main problems in microservices world - running thousands of services in the same infrastructure - avoid manual restarting when service will be in failed state - organization of service discovery (communication between services in the cluster) - 3rd party services (backup & restore, monitoring, logs rotation, etc.)
4. Choice criteria or What do we want? - scalable and manageable cluster for microservices - inter service communication - SQL/NoSQL DB support (built-in or as a separate service) - Big Data stack (Spark, HDFS, etc.) - Use any language for each microservice - Easy deployment
5. Choose your fighter!
6. Round 1 General set up and installations
7. Architecture - DC/OS Source: shorturl.at/nvEM4
8. Source: shorturl.at/nvEM4 Architecture - Kubernetes
9. Installation - Bare metal (shell scripts & ansible recipes) - Cloud native + Web UI + scripts (AWS EC2 based approach) + IaC (terraform recipes) + CLI tools (not for all vendors) - miniDC/OS for testing - Bare metal (shell scripts & ansible recipes) - Cloud native: + Web UI + CLI tools + IaC (terraform recipes) - miniKube for testing
10. 3rd party services installation
11. API clients - CLI tool: dcos - Marathon API client (Java, Python) - Mesos API client (Java, Python) - Pure REST API in other cases - CLI tools: + kubectl + kubeadm + kubefedctl - API Client libraries: + Go + Python + Java + JavaScript
12. Round 2 For developers
13. Workloads - Pods - Deployments - Jobs - Cron Jobs - Mesos Tasks - Pods - Controllers + Deployments + Jobs + Cron Jobs + ReplicaSet + DaemonSet + etc.
14. Deployment descriptors - Catalog - JSON and YAML manifests - Helm Charts - JSON and YAML manifests
15. Environment isolation - Docker - Vagrant env (via Mesos tasks) - containerd (via Mesos tasks) - Pod - Docker - rkt - Pod * Data from the https://stackshare.io/stackups/docker-vs-rkt
16. Docker Registry - Internal private (out of the box) - ECR (Elastic container registry) - ACR (Azure container registry) - External (e.g. JFrog) - ECR (Elastic container registry) - native support - GCR (Google container registry) - native support - ACR (Azure container registry) - IBM Cloud container registry - Any other private or public registries (e.g. JFrog)
17. HEALTH CHECKS - Health Check (HTTP, TCP, COMMAND and Mesos level health checks) - Liveness and Readiness probes (HTTP, Command and TCP)
18. Service discovery and Service Mesh support - Internals (influxdb.marathon.mesos.com) - Linkerd - via message bus - k8s native (service.namespace) - service mesh implementations: + Istio + Linkerd + Consul - via message bus
19. Storage Types - NFS Server - Local Persistent Volume - AWS EBS volumes out-of-the-box support - Other types of volumes should be mounted manually - Network file system - External Persistent Volumes from different cloud vendors (EBS, EFS, AzureFile, AzureDisk, etc.) - dynamic or static provisioning - k8s native storages (secret config map, emptyDir, etc.) *https://kubernetes.io/docs/concepts/storage/#types-of-volumes
20. Round 3 For administrators
21. Security and permissions Open source version: - standard users and groups (something like linux based) - Permissions via cmd (dcos security org groups grant [OPTIONS] G_ID) or HTTP API Enterprise components: - standard users and groups - secrets - LDAP - public and private zones - namespaces - roles - cluster roles - service accounts - role binding access control (RBAC) - secrets - config maps Bonus: You can set up your cloud user as the user in k8s
22. Networking - IP connectivity to containers (host/bridge) - build-in DNS based service discovery - cluster identity options (install 1 service on single node, N services per group, LIKE `rack_id` inside cluster, UNLIKE...) - Load balancing (simple built-in, HAproxy based...) Main approaches: - container-to-container inside pod - pod-to-pod - pod-to-service - external-to-service Implementation of this model may be different depends on the vendor (based on CNI plug-in)
23. Logging and Monitoring EQUALITY ADD-ONS
24. Autoscaling Services autoscaling: - marathon autoscaler Cluster autoscaling: - autoscaling groups Services autoscaling: - horizontal Pods Autoscaller NOTE: do not forget to add resources limits Cluster autoscaling: - autoscaling groups
25. Backup and restore Out of the box: - The entire DC/OS cluster (enterprise version) - Cassandra - MongoDB Out of the box: - Backup master and etcd External tools: - Velero - Kasten Cloud based: - Backup databases with cloud specific approach
26. Cluster federation & Hybrid cloud - Federation does not supported - Hybrid cloud available for enterprise version - Federation supported (KubeFed - currently in alpha state) - Hybrid clouds supported
27. Finish him! Pros: + many things are out of the box (DB, big data stack) + possibility to run non-docker stuff + good customization possibilities + great support from developers side (in case of enterprise version) + fast project growth Pros: + fast development in case of cloud dependent installation + best cloud native support ever + good API clients for different languages + proven scalability (GitHub uses k8s) + big open-source community + cluster federation & hybrid clouds Cons: - enterprise version is very expensive - limited support by cloud vendors - consist of different open-source projects - marathon - default orchestrator - worse than k8s - hard to understand for ordinary developer - smaller community then for kubernetes Cons: - open-source (wait for fix or contribute) - no 3rd party support out of the box - limited possibility for backup & restore - helm charts vs. catalog :) - hard to understand for ordinary developer
28. Fatality: K8S instead of Marathon in DCOS
29. Source: https://stackshare.io/stackups/dcos-vs-kubernetes Brutality: Internet activity
30. requirements win!
31. Find these slides here Thank you!