ContainerCamp: Kubernetes and Container Management

ContainerCamp: Kubernetes and Container Management

Slides from Container Camp 2015 - overview of Kubernetes

569f10721398d92f5033097ac6d9132c?s=128

Tim Hockin

April 17, 2015
Tweet

Transcript

  1. 1.

    Google confidential │ Do not distribute Google confidential │ Do

    not distribute Kubernetes and Container Management Container Camp 2015 Tim Hockin <thockin@google.com> Senior Staff Software Engineer @thockin
  2. 2.

    Google confidential │ Do not distribute Google has been developing

    and using containers to manage our applications for over 10 years. Images by Connie Zhou
  3. 3.

    Google confidential │ Do not distribute Everything at Google runs

    in containers: • Gmail, Web Search, Maps, ... • MapReduce, batch, ... • GFS, Colossus, ... • Even GCE itself: VMs run in containers
  4. 4.

    Google confidential │ Do not distribute Everything at Google runs

    in containers: • Gmail, Web Search, Maps, ... • MapReduce, batch, ... • GFS, Colossus, ... • Even GCE itself: VMs run in containers We launch over 2 billion containers per week.
  5. 5.

    Google confidential │ Do not distribute But it’s so different!

    • Deployment • Management, monitoring • Isolation (very complicated!) • Updates • Discovery • Scaling, replication, sets A fundamentally different way of managing applications requires different tooling and abstractions Images by Connie Zhou
  6. 6.

    Google confidential │ Do not distribute Enter Kubernetes Greek for

    “Helmsman”; also the root of the word “Governor” and “cybernetic” • Container orchestrator • Runs and manages containers • Supports multiple cloud and bare-metal environments • Inspired and informed by Google’s experiences and internal systems • 100% Open source, written in Go Manage applications, not machines
  7. 9.

    Google confidential │ Do not distribute users master nodes A

    50000 foot view CLI API UI apiserver kubelet kubelet kubelet scheduler
  8. 10.

    Google confidential │ Do not distribute A 50000 foot view

    apiserver kubelet kubelet kubelet scheduler Run X Replicas = 2 Memory = 4Gi CPU = 2.5
  9. 11.

    Google confidential │ Do not distribute A 50000 foot view

    apiserver kubelet kubelet kubelet scheduler SUCCESS UID=8675309
  10. 12.

    Google confidential │ Do not distribute A 50000 foot view

    apiserver kubelet kubelet kubelet scheduler Which nodes for X ?
  11. 13.

    Google confidential │ Do not distribute A 50000 foot view

    apiserver kubelet kubelet kubelet scheduler Run X Run X
  12. 14.

    Google confidential │ Do not distribute A 50000 foot view

    apiserver kubelet kubelet kubelet scheduler Registry pull X pull X
  13. 15.

    Google confidential │ Do not distribute A 50000 foot view

    apiserver kubelet kubelet kubelet scheduler Status X Status X X X
  14. 16.

    Google confidential │ Do not distribute A 50000 foot view

    apiserver kubelet kubelet kubelet scheduler X X GET X
  15. 17.

    Google confidential │ Do not distribute A 50000 foot view

    apiserver kubelet kubelet kubelet scheduler X X Status X
  16. 19.

    Google confidential │ Do not distribute All you really care

    about Run X Master Container Cluster X X Status X
  17. 20.

    Google confidential │ Do not distribute Container clusters: A story

    in two parts 1. Setting up a cluster • Choose a cloud: GCE, AWS, Azure, Rackspace, on-premises, ... • Choose a node OS: CoreOS, Atomic, RHEL, Debian, CentOS, Ubuntu, ... • Provision machines: Boot VMs, install and run kube components, ... • Configure networking: IP ranges for Pods, Services, SDN, ... • Start cluster services: DNS, logging, monitoring, ... • Manage nodes: kernel upgrades, OS updates, hardware failures... Not the easy or fun part, but unavoidable This is where things like Google Container Engine (GKE) really help
  18. 21.

    Google confidential │ Do not distribute 2. Using a cluster

    • Run Pods & Containers • Replication controllers • Services • Volumes • Secrets This is the fun part! A distinct set of problems from cluster setup and management Don’t make developers deal with cluster administration! Accelerate development by focusing on the applications, not the cluster Container clusters: A story in two parts
  19. 23.

    Google confidential │ Do not distribute 10.1.1.0/24 172.16.1.1 172.16.1.2 Docker

    networking 10.1.2.0/24 172.16.1.1 10.1.3.0/24 172.16.1.1
  20. 24.

    Google confidential │ Do not distribute 10.1.1.0/24 172.16.1.1 172.16.1.2 Docker

    networking 10.1.2.0/24 172.16.1.1 10.1.3.0/24 172.16.1.1 NAT NAT NAT NAT NAT
  21. 25.

    Google confidential │ Do not distribute Kubernetes networking Pod IPs

    are routable • docker default is private IP Pods can reach each other without NAT • even across nodes No brokering of port numbers • too complex, why bother? This is a fundamental requirement • several SDN solutions exist
  22. 26.

    Google confidential │ Do not distribute 10.1.1.0/24 10.1.1.93 10.1.1.113 10.1.2.0/24

    10.1.2.118 10.1.3.0/24 10.1.3.129 Kubernetes networking
  23. 27.

    Google confidential │ Do not distribute Concept: Pods Small group

    of containers & volumes Tightly coupled The atom of scheduling & placement in Kubernetes Shared namespace • share IP address & localhost • share IPC Mortal • can die, cannot be reborn Example: data puller & web server Pod File Puller Web Server Volume Consumers Content Manager
  24. 28.

    Google confidential │ Do not distribute Concept: Services A group

    of pods that work together • grouped by a selector Defines access policy • “load balanced” or “headless” for now Gets a stable virtual IP and port • called the service portal • also a DNS name VIP is captured by kube-proxy • watches the service constituency • updates when backends change Hides complexity - ideal for non-native apps Portal (VIP) Client
  25. 30.

    Google confidential │ Do not distribute Services kube-proxy Pod -

    Name = “pod1” - Labels = {“App”: “Nifty”} - Port = 9376 apiserver POST pods WATCH Services, Endpoints
  26. 31.

    Google confidential │ Do not distribute Services kube-proxy apiserver pod1

    10.240.1.1 : 9376 pod2 10.240.2.2 : 9376 pod3 10.240.3.3 : 9376 run pods Pod - Name = “pod1” - Labels = {“App”: “Nifty”} - Port = 9376 WATCH Services, Endpoints
  27. 32.

    Google confidential │ Do not distribute POST service pod1 10.240.1.1

    : 9376 pod2 10.240.2.2 : 9376 pod3 10.240.3.3 : 9376 Services kube-proxy Service - Name = “nifty-svc” - Selector = {“App”: “Nifty”} - Port = 80 - TargetPort = 9376 - PortalIP - 10.9.8.7 apiserver WATCH Services, Endpoints
  28. 33.

    Google confidential │ Do not distribute pod1 10.240.1.1 : 9376

    pod2 10.240.2.2 : 9376 pod3 10.240.3.3 : 9376 Services kube-proxy apiserver Service - Name = “nifty-svc” - Selector = {“App”: “Nifty”} - Port = 80 - TargetPort = 9376 - PortalIP - 10.9.8.7 WATCH Services, Endpoints new service!
  29. 34.

    Google confidential │ Do not distribute pod1 10.240.1.1 : 9376

    pod2 10.240.2.2 : 9376 pod3 10.240.3.3 : 9376 Services kube-proxy apiserver Linux listen on port X (random) Service - Name = “nifty-svc” - Selector = {“App”: “Nifty”} - Port = 80 - TargetPort = 9376 - PortalIP - 10.9.8.7 WATCH Services, Endpoints
  30. 35.

    Google confidential │ Do not distribute pod1 10.240.1.1 : 9376

    pod2 10.240.2.2 : 9376 pod3 10.240.3.3 : 9376 Services kube-proxy apiserver Linux listen on port X iptables redirect 10.9.8.7:80 to localhost:X Service - Name = “nifty-svc” - Selector = {“App”: “Nifty”} - Port = 80 - TargetPort = 9376 - PortalIP - 10.9.8.7 WATCH Services, Endpoints
  31. 36.

    Google confidential │ Do not distribute pod1 10.240.1.1 : 9376

    pod2 10.240.2.2 : 9376 pod3 10.240.3.3 : 9376 Services kube-proxy apiserver Linux listen on port X iptables redirect 10.9.8.7:80 to localhost:X Service - Name = “nifty-svc” - Selector = {“App”: “Nifty”} - Port = 80 - TargetPort = 9376 - PortalIP - 10.9.8.7 WATCH Services, Endpoints new endpoints!
  32. 37.

    Google confidential │ Do not distribute pod1 10.240.1.1 : 9376

    pod2 10.240.2.2 : 9376 pod3 10.240.3.3 : 9376 Services kube-proxy apiserver Linux listen on port X iptables redirect 10.9.8.7:80 to localhost:X Service - Name = “nifty-svc” - Selector = {“App”: “Nifty”} - Port = 80 - TargetPort = 9376 - PortalIP - 10.9.8.7
  33. 38.

    Google confidential │ Do not distribute pod1 10.240.1.1 : 9376

    pod2 10.240.2.2 : 9376 pod3 10.240.3.3 : 9376 Services kube-proxy apiserver Linux listen on port X iptables Client redirect 10.9.8.7:80 to localhost:X Service - Name = “nifty-svc” - Selector = {“App”: “Nifty”} - Port = 80 - TargetPort = 9376 - PortalIP - 10.9.8.7 connect to 10.9.8.7:80
  34. 39.

    Google confidential │ Do not distribute pod1 10.240.1.1 : 9376

    pod2 10.240.2.2 : 9376 pod3 10.240.3.3 : 9376 Services kube-proxy apiserver Linux listen on port X iptables Client redirect 10.9.8.7:80 to localhost:X Service - Name = “nifty-svc” - Selector = {“App”: “Nifty”} - Port = 80 - TargetPort = 9376 - PortalIP - 10.9.8.7 connect to 10.9.8.7:80
  35. 40.

    Google confidential │ Do not distribute pod1 10.240.1.1 : 9376

    pod2 10.240.2.2 : 9376 pod3 10.240.3.3 : 9376 Services kube-proxy apiserver Linux iptables Client Service - Name = “nifty-svc” - Selector = {“App”: “Nifty”} - Port = 80 - TargetPort = 9376 - PortalIP - 10.9.8.7 connect to localhost:X
  36. 41.

    Google confidential │ Do not distribute pod1 10.240.1.1 : 9376

    pod2 10.240.2.2 : 9376 pod3 10.240.3.3 : 9376 Services kube-proxy apiserver Linux listen on port X iptables Client Service - Name = “nifty-svc” - Selector = {“App”: “Nifty”} - Port = 80 - TargetPort = 9376 - PortalIP - 10.9.8.7 proxy for client
  37. 42.

    Google confidential │ Do not distribute Concept: Volumes Very similar

    to Docker’s concept Pod scoped storage Share the pod’s lifetime & fate Support many types of volume plugins • Empty directory • Host path • Git repository • GCE Persistent Disk • AWS Elastic Block Store • iSCSI Pod Container Container Git GitHub Host Host’s FS GCE GCE PD Empty • NFS • GlusterFS • Ceph File and RBD • Cinder • ...
  38. 43.

    Google confidential │ Do not distribute User owned Admin owned

    New: Persistent Volumes A higher-level abstraction - insulation from any one cloud environment Admin provisions them, users claim them Independent lifetime and fate Can be handed-off between pods and lives until user is done with it Dynamically “scheduled” and managed, like nodes and pods Pod ClaimRef PVClaim PersistentVolume GCE PD AWS ELB NFS iSCSI
  39. 44.

    Google confidential │ Do not distribute Docker, Rocket, LXC, Oh

    my! Currently built on Docker Work is in progress to abstract that (a bit) into a Runtime abstraction Interest in Rocket and LXC support Rocket support is in flight (we like plugins) Dynamically “scheduled” and managed, like nodes and pods Runtime Docker Rocket LXC? Kubelet
  40. 45.

    Google confidential │ Do not distribute What else is in

    new? • Network plugins • Secrets • Graceful termination • Quota • More volumes • Downward API • More platforms • Performance • Scalability • High availability masters • Scheduling • Cluster federation • Multi-cloud • Easier setup
  41. 46.

    Google confidential │ Do not distribute Kubernetes status & plans

    Open sourced in June, 2014 • won the 2014 BlackDuck “rookie of the year” award Google Container Engine (GKE) • hosted Kubernetes - don’t think about cluster setup Red Hat: OpenShift 3 • open PaaS on Kubernetes CoreOS: Tectonic • ready-to-run Kubernetes - don’t think about cluster setup Mirantis: Murano • Kubernetes and OpenStack Driving towards a 1.0 release in O(months) Roadmap: • https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/roadmap.md
  42. 47.

    Google confidential │ Do not distribute The Goal: Shake things

    up Containers are a new way of working Requires new concepts and new tools Google has a lot of experience... ...but we are listening to the users Workload portability is important!
  43. 48.

    Google confidential │ Do not distribute Kubernetes is Open -

    open community - open design - open to ideas - open source http://kubernetes.io https://github.com/GoogleCloudPlatform/kubernetes irc.freenode.net #google-containers @kubernetesio