Kubernetes: The Latest and Greatest

Kubernetes: The Latest and Greatest

Kubernetes is a open-source system, developed by Google, for automating the deployment, management, and scaling of containerized applications across a distributed cluster. Many companies are standardizing on Kubernetes as a technology for container orchestration including Red Hat, eBay, and of course Google. Kubernetes aims to provide better ways of managing related, distributed components across varied infrastructure. In this session, we'll take a detailed look at Kubernetes, the open source orchestration system for containers, and explore the key terms, concepts, and vocabulary introduced by the project. We'll also explore key features and improvement in Kubernetes 1.2 & 1.3 and how to use them in practice.

01dc8e954957a10b428aa60b28c89d52?s=128

Ian Lewis

July 15, 2016
Tweet

Transcript

  1. Ian Lewis Developer Advocate, Google Cloud Platform Kubernetes: The Latest

    and Greatest #containercon Japan
  2. Confidential & Proprietary Google Cloud Platform 2 Ian Lewis Developer

    Advocate - Google Cloud Platform Tokyo, Japan +Ian Lewis @IanMLewis
  3. Agenda • Kubernetes Architecture • Concepts, Lots of Concepts •

    What’s new in 1.3
  4. Confidential & Proprietary Google Cloud Platform 4 Why did we

    build this thing?
  5. For the last 15 years Google has been building the

    world’s fastest, most powerful infrastructure.
  6. None
  7. Cloud Technology Innovations 2012 2013 MapReduce Spanner/F1 2003 2006 2007

    2010 2011 GFS Omega Colossus Cloud Storage Dremel BigQuery Big Table Cloud Datastore Paxos impl. 2004 Cloud Bigtable
  8. Copyright 2015 Google Inc Google has been running all our

    services in Containers for over 10 years. We start over 2 billion containers every week. Images by Connie Zhou
  9. http://research.google.com/pubs/pub43438.html

  10. Image by Connie Zhou

  11. job hello_world = { runtime = { cell = 'ic'

    } // Cell (cluster) to run in binary = '.../hello_world_webserver' // Program to run args = { port = '%port%' } // Command line parameters requirements = { // Resource requirements ram = 100M disk = 100M cpu = 0.1 } replicas = 5 // Number of tasks } 10000 Developer View
  12. web browsers BorgMaster link shard UI shard BorgMaster link shard

    UI shard BorgMaster link shard UI shard BorgMaster link shard UI shard Scheduler borgcfg web browsers scheduler Borglet Borglet Borglet Borglet Config file BorgMaster link shard UI shard persistent store (Paxos) Binary Developer View What just happened?
  13. Developer View

  14. Hello world! Hello world! Hello world! Hello world! Hello world!

    Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Image by Connie Zhou Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world! Hello world!
  15. Confidential & Proprietary Google Cloud Platform 15 Kubernetes

  16. Enter Kubernetes Greek for “Helmsman”; also the root of the

    word “Governor” • Container orchestrator • Runs containers • Supports multiple cloud and bare- metal environments • Inspired and informed by Google’s experiences and internal systems • Open source, written in Go Manage applications, not machines
  17. Google Cloud Platform UI CLI API users master nodes etcd

    scheduler controllers master The 10000 Foot View kubelet kubelet kubelet kubelet kubelet kubelet kubelet kubelet kubelet
  18. web browsers BorgMaster link shard UI shard BorgMaster link shard

    UI shard BorgMaster link shard UI shard BorgMaster link shard UI shard Scheduler borgcfg web browsers scheduler Borglet Borglet Borglet Borglet Config file BorgMaster link shard UI shard persistent store (Paxos) Manifest Master Kubelet Kubelet Kubelet Binary Developer View Docker Image Docker Hub/Private Repo What just happened? Kubelet
  19. Confidential & Proprietary Google Cloud Platform 19 Pods

  20. Google Cloud Platform Pods (TODO) docker … --net=container: id --ipc=container:id

    -- pid=container:id https://github. com/docker/docker/issue s/10163 IPC Network PID Hostname cgroup Web Server cgroup File Puller localhost
  21. Google Cloud Platform Arbitrary metadata Attached to any API object

    Generally represent identity Queryable by selectors • think SQL ‘select ... where ...’ The only grouping mechanism • pods under a ReplicationController • pods in a Service • capabilities of a node (constraints) Labels
  22. Google Cloud Platform App: MyApp Phase: prod Role: FE App:

    MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE Selectors
  23. Google Cloud Platform App: MyApp Phase: prod Role: FE App:

    MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE App = MyApp Selectors
  24. Google Cloud Platform App: MyApp Phase: prod Role: FE App:

    MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE App = MyApp, Role = FE Selectors
  25. Google Cloud Platform App: MyApp Phase: prod Role: FE App:

    MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE App = MyApp, Role = BE Selectors
  26. Google Cloud Platform Selectors App: MyApp Phase: prod Role: FE

    App: MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE App = MyApp, Phase = prod
  27. Google Cloud Platform App: MyApp Phase: prod Role: FE App:

    MyApp Phase: test Role: FE App: MyApp Phase: prod Role: BE App: MyApp Phase: test Role: BE App = MyApp, Phase = test Selectors
  28. Confidential & Proprietary Google Cloud Platform 28 Configuration

  29. Google Cloud Platform ConfigMaps Goal: manage app configuration • ...without

    making overly-brittle container images 12-factor says config comes from the environment • Kubernetes is the environment Manage config via the Kubernetes API Inject config as a virtual volume into your Pods • late-binding, live-updated (atomic) • also available as env vars Status: GA in Kubernetes v1.2 node API Pod Config Map
  30. Google Cloud Platform Secrets Goal: grant a pod access to

    a secured something • don’t put secrets in the container image! 12-factor says config comes from the environment • Kubernetes is the environment Manage secrets via the Kubernetes API Inject secrets as virtual volumes into your Pods • late-binding, tmpfs - never touches disk • also available as env vars node API Pod Secret
  31. Confidential & Proprietary Google Cloud Platform 31 Volumes

  32. Google Cloud Platform PersistentVolumes A higher-level storage abstraction • insulation

    from any one cloud environment Admin provisions them, users claim them • NEW: auto-provisioning (alpha in v1.2) Independent lifetime from consumers • lives until user is done with it • can be handed-off between pods Dynamically “scheduled” and managed, like nodes and pods Claim
  33. Google Cloud Platform PersistentVolumes Cluster Admin

  34. Google Cloud Platform PersistentVolumes Provision Cluster Admin PersistentVolumes

  35. Google Cloud Platform PersistentVolumes User Cluster Admin PersistentVolumes

  36. Google Cloud Platform PersistentVolumes User PVClaim Create Cluster Admin PersistentVolumes

  37. Google Cloud Platform PersistentVolumes User PVClaim Binder Cluster Admin PersistentVolumes

  38. Google Cloud Platform PersistentVolumes User PVClaim Pod Create Cluster Admin

    PersistentVolumes
  39. Google Cloud Platform PersistentVolumes User PVClaim Pod Cluster Admin PersistentVolumes

    *
  40. Google Cloud Platform PersistentVolumes User PVClaim Pod Delete * Cluster

    Admin PersistentVolumes *
  41. Google Cloud Platform PersistentVolumes User PVClaim Cluster Admin PersistentVolumes *

  42. Google Cloud Platform PersistentVolumes User PVClaim Pod Create Cluster Admin

    PersistentVolumes *
  43. Google Cloud Platform PersistentVolumes User PVClaim Pod Cluster Admin PersistentVolumes

    *
  44. Google Cloud Platform PersistentVolumes User PVClaim Pod Delete Cluster Admin

    PersistentVolumes *
  45. Google Cloud Platform PersistentVolumes User PVClaim Delete Cluster Admin PersistentVolumes

    *
  46. Google Cloud Platform PersistentVolumes User Recycler Cluster Admin PersistentVolumes

  47. Confidential & Proprietary Google Cloud Platform 47 Controllers: Deployments &

    ReplicaSets
  48. Google Cloud Platform The 10000 Foot View etcd scheduler controller-manager

    apiserver kubelet docker kube-proxy iptables 1. User creates Deployment 2. API server saves info to etcd 3. CM finds Deployment and creates ReplicaSet, which creates Pods (unscheduled) 4. Scheduler schedules pods. 5. Kubelet sees pod scheduled to it and tells docker to run the container. 6. Docker pulls and runs the container. 1 2 6 3 5 4 Docker Hub / GCR
  49. Google Cloud Platform observe diff act Deployments & ReplicaSets

  50. Google Cloud Platform Deployments ReplicaSet - replicas: 3 - selector:

    - app: MyApp - version: v1 Deployment - name: MyApp kubectl create ...
  51. Google Cloud Platform Deployments ReplicaSet - replicas: 4 - selector:

    - app: MyApp - version: v1 Deployment - name: MyApp kubectl create ...
  52. Google Cloud Platform Deployments ReplicaSet - replicas: 3 - selector:

    - app: MyApp - version: v1 Deployment - name: MyApp kubectl create ...
  53. Google Cloud Platform Deployments ReplicaSet - replicas: 3 - selector:

    - app: MyApp - version: v1 Deployment - name: MyApp kubectl create ...
  54. Google Cloud Platform Rolling Updates ReplicaSet - replicas: 3 -

    selector: - app: MyApp - version: v1 Deployment - name: MyApp kubectl apply ...
  55. Google Cloud Platform ReplicaSet - replicas: 3 - selector: -

    app: MyApp - version: v1 Rolling Updates ReplicaSet - replicas: 0 - selector: - app: MyApp - version: v2 Deployment - name: MyApp
  56. Google Cloud Platform ReplicaSet - replicas: 3 - selector: -

    app: MyApp - version: v1 ReplicaSet - replicas: 1 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp
  57. Google Cloud Platform ReplicaSet - replicas: 2 - selector: -

    app: MyApp - version: v1 ReplicaSet - replicas: 1 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp
  58. Google Cloud Platform ReplicaSet - replicas: 2 - selector: -

    app: MyApp - version: v1 ReplicaSet - replicas: 2 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp
  59. Google Cloud Platform ReplicaSet - replicas: 1 - selector: -

    app: MyApp - version: v1 ReplicaSet - replicas: 2 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp
  60. Google Cloud Platform ReplicaSet - replicas: 1 - selector: -

    app: MyApp - version: v1 ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp
  61. Google Cloud Platform ReplicaSet - replicas: 0 - selector: -

    app: MyApp - version: v1 ReplicaSet - replicas: 3 - selector: - app: MyApp - version: v2 Rolling Updates Deployment - name: MyApp
  62. Confidential & Proprietary Google Cloud Platform 62 Services

  63. Google confidential │ Do not distribute Services A group of

    pods that work together • grouped by a selector Defines access policy • “load balanced” or “headless” Gets a stable virtual IP and port • sometimes called the service portal • also a DNS name VIP is managed by kube-proxy • watches all services • updates iptables when backends change Hides complexity - ideal for non-native apps Virtual IP Client
  64. Google Cloud Platform External Services Services VIPs are only available

    inside the cluster Need to receive traffic from “the outside world” Service “type” • NodePort: expose on a port on every node • LoadBalancer: provision a cloud load-balancer DiY load-balancer solutions • socat (for nodePort remapping) • haproxy • nginx Ingress (L7 LB)
  65. Google Cloud Platform Ingress (L7) Many apps are HTTP/HTTPS Services

    are L4 (IP + port) Ingress maps incoming traffic to backend services • by HTTP host headers • by HTTP URL paths HAProxy, NGINX, AWS and GCE implementations in progress Now with SSL! Status: BETA in Kubernetes v1.2 Client URL Map
  66. Confidential & Proprietary Google Cloud Platform 66 Health Checks

  67. Google confidential │ Do not distribute Health Checks Makes sure

    containers are healthy. Includes liveness and readiness probes. Liveness Probes • Checks if a container is alive. Hasn’t crashed or deadlocked. Readiness Probes • Checks to make sure a container is ready to serve traffic.
  68. Google confidential │ Do not distribute Liveness Checks if a

    container is alive. Hasn’t crashed or deadlocked. If a container fails a liveness check too many times it will be restarted.
  69. Google confidential │ Do not distribute Readiness Checks if a

    container is ready to serve traffic. Typically, an app will check to make sure it’s dependencies are available. If a container fails a readiness check it many times it’s pod is removed from service endpoints. This makes sure services only use pods that can serve traffic. Virtual IP Client
  70. Google Cloud Platform Graceful Termination Goal: Give pods time to

    clean up • finish in-flight operations • log state • flush to disk • 30 seconds by default Catch SIGTERM, cleanup, exit ASAP Pod status “Terminating” Declarative: ‘DELETE’ appears as an object field in the API
  71. Confidential & Proprietary Google Cloud Platform 71 DaemonSets

  72. Google Cloud Platform DaemonSets Problem: how to run a Pod

    on every node? • or a subset of nodes Similar to ReplicaSet • principle: do one thing, don’t overload “Which nodes?” is a selector Use familiar tools and patterns Status: BETA in Kubernetes v1.2 Pod
  73. Confidential & Proprietary Google Cloud Platform 73 Jobs

  74. Google Cloud Platform Jobs Run-to-completion, as opposed to run-forever •

    Express parallelism vs. required completions • Workflow: restart on failure • Build/test: don’t restart on failure Aggregates success/failure counts Built for batch and big-data work Status: GA in Kubernetes v1.2 ...
  75. Confidential & Proprietary Google Cloud Platform 75 Namespaces

  76. Google Cloud Platform Namespaces Problem: I have too much stuff!

    • name collisions in the API • poor isolation between users • don’t want to expose things like Secrets Solution: Slice up the cluster • create new Namespaces as needed • per-user, per-app, per-department, etc. • part of the API - NOT private machines • most API objects are namespaced • part of the REST URL path • Namespaces are just another API object • One-step cleanup - delete the Namespace • Obvious hook for policy enforcement (e.g. quota)
  77. Google Cloud Platform Resource Isolation Principles: • Apps must not

    be able to affect each other’s performance • if so it is an isolation failure • Repeated runs of the same app should see ~equal behavior • QoS levels drives resource decisions in (soft) real-time • Correct in all cases, optimal in some • reduce unreliable components • SLOs are the lingua franca
  78. Google Cloud Platform Pros: • Sharing - users don’t worry

    about interference (aka the noisy neighbor problem) • Predictable - allows us to offer strong SLAs to apps Cons: • Stranding - arbitrary slices mean some resources get lost • Confusing - how do I know how much I need? • analog: what size VM should I use? • smart auto-scaling is needed! • Expensive - you pay for certainty In reality this is a multi-dimensional bin-packing problem: CPU, memory, disk space, IO bandwidth, network bandwidth, ... Strong isolation
  79. Google Cloud Platform Requests and Limits Request: • how much

    of a resource you are asking to use, with a strong guarantee of availability • CPU (seconds/second) • RAM (bytes) • scheduler will not over-commit requests Limit: • max amount of a resource you can access Repercussions: • Usage > Request: resources might be available • Usage > Limit: throttled or killed
  80. Google Cloud Platform Quality of Service Defined in terms of

    Request and Limit Guaranteed: highest protection • request > 0 && limit == request Burstable: medium protection • request > 0 && limit > request Best Effort: lowest protection • request == 0 What does “protection” mean? • OOM score • CPU scheduling
  81. Google Cloud Platform ResourceQuota Admission control: apply limits in aggregate

    Per-namespace: ensure no user/app/department abuses the cluster Reminiscent of disk quota by design Applies to each type of resource • CPU and memory for now Disallows pods without resources
  82. Google Cloud Platform LimitRange Admission control: limit the limits •

    min and max • ratio of limit/request Default values for unspecified limits Per-namespace Together with ResourceQuota gives cluster admins powerful tools
  83. Confidential & Proprietary Google Cloud Platform 83 Kubernetes 1.3

  84. Kubernetes 1.3 • Supports up to 60,000 containers on 2000

    nodes • PetSet (alpha) • Federation • Cross cluster service discovery • New in 1.2: • Deployments • ConfigMaps
  85. Google Cloud Platform PetSets (working name) Goal: enable clustered software

    on Kubernetes • mysql, redis, zookeeper, ... Clustered apps need “identity” and sequencing guarantees • stable hostname, available in DNS • an ordinal index • stable storage: linked to the ordinal & hostname • discovery of peers for quorum • startup/teardown ordering Status: ALPHA in Kubernetes v1.3
  86. Google Container Engine - Inspired by a decade within Google

    - Reimagines Cluster computing - Designed for a multi-cloud world Photo by Connie Zhou
  87. Google Container Engine New service for cluster-based compute • Provisioned

    cluster in seconds. Fully configured. • Fine-grained control over cluster. • Designed for multi-cloud. Runs Kubernetes. Releases • Now GA!! • No additional cost for up to 5 nodes Resources • Google Container Engine: http://cloud.google.com/container-engine • Kubernetes: http://kubernetes.io
  88. Thank You