Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kubernetes: Container Orchestration at Scale

Kubernetes: Container Orchestration at Scale

Talk at Container Days Boston 2015.

Maxwell Forbes

June 05, 2015
Tweet

Other Decks in Technology

Transcript

  1. Google confidential │ Do not distribute
    Google confidential │ Do not distribute
    Max Forbes
    Container Days Boston 2015
    Thanks to Brendan Burns and Tim Hockin for nearly all of the slides.
    Kubernetes
    Container Orchestration at Scale

    View Slide

  2. Google confidential │ Do not distribute
    Everything at Google runs in
    containers:
    • Gmail, Web Search, Maps, ...
    • MapReduce, batch, ...
    • GFS, Colossus, ...
    • Even GCE itself: VMs in
    containers

    View Slide

  3. Google confidential │ Do not distribute
    Everything at Google runs in
    containers:
    • Gmail, Web Search, Maps, ...
    • MapReduce, batch, ...
    • GFS, Colossus, ...
    • Even GCE itself: VMs in
    containers
    We launch over 2 billion
    containers per week.

    View Slide

  4. Google confidential │ Do not distribute
    More than just “running” containers

    View Slide

  5. Google confidential │ Do not distribute
    More than just “running” containers
    Scheduling: Where should my job be run?

    View Slide

  6. Google confidential │ Do not distribute
    More than just “running” containers
    Scheduling: Where should my job be run?
    Lifecycle: Keep my job running

    View Slide

  7. Google confidential │ Do not distribute
    More than just “running” containers
    Scheduling: Where should my job be run?
    Lifecycle: Keep my job running
    Discovery: Where is my job now?

    View Slide

  8. Google confidential │ Do not distribute
    More than just “running” containers
    Scheduling: Where should my job be run?
    Lifecycle: Keep my job running
    Discovery: Where is my job now?
    Constituency: Who is part of my job?

    View Slide

  9. Google confidential │ Do not distribute
    More than just “running” containers
    Scheduling: Where should my job be run?
    Lifecycle: Keep my job running
    Discovery: Where is my job now?
    Constituency: Who is part of my job?
    Scale-up: Making my jobs bigger or smaller

    View Slide

  10. Google confidential │ Do not distribute
    More than just “running” containers
    Scheduling: Where should my job be run?
    Lifecycle: Keep my job running
    Discovery: Where is my job now?
    Constituency: Who is part of my job?
    Scale-up: Making my jobs bigger or smaller
    Auth{n,z}: Who can do things to my job?

    View Slide

  11. Google confidential │ Do not distribute
    More than just “running” containers
    Scheduling: Where should my job be run?
    Lifecycle: Keep my job running
    Discovery: Where is my job now?
    Constituency: Who is part of my job?
    Scale-up: Making my jobs bigger or smaller
    Auth{n,z}: Who can do things to my job?
    Monitoring: What’s happening with my job?

    View Slide

  12. Google confidential │ Do not distribute
    More than just “running” containers
    Scheduling: Where should my job be run?
    Lifecycle: Keep my job running
    Discovery: Where is my job now?
    Constituency: Who is part of my job?
    Scale-up: Making my jobs bigger or smaller
    Auth{n,z}: Who can do things to my job?
    Monitoring: What’s happening with my job?
    Health: How is my job feeling?

    View Slide

  13. Google confidential │ Do not distribute
    More than just “running” containers
    Scheduling: Where should my job be run?
    Lifecycle: Keep my job running
    Discovery: Where is my job now?
    Constituency: Who is part of my job?
    Scale-up: Making my jobs bigger or smaller
    Auth{n,z}: Who can do things to my job?
    Monitoring: What’s happening with my job?
    Health: How is my job feeling?
    ...

    View Slide

  14. View Slide

  15. Google confidential │ Do not distribute
    Kubernetes
    Greek for “Helmsman”; also the root of
    the word “Governor”
    • Container orchestration
    • Runs Docker containers
    • Supports multiple cloud and bare-metal
    environments
    • Inspired and informed by Google’s
    experiences and internal systems
    • Open source, written in Go
    Manage applications, not machines

    View Slide

  16. Google confidential │ Do not distribute
    A 50000 foot view

    View Slide

  17. Google confidential │ Do not distribute
    users master nodes
    A 50000 foot view
    CLI
    API
    UI
    apiserver
    kubelet
    kubelet
    kubelet
    scheduler

    View Slide

  18. Google confidential │ Do not distribute
    A 50000 foot view
    apiserver
    kubelet
    kubelet
    kubelet
    scheduler
    Run X
    Replicas = 2
    Memory = 4Gi
    CPU = 2.5

    View Slide

  19. Google confidential │ Do not distribute
    A 50000 foot view
    apiserver
    kubelet
    kubelet
    kubelet
    scheduler
    SUCCESS
    UID=8675309

    View Slide

  20. Google confidential │ Do not distribute
    A 50000 foot view
    apiserver
    kubelet
    kubelet
    kubelet
    scheduler
    Which nodes
    for X ?

    View Slide

  21. Google confidential │ Do not distribute
    A 50000 foot view
    apiserver
    kubelet
    kubelet
    kubelet
    scheduler
    Run X
    Run X

    View Slide

  22. Google confidential │ Do not distribute
    A 50000 foot view
    apiserver
    kubelet
    kubelet
    kubelet
    scheduler
    Registry
    pull X
    pull X

    View Slide

  23. Google confidential │ Do not distribute
    A 50000 foot view
    apiserver
    kubelet
    kubelet
    kubelet
    scheduler
    Status
    X
    Status
    X
    X
    X

    View Slide

  24. Google confidential │ Do not distribute
    A 50000 foot view
    apiserver
    kubelet
    kubelet
    kubelet
    scheduler
    X
    X
    GET X

    View Slide

  25. Google confidential │ Do not distribute
    A 50000 foot view
    apiserver
    kubelet
    kubelet
    kubelet
    scheduler
    X
    X
    Status X

    View Slide

  26. Google confidential │ Do not distribute
    All you really care about
    Run X Master
    Container
    Cluster
    X
    X
    Status X

    View Slide

  27. Google confidential │ Do not distribute
    Design principles

    View Slide

  28. Google confidential │ Do not distribute
    Design principles
    Declarative > imperative: State your desired results, let the system actuate

    View Slide

  29. Google confidential │ Do not distribute
    Design principles
    Declarative > imperative: State your desired results, let the system actuate
    Control loops: Observe, rectify, repeat

    View Slide

  30. Google confidential │ Do not distribute
    Design principles
    Declarative > imperative: State your desired results, let the system actuate
    Control loops: Observe, rectify, repeat
    Simple > Complex: Try to do as little as possible

    View Slide

  31. Google confidential │ Do not distribute
    Design principles
    Declarative > imperative: State your desired results, let the system actuate
    Control loops: Observe, rectify, repeat
    Simple > Complex: Try to do as little as possible
    Modularity: Components, interfaces, & plugins

    View Slide

  32. Google confidential │ Do not distribute
    Design principles
    Declarative > imperative: State your desired results, let the system actuate
    Control loops: Observe, rectify, repeat
    Simple > Complex: Try to do as little as possible
    Modularity: Components, interfaces, & plugins
    Legacy compatible: Requiring apps to change is a non-starter

    View Slide

  33. Google confidential │ Do not distribute
    Design principles
    Declarative > imperative: State your desired results, let the system actuate
    Control loops: Observe, rectify, repeat
    Simple > Complex: Try to do as little as possible
    Modularity: Components, interfaces, & plugins
    Legacy compatible: Requiring apps to change is a non-starter
    No grouping: Labels are the only groups

    View Slide

  34. Google confidential │ Do not distribute
    Design principles
    Declarative > imperative: State your desired results, let the system actuate
    Control loops: Observe, rectify, repeat
    Simple > Complex: Try to do as little as possible
    Modularity: Components, interfaces, & plugins
    Legacy compatible: Requiring apps to change is a non-starter
    No grouping: Labels are the only groups
    Cattle > Pets: Manage your workload in bulk

    View Slide

  35. Google confidential │ Do not distribute
    Pets vs. Cattle

    View Slide

  36. Google confidential │ Do not distribute
    Design principles
    Declarative > imperative: State your desired results, let the system actuate
    Control loops: Observe, rectify, repeat
    Simple > Complex: Try to do as little as possible
    Modularity: Components, interfaces, & plugins
    Legacy compatible: Requiring apps to change is a non-starter
    No grouping: Labels are the only groups
    Cattle > Pets: Manage your workload in bulk

    View Slide

  37. Google confidential │ Do not distribute
    Design principles
    Declarative > imperative: State your desired results, let the system actuate
    Control loops: Observe, rectify, repeat
    Simple > Complex: Try to do as little as possible
    Modularity: Components, interfaces, & plugins
    Legacy compatible: Requiring apps to change is a non-starter
    No grouping: Labels are the only groups
    Cattle > Pets: Manage your workload in bulk
    Open > Closed: Open Source, standards, REST, JSON, etc.

    View Slide

  38. Google confidential │ Do not distribute
    Primary concepts

    View Slide

  39. Google confidential │ Do not distribute
    Primary concepts
    0. Container: A sealed application package (Docker)

    View Slide

  40. Google confidential │ Do not distribute
    Primary concepts
    0. Container: A sealed application package (Docker)
    1. Pod: A small group of tightly coupled Containers
    example: content syncer & web server

    View Slide

  41. Google confidential │ Do not distribute
    Primary concepts
    0. Container: A sealed application package (Docker)
    1. Pod: A small group of tightly coupled Containers
    example: content syncer & web server
    2. Controller: A loop that drives current state towards desired state
    example: replication controller

    View Slide

  42. Google confidential │ Do not distribute
    Primary concepts
    0. Container: A sealed application package (Docker)
    1. Pod: A small group of tightly coupled Containers
    example: content syncer & web server
    2. Controller: A loop that drives current state towards desired state
    example: replication controller

    View Slide

  43. Google confidential │ Do not distribute
    Primary concepts
    0. Container: A sealed application package (Docker)
    1. Pod: A small group of tightly coupled Containers
    example: content syncer & web server
    2. Controller: A loop that drives current state towards desired state
    example: replication controller
    3. Service: A set of running pods that work together
    example: load-balanced backends

    View Slide

  44. Google confidential │ Do not distribute
    Primary concepts
    0. Container: A sealed application package (Docker)
    1. Pod: A small group of tightly coupled Containers
    example: content syncer & web server
    2. Controller: A loop that drives current state towards desired state
    example: replication controller
    3. Service: A set of running pods that work together
    example: load-balanced backends
    4. Labels: Identifying metadata attached to other objects
    example: phase=canary vs. phase=prod
    5. Selector: A query against labels, producing a set result
    example: all pods where label phase == prod

    View Slide

  45. Google confidential │ Do not distribute
    Pods

    View Slide

  46. Google confidential │ Do not distribute
    Pods
    Small group of containers & volumes

    View Slide

  47. Google confidential │ Do not distribute
    Pods
    Small group of containers & volumes
    Tightly coupled

    View Slide

  48. Google confidential │ Do not distribute
    Pods
    Small group of containers & volumes
    Tightly coupled
    The atom of cluster scheduling &
    placement

    View Slide

  49. Google confidential │ Do not distribute
    Pods
    Small group of containers & volumes
    Tightly coupled
    The atom of cluster scheduling &
    placement
    Shared namespace
    • share IP address & localhost

    View Slide

  50. Google confidential │ Do not distribute
    Pods
    Small group of containers & volumes
    Tightly coupled
    The atom of cluster scheduling &
    placement
    Shared namespace
    • share IP address & localhost
    Ephemeral
    • can die and be replaced

    View Slide

  51. Google confidential │ Do not distribute
    Pets vs. Cattle

    View Slide

  52. Google confidential │ Do not distribute
    Pods
    Small group of containers & volumes
    Tightly coupled
    The atom of cluster scheduling &
    placement
    Shared namespace
    • share IP address & localhost
    Ephemeral
    • can die and be replaced

    View Slide

  53. Google confidential │ Do not distribute
    Pods
    Small group of containers & volumes
    Tightly coupled
    The atom of cluster scheduling &
    placement
    Shared namespace
    • share IP address & localhost
    Ephemeral
    • can die and be replaced
    Example: data puller & web server
    Pod
    File Puller Web Server
    Volume
    Consumers
    Content
    Manager

    View Slide

  54. Google confidential │ Do not distribute
    Why pods?

    View Slide

  55. Google confidential │ Do not distribute
    Why pods?
    Pod
    Web Server
    Volume
    Consumers
    Content
    Manager
    File Puller

    View Slide

  56. Google confidential │ Do not distribute
    Why pods?
    Pod
    File Puller Web Server
    Volume
    Consumers
    Content
    Manager
    • infeasible for provider to build
    and maintain all variants of this
    “as a service”

    View Slide

  57. Google confidential │ Do not distribute
    Why pods?
    Pod
    Scary C
    program
    data
    collector
    :-(

    View Slide

  58. Google confidential │ Do not distribute
    Why pods?
    Pod
    Scary C
    program
    adapter
    data
    collector

    View Slide

  59. Google confidential │ Do not distribute
    Why pods?
    Pod
    component
    A
    component
    B

    View Slide

  60. Google confidential │ Do not distribute
    Why pods?
    Pod
    component
    A
    component
    B

    View Slide

  61. Google confidential │ Do not distribute
    Why pods?
    Pod
    app
    DB
    client
    DB

    View Slide

  62. Google confidential │ Do not distribute
    Why IP-per-pod?
    No port mangling.

    View Slide

  63. Google confidential │ Do not distribute
    Why not put everything in one container?
    - transparency
    - decouple software dependencies
    - ease of use
    - efficiency

    View Slide

  64. Google confidential │ Do not distribute
    Why not something besides pods?
    like co-scheduling?
    - simpler to have scheduling atom
    - other benefits of pods
    - resource sharing
    - IPC
    - shared fate
    - simplified management

    View Slide

  65. Google confidential │ Do not distribute
    Pod lifecycle

    View Slide

  66. Google confidential │ Do not distribute
    Pod lifecycle
    Once scheduled to a node, pods do not move
    • restart policy means restart in-place

    View Slide

  67. Google confidential │ Do not distribute
    Pod lifecycle
    Once scheduled to a node, pods do not move
    • restart policy means restart in-place
    Pods can be observed pending, running, succeeded, or failed
    • failed is really the end - no more restarts
    • no complex state machine logic

    View Slide

  68. Google confidential │ Do not distribute
    Pod lifecycle
    Once scheduled to a node, pods do not move
    • restart policy means restart in-place
    Pods can be observed pending, running, succeeded, or failed
    • failed is really the end - no more restarts
    • no complex state machine logic
    Pods are not rescheduled by the scheduler or apiserver
    • even if a node dies
    • controllers are responsible for this
    • keeps the scheduler simple
    Apps should consider these rules
    • Services hide this
    • Makes pod-to-pod communication more formal

    View Slide

  69. Google confidential │ Do not distribute
    Labels

    View Slide

  70. Google confidential │ Do not distribute
    Labels
    Arbitrary metadata
    Attached to any API object
    Generally represent identity

    View Slide

  71. Google confidential │ Do not distribute
    Labels

    View Slide

  72. Google confidential │ Do not distribute
    Labels
    - "release" : "stable", "canary", …

    View Slide

  73. Google confidential │ Do not distribute
    Labels
    - "release" : "stable", "canary", …
    - "environment" : "dev", "qa", "production" ...

    View Slide

  74. Google confidential │ Do not distribute
    Labels
    - "release" : "stable", "canary", …
    - "environment" : "dev", "qa", "production" ...
    - "tier" : "frontend", "backend", "middleware", …

    View Slide

  75. Google confidential │ Do not distribute
    Labels
    - "release" : "stable", "canary", …
    - "environment" : "dev", "qa", "production" ...
    - "tier" : "frontend", "backend", "middleware", …
    - "partition" : "customerA", "customerB", …

    View Slide

  76. Google confidential │ Do not distribute
    Labels
    - "release" : "stable", "canary", …
    - "environment" : "dev", "qa", "production" ...
    - "tier" : "frontend", "backend", "middleware", …
    - "partition" : "customerA", "customerB", …
    - "track" : "daily", "weekly", ...

    View Slide

  77. Google confidential │ Do not distribute
    Labels
    Arbitrary metadata
    Attached to any API object
    Generally represent identity

    View Slide

  78. Google confidential │ Do not distribute
    Labels
    Arbitrary metadata
    Attached to any API object
    Generally represent identity
    Queryable by selectors
    • think SQL ‘select ... where ...’
    The only grouping mechanism
    • pods under a ReplicationController
    • pods in a Service
    • capabilities of a node (constraints)
    Example: “phase: canary”

    View Slide

  79. Google confidential │ Do not distribute
    Selectors
    App: Nifty
    Phase: Dev
    Role: FE
    App: Nifty
    Phase: Test
    Role: FE
    App: Nifty
    Phase: Dev
    Role: BE
    App: Nifty
    Phase: Test
    Role: BE

    View Slide

  80. Google confidential │ Do not distribute
    App == Nifty
    App: Nifty
    Phase: Dev
    Role: FE
    App: Nifty
    Phase: Test
    Role: FE
    App: Nifty
    Phase: Dev
    Role: BE
    App: Nifty
    Phase: Test
    Role: BE
    Selectors

    View Slide

  81. Google confidential │ Do not distribute
    App == Nifty
    Role == FE
    App: Nifty
    Phase: Dev
    Role: FE
    App: Nifty
    Phase: Test
    Role: FE
    App: Nifty
    Phase: Dev
    Role: BE
    App: Nifty
    Phase: Test
    Role: BE
    Selectors

    View Slide

  82. Google confidential │ Do not distribute
    App == Nifty
    Role == BE
    App: Nifty
    Phase: Dev
    Role: FE
    App: Nifty
    Phase: Test
    Role: FE
    App: Nifty
    Phase: Dev
    Role: BE
    App: Nifty
    Phase: Test
    Role: BE
    Selectors

    View Slide

  83. Google confidential │ Do not distribute
    App == Nifty
    Phase == Dev
    App: Nifty
    Phase: Dev
    Role: FE
    App: Nifty
    Phase: Test
    Role: FE
    App: Nifty
    Phase: Dev
    Role: BE
    App: Nifty
    Phase: Test
    Role: BE
    Selectors

    View Slide

  84. Google confidential │ Do not distribute
    App == Nifty
    Phase == Test
    App: Nifty
    Phase: Dev
    Role: FE
    App: Nifty
    Phase: Test
    Role: FE
    App: Nifty
    Phase: Dev
    Role: BE
    App: Nifty
    Phase: Test
    Role: BE
    Selectors

    View Slide

  85. Google confidential │ Do not distribute
    Replication Controllers
    Canonical example of control loops
    Runs out-of-process wrt API server
    Have 1 job: ensure N copies of a pod
    • if too few, start new ones
    • if too many, kill some
    • group == selector
    Cleanly layered on top of the core
    • all access is by public APIs
    Replicated pods are fungible
    • No implied ordinality or identity
    Replication Controller
    - Name = “nifty-rc”
    - Selector = {“App”: “Nifty”}
    - PodTemplate = { ... }
    - NumReplicas = 4
    API Server
    How
    many?
    3
    Start 1
    more
    OK
    How
    many?
    4

    View Slide

  86. Google confidential │ Do not distribute
    Replication Controllers
    node 1
    f0118
    node 3
    node 4
    node 2
    d9376
    b0111
    a1209
    Replication Controller
    - Desired = 4
    - Current = 4

    View Slide

  87. Google confidential │ Do not distribute
    Replication Controllers
    node 1
    f0118
    node 3
    node 4
    node 2
    Replication Controller
    - Desired = 4
    - Current = 4
    d9376
    b0111
    a1209

    View Slide

  88. Google confidential │ Do not distribute
    Replication Controllers
    node 1
    f0118
    node 3
    node 4
    Replication Controller
    - Desired = 4
    - Current = 3
    b0111
    a1209

    View Slide

  89. Google confidential │ Do not distribute
    Replication Controllers
    node 1
    f0118
    node 3
    node 4
    Replication Controller
    - Desired = 4
    - Current = 4
    b0111
    a1209
    c9bad

    View Slide

  90. Google confidential │ Do not distribute
    Replication Controllers
    node 1
    f0118
    node 3
    node 4
    node 2
    Replication Controller
    - Desired = 4
    - Current = 5
    d9376
    b0111
    a1209
    c9bad

    View Slide

  91. Google confidential │ Do not distribute
    Replication Controllers
    node 1
    f0118
    node 3
    node 4
    node 2
    Replication Controller
    - Desired = 4
    - Current = 4
    d9376
    b0111
    a1209
    c9bad

    View Slide

  92. Google confidential │ Do not distribute
    Pod networking
    Pod IPs are routable
    • Docker default is private IP
    Pods can reach each other without NAT
    • even across nodes
    No brokering of port numbers
    This is a fundamental requirement
    • several SDN solutions

    View Slide

  93. Google confidential │ Do not distribute
    Services
    A group of pods that act as one == Service
    • group == selector
    Defines access policy
    • only “load balanced” for now
    Gets a stable virtual IP and port
    • called the service portal
    • also a DNS name
    VIP is captured by kube-proxy
    • watches the service constituency
    • updates when backends change
    Hide complexity - ideal for non-native apps
    Portal (VIP)
    Client

    View Slide

  94. Google confidential │ Do not distribute
    Services
    10.0.0.1 : 9376
    Client
    kube-proxy
    Service
    - Name = “nifty-svc”
    - Selector = {“App”: “Nifty”}
    - Port = 9376
    - ContainerPort = 8080
    Portal IP is assigned
    iptables
    DNAT
    TCP / UDP
    apiserver
    watch
    10.240.2.2 : 8080
    10.240.1.1 : 8080 10.240.3.3 : 8080
    TCP / UDP

    View Slide

  95. Google confidential │ Do not distribute
    Services
    A group of pods that act as one == Service
    • group == selector
    Defines access policy
    • only “load balanced” for now
    Gets a stable virtual IP and port
    • called the service portal
    • also a DNS name
    VIP is captured by kube-proxy
    • watches the service constituency
    • updates when backends change
    Hide complexity - ideal for non-native apps
    Portal (VIP)
    Client

    View Slide

  96. Google confidential │ Do not distribute
    WATCH
    Services,
    Endpoints
    Services
    kube-proxy
    apiserver

    View Slide

  97. Google confidential │ Do not distribute
    Services
    kube-proxy
    Pod
    - Name = “pod1”
    - Labels = {“App”: “Nifty”}
    - Port = 9376
    apiserver
    POST
    pods
    WATCH
    Services,
    Endpoints

    View Slide

  98. Google confidential │ Do not distribute
    Services
    kube-proxy
    apiserver
    pod1
    10.240.1.1 : 9376
    pod2
    10.240.2.2 : 9376
    pod3
    10.240.3.3 : 9376
    run
    pods
    Pod
    - Name = “pod1”
    - Labels = {“App”: “Nifty”}
    - Port = 9376
    WATCH
    Services,
    Endpoints

    View Slide

  99. Google confidential │ Do not distribute
    POST
    service
    pod1
    10.240.1.1 : 9376
    pod2
    10.240.2.2 : 9376
    pod3
    10.240.3.3 : 9376
    Services
    kube-proxy
    Service
    - Name = “nifty-svc”
    - Selector = {“App”: “Nifty”}
    - Port = 80
    - TargetPort = 9376
    - PortalIP - 10.9.8.7
    apiserver
    WATCH
    Services,
    Endpoints

    View Slide

  100. Google confidential │ Do not distribute
    pod1
    10.240.1.1 : 9376
    pod2
    10.240.2.2 : 9376
    pod3
    10.240.3.3 : 9376
    Services
    kube-proxy
    apiserver
    Service
    - Name = “nifty-svc”
    - Selector = {“App”: “Nifty”}
    - Port = 80
    - TargetPort = 9376
    - PortalIP - 10.9.8.7
    WATCH
    Services,
    Endpoints
    new
    service!

    View Slide

  101. Google confidential │ Do not distribute
    pod1
    10.240.1.1 : 9376
    pod2
    10.240.2.2 : 9376
    pod3
    10.240.3.3 : 9376
    Services
    kube-proxy
    apiserver
    Linux
    listen on
    port X
    (random)
    Service
    - Name = “nifty-svc”
    - Selector = {“App”: “Nifty”}
    - Port = 80
    - TargetPort = 9376
    - PortalIP - 10.9.8.7
    WATCH
    Services,
    Endpoints

    View Slide

  102. Google confidential │ Do not distribute
    pod1
    10.240.1.1 : 9376
    pod2
    10.240.2.2 : 9376
    pod3
    10.240.3.3 : 9376
    Services
    kube-proxy
    apiserver
    Linux
    listen on
    port X
    iptables
    redirect 10.9.8.7:80
    to localhost:X
    Service
    - Name = “nifty-svc”
    - Selector = {“App”: “Nifty”}
    - Port = 80
    - TargetPort = 9376
    - PortalIP - 10.9.8.7
    WATCH
    Services,
    Endpoints

    View Slide

  103. Google confidential │ Do not distribute
    pod1
    10.240.1.1 : 9376
    pod2
    10.240.2.2 : 9376
    pod3
    10.240.3.3 : 9376
    Services
    kube-proxy
    apiserver
    Linux
    listen on
    port X
    iptables
    redirect 10.9.8.7:80
    to localhost:X
    Service
    - Name = “nifty-svc”
    - Selector = {“App”: “Nifty”}
    - Port = 80
    - TargetPort = 9376
    - PortalIP - 10.9.8.7
    WATCH
    Services,
    Endpoints
    new
    endpoints!

    View Slide

  104. Google confidential │ Do not distribute
    pod1
    10.240.1.1 : 9376
    pod2
    10.240.2.2 : 9376
    pod3
    10.240.3.3 : 9376
    Services
    kube-proxy
    apiserver
    Linux
    listen on
    port X
    iptables
    redirect 10.9.8.7:80
    to localhost:X
    Service
    - Name = “nifty-svc”
    - Selector = {“App”: “Nifty”}
    - Port = 80
    - TargetPort = 9376
    - PortalIP - 10.9.8.7

    View Slide

  105. Google confidential │ Do not distribute
    pod1
    10.240.1.1 : 9376
    pod2
    10.240.2.2 : 9376
    pod3
    10.240.3.3 : 9376
    Services
    kube-proxy
    apiserver
    Linux
    listen on
    port X
    iptables
    Client
    redirect 10.9.8.7:80
    to localhost:X
    Service
    - Name = “nifty-svc”
    - Selector = {“App”: “Nifty”}
    - Port = 80
    - TargetPort = 9376
    - PortalIP - 10.9.8.7
    connect to
    10.9.8.7:80

    View Slide

  106. Google confidential │ Do not distribute
    pod1
    10.240.1.1 : 9376
    pod2
    10.240.2.2 : 9376
    pod3
    10.240.3.3 : 9376
    Services
    kube-proxy
    apiserver
    Linux
    listen on
    port X
    iptables
    Client
    redirect 10.9.8.7:80
    to localhost:X
    Service
    - Name = “nifty-svc”
    - Selector = {“App”: “Nifty”}
    - Port = 80
    - TargetPort = 9376
    - PortalIP - 10.9.8.7
    connect to
    10.9.8.7:80

    View Slide

  107. Google confidential │ Do not distribute
    pod1
    10.240.1.1 : 9376
    pod2
    10.240.2.2 : 9376
    pod3
    10.240.3.3 : 9376
    Services
    kube-proxy
    apiserver
    Linux
    iptables
    Client
    Service
    - Name = “nifty-svc”
    - Selector = {“App”: “Nifty”}
    - Port = 80
    - TargetPort = 9376
    - PortalIP - 10.9.8.7
    connect to
    localhost:X

    View Slide

  108. Google confidential │ Do not distribute
    pod1
    10.240.1.1 : 9376
    pod2
    10.240.2.2 : 9376
    pod3
    10.240.3.3 : 9376
    Services
    kube-proxy
    apiserver
    Linux
    listen on port X
    iptables
    Client
    Service
    - Name = “nifty-svc”
    - Selector = {“App”: “Nifty”}
    - Port = 80
    - TargetPort = 9376
    - PortalIP - 10.9.8.7
    proxy for client

    View Slide

  109. Google confidential │ Do not distribute
    Events
    A central place for information about your cluster
    • filed by any component: kubelet, scheduler, etc
    Real-time information on the current state of your pod
    • kubectl describe pod foo
    Real-time information on the current state of your cluster
    • kubectl get --watch-only events
    • You can also ask only for events that mention some object you care about.

    View Slide

  110. Google confidential │ Do not distribute
    Monitoring
    Optional add-on to Kubernetes clusters
    Run cAdvisor as a pod on each node
    • gather stats from all containers
    • export via REST
    Run Heapster as a pod in the cluster
    • just another pod, no special access
    • aggregate stats
    Run Influx and Grafana in the cluster
    • more pods
    • alternately: store in Google Cloud Monitoring

    View Slide

  111. Google confidential │ Do not distribute
    Logging
    Optional add-on to Kubernetes clusters
    Run fluentd as a pod on each node
    • gather logs from all containers
    • export to elasticsearch
    Run Elasticsearch as a pod in the cluster
    • just another pod, no special access
    • aggregate logs
    Run Kibana in the cluster
    • yet another pod
    • alternately: store in Google Cloud Logging

    View Slide

  112. DEMO

    View Slide

  113. Google confidential │ Do not distribute
    Kubernetes is Open Source
    We want your help!
    http://kubernetes.io
    https://github.com/GoogleCloudPlatform/kubernetes
    irc.freenode.net #google-containers
    @kubernetesio

    View Slide

  114. Google confidential │ Do not distribute
    Questions?
    Images by Connie Zhou
    http://kubernetes.io

    View Slide