Upgrade to Pro — share decks privately, control downloads, hide ads and more …

cluster management, cognitive shifts, & 
kubernetes

cluster management, cognitive shifts, & 
kubernetes

How did we get into the cluster management game? This briefly covers some of the lessons that we have learned over the years from the system internally known as Borg. Then, we step through an application running on kubernetes and see how those lessons are present in what kubernetes does and how it's designed.

juliaferraioli

July 07, 2015
Tweet

More Decks by juliaferraioli

Other Decks in Technology

Transcript

  1. cluster management,
    cognitive shifts, &

    kubernetes
    seattle.rb
    @juliaferraioli, kubernautical advocate

    View Slide

  2. cluster management
    with Borg

    View Slide

  3. cluster management 


    with Borg
    the system we
    internally call

    View Slide

  4. @juliaferraioli Image by Connie Zhou

    View Slide

  5. @juliaferraioli
    developer view

    View Slide

  6. @juliaferraioli
    job hello_world = {
    developer view

    View Slide

  7. @juliaferraioli
    job hello_world = {
    runtime = { cell = 'ic' } // Cell (cluster) to run in
    developer view

    View Slide

  8. @juliaferraioli
    job hello_world = {
    runtime = { cell = 'ic' } // Cell (cluster) to run in
    binary = '.../hello_world_webserver' // Program to run
    developer view

    View Slide

  9. @juliaferraioli
    job hello_world = {
    runtime = { cell = 'ic' } // Cell (cluster) to run in
    binary = '.../hello_world_webserver' // Program to run
    args = { port = '%port%' } // Command line parameters
    developer view

    View Slide

  10. @juliaferraioli
    job hello_world = {
    runtime = { cell = 'ic' } // Cell (cluster) to run in
    binary = '.../hello_world_webserver' // Program to run
    args = { port = '%port%' } // Command line parameters
    requirements = { // Resource requirements
    developer view

    View Slide

  11. @juliaferraioli
    job hello_world = {
    runtime = { cell = 'ic' } // Cell (cluster) to run in
    binary = '.../hello_world_webserver' // Program to run
    args = { port = '%port%' } // Command line parameters
    requirements = { // Resource requirements
    ram = 100M
    developer view

    View Slide

  12. @juliaferraioli
    job hello_world = {
    runtime = { cell = 'ic' } // Cell (cluster) to run in
    binary = '.../hello_world_webserver' // Program to run
    args = { port = '%port%' } // Command line parameters
    requirements = { // Resource requirements
    ram = 100M
    disk = 100M
    developer view

    View Slide

  13. @juliaferraioli
    job hello_world = {
    runtime = { cell = 'ic' } // Cell (cluster) to run in
    binary = '.../hello_world_webserver' // Program to run
    args = { port = '%port%' } // Command line parameters
    requirements = { // Resource requirements
    ram = 100M
    disk = 100M
    cpu = 0.1
    developer view

    View Slide

  14. @juliaferraioli
    job hello_world = {
    runtime = { cell = 'ic' } // Cell (cluster) to run in
    binary = '.../hello_world_webserver' // Program to run
    args = { port = '%port%' } // Command line parameters
    requirements = { // Resource requirements
    ram = 100M
    disk = 100M
    cpu = 0.1
    }
    developer view

    View Slide

  15. @juliaferraioli
    job hello_world = {
    runtime = { cell = 'ic' } // Cell (cluster) to run in
    binary = '.../hello_world_webserver' // Program to run
    args = { port = '%port%' } // Command line parameters
    requirements = { // Resource requirements
    ram = 100M
    disk = 100M
    cpu = 0.1
    }
    replicas = 5 // Number of tasks
    developer view

    View Slide

  16. @juliaferraioli
    job hello_world = {
    runtime = { cell = 'ic' } // Cell (cluster) to run in
    binary = '.../hello_world_webserver' // Program to run
    args = { port = '%port%' } // Command line parameters
    requirements = { // Resource requirements
    ram = 100M
    disk = 100M
    cpu = 0.1
    }
    replicas = 5 // Number of tasks
    }
    developer view

    View Slide

  17. @juliaferraioli
    job hello_world = {
    runtime = { cell = 'ic' } // Cell (cluster) to run in
    binary = '.../hello_world_webserver' // Program to run
    args = { port = '%port%' } // Command line parameters
    requirements = { // Resource requirements
    ram = 100M
    disk = 100M
    cpu = 0.1
    }
    replicas = 5 // Number of tasks
    }
    10000
    developer view

    View Slide

  18. @juliaferraioli Image by Connie Zhou
    Hello
    world!
    Hello
    world!
    Hello
    world! Hello
    world!
    Hello
    world! Hello
    world!
    Hello
    world!
    Hello
    world!

    View Slide

  19. @juliaferraioli
    Hello
    world!
    Hello
    world!
    Hello
    world!
    Hello
    world!
    Hello
    world! Hello
    world! Hello
    world!
    Hello
    world!
    Hello
    world!
    Hello
    world!
    Hello
    world!
    Hello
    world!
    Hello
    world!
    Hello
    world!
    Hello
    world!
    Hello
    world!
    Hello
    world!
    Hello
    world!
    Hello
    world!
    Hello
    world!
    Hello
    world!
    Hello
    world!
    Hello
    world! Hello
    world!
    Hello
    world!
    Hello
    world!
    Hello
    world!
    Image by Connie Zhou
    Hello
    world!
    Hello
    world!
    Hello
    world! Hello
    world!
    Hello
    world! Hello
    world!
    Hello
    world!
    Hello
    world!
    Hello
    world!
    Hello
    world!
    Hello
    world! Hello
    world!
    Hello
    world! Hello
    world!
    Hello
    world!
    Hello
    world!
    Hello
    world!
    Hello
    world!
    Hello
    world! Hello
    world!
    Hello
    world! Hello
    world!
    Hello
    world!
    Hello
    world!

    View Slide

  20. @juliaferraioli
    developer view

    View Slide

  21. @juliaferraioli
    developer view

    View Slide

  22. @juliaferraioli
    task-eviction
    rates and causes
    failures

    View Slide

  23. @juliaferraioli
    advanced bin-
    packing
    algorithms
    experimental placement
    of production VM
    workload, July 2014
    one

    machine
    efficiency

    View Slide

  24. @juliaferraioli
    advanced bin-
    packing
    algorithms
    experimental placement
    of production VM
    workload, July 2014
    available resources
    one

    machine
    efficiency

    View Slide

  25. @juliaferraioli
    advanced bin-
    packing
    algorithms
    experimental placement
    of production VM
    workload, July 2014
    stranded resources
    available resources
    one

    machine
    efficiency

    View Slide

  26. @juliaferraioli
    Borg paper:
    http://goo.gl/1C4nuo
    Images by Connie Zhou
    observations:
    1. resiliency is achieved only by ruthless
    attention to detail
    a. ubiquitous software fault tolerance
    b. persistent, declarative specs

    2. we get efficiency by:
    a. sharing resources
    b. reclaiming unused allocations
    1. containers make users more productive

    View Slide

  27. @juliaferraioli
    app
    libs
    kernel
    libs
    app app
    kernel
    app
    libs
    libs
    kernel
    kernel
    not so long ago: virtual machines

    View Slide

  28. @juliaferraioli
    libs
    app
    kernel
    libs
    app
    libs
    app
    libs
    app
    the new kid in town: containers

    View Slide

  29. A fundamentally different way of managing
    applications

    View Slide

  30. enter kubernetes,
    stage center

    View Slide

  31. @juliaferraioli
    κυβερνήτης:
    “Helmsman”;
    root of “Governor” and “Cybernetic”
    manage applications, not machines
    kubernetes (or k8s)

    View Slide

  32. demo, part 1

    View Slide

  33. @juliaferraioli

    View Slide

  34. @juliaferraioli
    demo overview
    Server
    Debian
    Docker
    Engine
    PHP
    libs
    Redis
    libs

    View Slide

  35. @juliaferraioli
    A loop that drives
    current state towards
    desired state
    A small group of
    tightly coupled
    containers
    Pod
    Replication
    Controller Service
    A set of running pods
    that work together
    Arbitrary metadata to
    organize components
    Labels
    kubernetes cluster

    View Slide

  36. @juliaferraioli
    Dashboard
    show: type = FE
    Object Pod
    frontend
    Object
    type = FE type = FE
    ● metadata with semantic meaning
    ● membership identifier
    ● the only grouping mechanism
    behavior benefits
    ➔ allow for intent of many users (e.g. dashboards)
    ➔ build higher level systems
    ➔ queryable by selectors
    labels

    View Slide

  37. @juliaferraioli
    Dashboard
    show: type = FE
    Object Pod
    frontend
    Pod
    frontend
    Object Object
    Dashboard
    show: version = v2
    type = FE
    version = v2
    type = FE type = FE
    version = v2
    ● metadata with semantic meaning
    ● membership identifier
    ● the only grouping mechanism
    behavior benefits
    ➔ allow for intent of many users (e.g. dashboards)
    ➔ build higher level systems
    ➔ queryable by selectors
    labels

    View Slide

  38. @juliaferraioli
    A loop that drives
    current state towards
    desired state
    A set of running pods
    that work together
    Replication
    Controller Service
    Arbitrary metadata to
    organize components
    Labels
    A small group of
    tightly coupled
    containers
    Pod
    kubernetes cluster

    View Slide

  39. @juliaferraioli
    small group of containers & volumes
    containers are tightly coupled
    the atom of cluster scheduling
    shared namespace
    • Shared network IP and port namespace
    ephemeral
    • can die and be replaced Pod
    Site
    generator
    Web Server
    Volume
    Consumers
    Content
    Manager
    pods

    View Slide

  40. @juliaferraioli
    pods
    template:

    metadata:

    labels:

    name: frontend

    spec:

    containers:

    - name: php-redis

    image: kubernetes/example-guestbook-php-redis:v2

    ports:

    - containerPort: 80

    View Slide

  41. @juliaferraioli
    A set of running pods
    that work together
    Service
    Arbitrary metadata to
    organize components
    Labels
    A small group of
    tightly coupled
    containers
    A loop that drives
    current state towards
    desired state
    Pod
    Replication
    Controller
    kubernetes cluster

    View Slide

  42. @juliaferraioli
    Replication
    Controller
    Pod Pod
    frontend
    Pod
    frontend
    Pod Pod
    Replication
    Controller
    #pods = 1
    version = v2
    show: version = v2
    version= v1 version = v1 version = v2
    Replication
    Controller
    #pods = 2
    version = v1
    show: version = v2 Behavior Benefits
    ● keeps pods running
    ● gives direct control of pod #s
    ● grouped by label selector
    ➔ recreates pods, maintains desired state
    ➔ fine-grained control for scaling
    ➔ standard grouping semantics
    replication controllers

    View Slide

  43. @juliaferraioli
    Replication Controller
    Replication Controller
    - Name = “frontend”
    - Selector = {“Name”: “frontend”}
    - PodTemplate = { ... }
    - NumReplicas = 3
    API Server
    2
    Start 1
    more
    OK 3
    How
    many?
    How
    many?
    canonical example of control loops
    have one job: ensure N copies of a pod
    replicated pods are fungible
    replication controllers

    View Slide

  44. @juliaferraioli
    replication controllers
    kind: ReplicationController

    metadata:

    name: frontend

    labels:

    name: frontend

    spec:

    replicas: 3

    selector:

    name: frontend

    template:
    $ kubectl create -f frontend-controller.yaml

    View Slide

  45. @juliaferraioli
    kubernetes cluster
    Arbitrary metadata to
    organize components
    Labels
    A small group of
    tightly coupled
    containers
    A loop that drives
    current state towards
    desired state
    A set of running pods
    that work together
    Pod
    Replication
    Controller Service

    View Slide

  46. @juliaferraioli
    Portal (VIP)
    Client
    Pod
    Container
    Container
    Container
    Container
    Pod
    Container
    Container
    Container
    Container
    Pod
    Container
    Container
    Container
    Container
    a group of pods that act as one == Service
    defines access policy
    gets a stable virtual IP and port
    VIP is captured by kube-proxy
    services

    View Slide

  47. @juliaferraioli
    Service
    Label selectors:
    version = 1.0
    type = Frontend
    Service
    Label selectors:
    version = v1
    type = FE
    Replication
    Controller
    Pod Pod
    frontend
    Pod
    version= v1 version = v1
    Replication
    Controller
    #pods = 2
    show: version = v2
    type = FE type = FE
    VIP
    services

    View Slide

  48. @juliaferraioli
    services
    kind: Service

    metadata:

    name: frontend

    labels:

    name: frontend

    spec:

    type: LoadBalancer

    ports:

    - port: 80

    selector:

    name: frontend
    $ kubectl create -f frontend-service.yaml

    View Slide

  49. ...in technicolor!
    what’s doing what...

    View Slide

  50. @juliaferraioli
    Master
    APIs
    Scheduling
    REST
    (pods, services,
    controllers)
    AuthN
    Scheduler
    Replication
    Controller
    Node3
    Kubelet Proxy
    Pod
    Container
    Container
    Container
    Container
    Pod
    Container
    Container
    Container
    Container
    Node3
    Kubelet Proxy
    Pod
    Container
    Container
    Container
    Container
    Pod
    Container
    Container
    Container
    Container
    Node1
    Kubelet Proxy
    Pod
    Container
    Container
    Container
    Container
    Pod
    Container
    Container
    Container
    Container
    $ kubectl proxy --www=k8s-visualizer/
    visualizing kubernetes

    View Slide

  51. demo, part 2

    View Slide

  52. @juliaferraioli

    View Slide

  53. @juliaferraioli
    open sourced in June, 2014
    Google launched Google Container Engine (GKE)
    • https://cloud.google.com/container-engine/
    roadmap:
    • https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/roadmap.md
    where we are today
    http://www.kuberneteslaunch.com/

    View Slide

  54. @juliaferraioli
    find us online…
    http://kubernetes.io
    https://github.com/GoogleCloudPlatform/kubernetes
    irc.freenode.net #google-containers
    @kubernetesio

    View Slide

  55. @juliaferraioli
    thanks to...
    Brian Dorsey - Developer Advocate, Google
    Aja Hammerly - Developer Advocate, Google
    Amy Unruh - Developer Programs Engineer, Google
    Mandy Waite - Developer Advocate, Google
    John Wilkes - Principal Engineer, Google

    View Slide

  56. https://flic.kr/p/6jk7ZL

    View Slide