$30 off During Our Annual Pro Sale. View Details »

Scheduling Applications at Scale with Nomad

Scheduling Applications at Scale with Nomad

Session details on Linux Foundation OSS Summit Website: https://ossna18.sched.com/event/FAP0/scheduling-applications-at-scale-with-nomad-anubhav-mishra-hashicorp

Scheduler frameworks enable reliable and repeatable application deploys. In this session, attendees will use Nomad, a single binary cluster scheduler, to build a multi-region, self-healing production environment that runs a diverse set of workloads. They will also get hands on experience in writing and submitting job specifications, interacting with the API, and deployment strategies. This session will cover the following:
- Nomad Overview
- Installing and Configuring Nomad
- Creating, Running, and Inspecting Jobs
- Service Registration
- Interacting via the HTTP API
- Advanced Job Strategies (rolling updates, blue-green)

Anubhav Mishra

August 29, 2018
Tweet

More Decks by Anubhav Mishra

Other Decks in Technology

Transcript

  1. Scheduling Applications at Scale with
    Nomad
    Photo by Sašo Tušar on Unsplash
    @anubhavm

    View Slide

  2. @anubhavm
    Anubhav Mishra
    Developer Advocate, HashiCorp

    View Slide

  3. @anubhavm
    Anubhav Mishra
    Developer Advocate, HashiCorp
    has stickers
    Atlan&s
    Gopher Artwork by Ashley McNamara

    View Slide

  4. PROVISION, SECURE AND RUN ANY INFRASTRUCTURE
    Nomad Consul
    Vault
    Vagrant Packer Terraform
    Consul Enterprise
    Terraform Enterprise
    Vault Enterprise
    PRODUCT SUITE
    OSS TOOL SUITE
    RUN
    Applications
    SECURE
    Application Infrastructure
    PROVISION
    Infrastructure
    FOR INDIVIDUALS FOR TEAMS
    Nomad Enterprise

    View Slide

  5. View Slide

  6. Copyright © 2017 HashiCorp
    @anubhavm
     !6
    Globally Distributed
    Optimistically Concurrent
    Scheduler

    View Slide

  7. Copyright © 2017 HashiCorp
    @anubhavm
     !7
    Globally Distributed
    Optimistically Concurrent
    Scheduler

    View Slide

  8. Scheduler
    Photo by Emma Matthews on Unsplash

    View Slide

  9. Copyright © 2017 HashiCorp
    @anubhavm
     !9
    A person or machine that helps scheduling during
    each day of work.
    Scheduler

    View Slide

  10. Copyright © 2017 HashiCorp
    @anubhavm
     !10

    View Slide

  11. Copyright © 2017 HashiCorp
    @anubhavm
     !11
    A computer program that controls or manages the
    execution of jobs / processes / operations.
    Scheduler (Computing)

    View Slide

  12. Copyright © 2017 HashiCorp
    @anubhavm
     !12

    View Slide

  13. Copyright © 2017 HashiCorp
    @anubhavm
     !13
    Schedulers map a set of work to a set of resources

    View Slide

  14. Copyright © 2017 HashiCorp
    @anubhavm

    CPU Scheduler
    !14
    CORE
    CORE
    CORE
    CORE
    CPU
    SCHEDULER
    KERNEL
    APACHE
    REDIS
    BASH

    View Slide

  15. Copyright © 2017 HashiCorp
    @anubhavm

    CPU Scheduler
    !15
    CORE
    CORE
    CPU
    SCHEDULER
    KERNEL
    APACHE
    REDIS
    BASH

    View Slide

  16. Copyright © 2017 HashiCorp
    @anubhavm

    CPU Scheduler
    !16
    CORE
    CORE
    CPU
    SCHEDULER
    KERNEL
    APACHE
    REDIS
    BASH

    View Slide

  17. Copyright © 2017 HashiCorp
    @anubhavm

    CPU Scheduler
    !17
    CORE
    CORE
    CPU
    SCHEDULER
    KERNEL
    APACHE
    REDIS
    BASH

    View Slide

  18. Copyright © 2017 HashiCorp
    @anubhavm

    Scheduler Advantages
    !18
    Higher Resource Utilization
    Decouple Work from Resources
    Better Quality of Service

    View Slide

  19. Copyright © 2017 HashiCorp
    @anubhavm

    Nop! Schedulers Aren’t New Concept
    !19

    View Slide

  20. Copyright © 2017 HashiCorp
    @anubhavm

    Landscape
    !20

    View Slide

  21. View Slide

  22. Designing

    View Slide

  23. @anubhavm
     !23
    Multi-Datacenter
    Multi-Region
    Flexible Workloads
    Job Priorities
    Bin Packing
    Large Scale
    Operationally Simple

    View Slide

  24. Copyright © 2017 HashiCorp
    @anubhavm

    Thousands of regions
    Tens of thousands of clients per region
    Thousands of jobs per region
    Scaling Requirements
    !24

    View Slide

  25. Copyright © 2017 HashiCorp
    @anubhavm

    Our Past Experience
    !25
    GOSSIP CONSENSUS

    View Slide

  26. @anubhavm
     !26
    Cluster Management
    Gossip Based (P2P)
    Membership
    Failure Detection
    Event System
    Serf

    View Slide

  27. @anubhavm
     !27
    Service Mesh
    Service Discovery
    Configuration
    Coordination (Locking)
    Central Servers +
    Distributed Clients

    View Slide

  28. Copyright © 2017 HashiCorp
    @anubhavm

    Our Past Experience
    !28
    GOSSIP CONSENSUS
    Mature Libraries Proven Design Patterns

    View Slide

  29. Copyright © 2017 HashiCorp
    @anubhavm

    Our Past Experience
    !29
    GOSSIP CONSENSUS
    Mature Libraries Proven Design Patterns
    ?

    View Slide

  30. Copyright © 2017 HashiCorp
    @anubhavm

    Our Past Experience
    !30
    GOSSIP CONSENSUS

    View Slide

  31. View Slide

  32. @anubhavm
     !32
    Inspired by Google Omega
    Optimistic Concurrency
    State Coordination
    Service & Batch workloads
    Pluggable Architecture

    View Slide

  33. Copyright © 2017 HashiCorp
    @anubhavm

    Consul Cluster
    !33
    CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT
    SERVER SERVER SERVER
    REPLICATION REPLICATION
    RPC
    RPC
    LAN GOSSIP
    SERVER SERVER SERVER
    REPLICATION REPLICATION
    WAN GOSSIP

    View Slide

  34. Copyright © 2017 HashiCorp
    @anubhavm

    Single Region Architecture
    !34
    SERVER SERVER SERVER
    CLIENT CLIENT CLIENT
    DC1 DC2 DC3
    FOLLOWER LEADER FOLLOWER
    REPLICATION
    FORWARDING
    REPLICATION
    FORWARDING
    RPC RPC RPC

    View Slide

  35. Copyright © 2017 HashiCorp
    @anubhavm

    Multi Region Architecture
    !35
    SERVER SERVER SERVER
    FOLLOWER LEADER FOLLOWER
    REPLICATION
    FORWARDING
    REPLICATION
    REGION B

    GOSSIP
    REPLICATION REPLICATION
    FORWARDING
    REGION FORWARDING
     REGION A
    SERVER
    FOLLOWER
    SERVER SERVER
    LEADER FOLLOWER

    View Slide

  36. Copyright © 2017 HashiCorp
    @anubhavm
     !36
    Region is Isolation Domain
    1-N Datacenters Per Region
    Flexibility to do 1:1 (Consul)
    Scheduling Boundary

    View Slide

  37. Installing and
    Configuring

    View Slide

  38. Copyright © 2017 HashiCorp
    @anubhavm

    Installing Nomad
    !38
    Client Server

    View Slide

  39. Copyright © 2017 HashiCorp
    @anubhavm

    Fingerprinting
    !39
    Type Examples
    Operating System Kernel, OS, Version
    Hardware CPU, Memory, Disk
    Apps (Capabilities) Docker, Java, Consul
    Environment AWS, GCE

    View Slide

  40. Copyright © 2017 HashiCorp
    @anubhavm
     !40
    Constrain Placement and Bin Pack

    View Slide

  41. Copyright © 2017 HashiCorp
    @anubhavm
     !41
    “Task Requires Linux, Docker, and PCI-Compliant Hardware”
    expressed as constraints in job file

    View Slide

  42. Copyright © 2017 HashiCorp
    @anubhavm
     !42
    “Task needs 512MB RAM and 1 Core”
    expressed as resource in job file

    View Slide

  43. Terminal
    $ nomad agent -config=/path/to/config.hcl

    View Slide

  44. Terminal
    $ echo "let’s try it?"

    View Slide

  45. Job File

    View Slide

  46. @anubhavm
     !X
    job "redis" {
    datacenters = ["us-east-1"]
    task "redis" {
    driver = "docker"
    config { image = "redis:latest" }
    resources {
    cpu = 500 # Mhz
    memory = 256 # MB
    network {
    mbits = 10
    port "redis" {}
    }
    }
    }
    }

    View Slide

  47. @anubhavm
     !X
    job "webserver" {
    datacenters = ["us-east-1"]
    task "webserver" {
    driver = "exec"
    config {
    command = "yet-another-golang-webserver"
    }
    artifact {
    source = "https://github.com/download/../yet-another-golang-webserver"
    }
    resources {
    cpu = 500 # Mhz
    memory = 128 # MB
    network {
    port "http" {
    static = 8080
    }
    }
    }
    }
    }

    View Slide

  48. Terminal
    $ echo "let’s try it?"

    View Slide

  49. Copyright © 2017 HashiCorp
    @anubhavm
     !49
    Job specification declares what to run

    View Slide

  50. Copyright © 2017 HashiCorp
    @anubhavm
     !50
    Nomad determines how and where to run

    View Slide

  51. Copyright © 2017 HashiCorp
    @anubhavm
     !51
    Nomad abstracts work from resources

    View Slide

  52. Job Types

    View Slide

  53. Copyright © 2017 HashiCorp
    @anubhavm

    Nomad has three scheduler types that can be used when creating
    your job: service, batch, and system.
    Job Types
    !53

    View Slide

  54. Copyright © 2017 HashiCorp
    @anubhavm

    Service Scheduler Job Type
    The service scheduler is designed for scheduling long-lived services
    that should never go down. The service scheduler ranks a large
    portion of the nodes that meet the jobs constraints and selects the
    optimal node to place a task group on.
    Examples: webapp, redis
    Job Types: Service
    !54

    View Slide

  55. Copyright © 2017 HashiCorp
    @anubhavm

    Batch Scheduler Job Type
    Batch jobs are less sensitive to short-term performance fluctuations
    and are short lived, finishing after some period.
    Examples: billing, data replication
    Job Types: Batch
    !55

    View Slide

  56. Copyright © 2017 HashiCorp
    @anubhavm

    System Scheduler Job Type
    The system scheduler is used to register jobs that should be run on
    all clients that meet the job's constraints. The system scheduler is
    also invoked when clients join the cluster or transition into the ready
    state.
    Examples: logging agent, security auditing tool
    Job Types: System
    !56

    View Slide

  57. Drivers

    View Slide

  58. @anubhavm
     !58
    Containerized
    Virtualized
    Standalone
    Docker
    Qemu / KVM
    Java Jar
    Static Binaries
    rkt
    LXC

    View Slide

  59. @anubhavm
     !59
    Containerized
    Virtualized
    Standalone
    Docker
    Qemu / KVM
    Java Jar
    Static Binaries
    rkt
    LXC
    Windows Server Containers
    Hyper-V
    Xen
    C#

    View Slide

  60. Service Discovery

    View Slide

  61. Copyright © 2017 HashiCorp
    @anubhavm

    Consul is a free and open-source tool by HashiCorp that implements
    service discovery.
    It uses the RAFT and gossip protocols to reach massive scale.
    It has integrations with health checks, so unhealthy services are not
    added to the service discovery layer.
    Similar client-server model to Nomad.
    Service Discovery using Consul
    !61

    View Slide

  62. @anubhavm
     !X
    job "redis" {
    datacenters = [“us-east-1”]
    service {
    name = "redis"
    port = "redis"
    check {
    type = "tcp"
    port = "redis"
    interval = "10s"
    timeout = "5s"
    }
    }
    }

    View Slide

  63. Terminal
    $ echo "let’s try it?"

    View Slide

  64. Advanced Job
    Strategies

    View Slide

  65. Copyright © 2017 HashiCorp
    @anubhavm

    Nomad currently supports two advanced job strategies:
    Rolling Upgrades, Blue/Green & Canary Deployments
    Advanced Job Strategies
    !65

    View Slide

  66. @anubhavm
     !X
    job "redis" {
    # Add an update stanza to enable rolling updates of the service
    update {
    max_parallel = 1
    min_healthy_time = "30s"
    healthy_deadline = "10m"
    }
    .....
    }

    View Slide

  67. @anubhavm
     !X
    job "redis" {
    # Add an update stanza to enable rolling updates of the service
    update {
    max_parallel = 1
    canary = 5
    min_healthy_time = "30s"
    auto_revert = true
    }
    .....
    }

    View Slide

  68. Terminal
    $ echo "let’s try it?"

    View Slide

  69. @anubhavm
     !69
    Multi-Datacenter
    Multi-Region
    Flexible Workloads
    Job Priorities
    Bin Packing
    Large Scale
    Operationally Simple

    View Slide

  70. @anubhavm
     !70
    Multi-Datacenter
    Multi-Region
    Flexible Workloads
    Job Priorities
    Bin Packing
    Large Scale
    Operationally Simple

    View Slide

  71. s
    Copyright © 2017 HashiCorp @anubhavm
     !71
    ONE
    MORE
    DEMO

    View Slide

  72. Nomad
    Million Container
    Challenge
    1,000 Jobs
    1,000 Tasks per Job
    5,000 Hosts on GCE
    1,000,000 Containers

    View Slide

  73. Copyright © 2017 HashiCorp
    @anubhavm
     !73

    View Slide

  74. View Slide

  75. Copyright © 2017 HashiCorp
    @anubhavm
     !75
    “640 KB ought to be enough for anybody.”
    - Bill Gates

    View Slide

  76. Copyright © 2017 HashiCorp
    @anubhavm
     !76
    2nd Largest Hedge Fund
    18K Cores
    5 Hours
    2,200 Containers/second

    View Slide

  77. Copyright © 2017 HashiCorp
    @anubhavm
     !77
    CircleCI 7+ Million Builds a Month
    Sustain 400-1000 Jobs a Minute
    Great Talk By Danielle Tomlinson: https://youtu.be/b8NQO_vFAYo

    View Slide

  78. Copyright © 2017 HashiCorp
    @anubhavm
     !78
    Globally Distributed
    Optimistically Concurrent
    Scheduler

    View Slide

  79. Copyright © 2017 HashiCorp
    @anubhavm
     !79
    Higher Resource Utilization
    Decouple Work from Resources
    Better Quality of Service

    View Slide

  80. www.hashicorp.com
    FOR EVERYONE, EVERYWHERE
    Thank You!
    I have stickers!
    Ask me anything. @anubhavm
    Anubhav Mishra

    View Slide