Scheduling Applications at Scale with Nomad

Scheduling Applications at Scale with Nomad

Session details on Linux Foundation OSS Summit Website: https://ossna18.sched.com/event/FAP0/scheduling-applications-at-scale-with-nomad-anubhav-mishra-hashicorp

Scheduler frameworks enable reliable and repeatable application deploys. In this session, attendees will use Nomad, a single binary cluster scheduler, to build a multi-region, self-healing production environment that runs a diverse set of workloads. They will also get hands on experience in writing and submitting job specifications, interacting with the API, and deployment strategies. This session will cover the following:
- Nomad Overview
- Installing and Configuring Nomad
- Creating, Running, and Inspecting Jobs
- Service Registration
- Interacting via the HTTP API
- Advanced Job Strategies (rolling updates, blue-green)

26896287bc831a13e768cea7efe29632?s=128

Anubhav Mishra

August 29, 2018
Tweet

Transcript

  1. Scheduling Applications at Scale with Nomad Photo by Sašo Tušar

    on Unsplash @anubhavm 
  2. @anubhavm Anubhav Mishra Developer Advocate, HashiCorp

  3. @anubhavm Anubhav Mishra Developer Advocate, HashiCorp has stickers Atlan&s Gopher

    Artwork by Ashley McNamara
  4. PROVISION, SECURE AND RUN ANY INFRASTRUCTURE Nomad Consul Vault Vagrant

    Packer Terraform Consul Enterprise Terraform Enterprise Vault Enterprise PRODUCT SUITE OSS TOOL SUITE RUN Applications SECURE Application Infrastructure PROVISION Infrastructure FOR INDIVIDUALS FOR TEAMS Nomad Enterprise
  5. None
  6. Copyright © 2017 HashiCorp @anubhavm  !6 Globally Distributed Optimistically

    Concurrent Scheduler
  7. Copyright © 2017 HashiCorp @anubhavm  !7 Globally Distributed Optimistically

    Concurrent Scheduler
  8. Scheduler Photo by Emma Matthews on Unsplash

  9. Copyright © 2017 HashiCorp @anubhavm  !9 A person or

    machine that helps scheduling during each day of work. Scheduler
  10. Copyright © 2017 HashiCorp @anubhavm  !10

  11. Copyright © 2017 HashiCorp @anubhavm  !11 A computer program

    that controls or manages the execution of jobs / processes / operations. Scheduler (Computing)
  12. Copyright © 2017 HashiCorp @anubhavm  !12

  13. Copyright © 2017 HashiCorp @anubhavm  !13 Schedulers map a

    set of work to a set of resources
  14. Copyright © 2017 HashiCorp @anubhavm  CPU Scheduler !14 CORE

    CORE CORE CORE CPU SCHEDULER KERNEL APACHE REDIS BASH
  15. Copyright © 2017 HashiCorp @anubhavm  CPU Scheduler !15 CORE

    CORE CPU SCHEDULER KERNEL APACHE REDIS BASH
  16. Copyright © 2017 HashiCorp @anubhavm  CPU Scheduler !16 CORE

    CORE CPU SCHEDULER KERNEL APACHE REDIS BASH
  17. Copyright © 2017 HashiCorp @anubhavm  CPU Scheduler !17 CORE

    CORE CPU SCHEDULER KERNEL APACHE REDIS BASH
  18. Copyright © 2017 HashiCorp @anubhavm  Scheduler Advantages !18 Higher

    Resource Utilization Decouple Work from Resources Better Quality of Service
  19. Copyright © 2017 HashiCorp @anubhavm  Nop! Schedulers Aren’t New

    Concept !19
  20. Copyright © 2017 HashiCorp @anubhavm  Landscape !20

  21. None
  22. Designing

  23. @anubhavm  !23 Multi-Datacenter Multi-Region Flexible Workloads Job Priorities Bin

    Packing Large Scale Operationally Simple
  24. Copyright © 2017 HashiCorp @anubhavm  Thousands of regions Tens

    of thousands of clients per region Thousands of jobs per region Scaling Requirements !24
  25. Copyright © 2017 HashiCorp @anubhavm  Our Past Experience !25

    GOSSIP CONSENSUS
  26. @anubhavm  !26 Cluster Management Gossip Based (P2P) Membership Failure

    Detection Event System Serf
  27. @anubhavm  !27 Service Mesh Service Discovery Configuration Coordination (Locking)

    Central Servers + Distributed Clients
  28. Copyright © 2017 HashiCorp @anubhavm  Our Past Experience !28

    GOSSIP CONSENSUS Mature Libraries Proven Design Patterns
  29. Copyright © 2017 HashiCorp @anubhavm  Our Past Experience !29

    GOSSIP CONSENSUS Mature Libraries Proven Design Patterns ?
  30. Copyright © 2017 HashiCorp @anubhavm  Our Past Experience !30

    GOSSIP CONSENSUS
  31. None
  32. @anubhavm  !32 Inspired by Google Omega Optimistic Concurrency State

    Coordination Service & Batch workloads Pluggable Architecture
  33. Copyright © 2017 HashiCorp @anubhavm  Consul Cluster !33 CLIENT

    CLIENT CLIENT CLIENT CLIENT CLIENT SERVER SERVER SERVER REPLICATION REPLICATION RPC RPC LAN GOSSIP SERVER SERVER SERVER REPLICATION REPLICATION WAN GOSSIP
  34. Copyright © 2017 HashiCorp @anubhavm  Single Region Architecture !34

    SERVER SERVER SERVER CLIENT CLIENT CLIENT DC1 DC2 DC3 FOLLOWER LEADER FOLLOWER REPLICATION FORWARDING REPLICATION FORWARDING RPC RPC RPC
  35. Copyright © 2017 HashiCorp @anubhavm  Multi Region Architecture !35

    SERVER SERVER SERVER FOLLOWER LEADER FOLLOWER REPLICATION FORWARDING REPLICATION REGION B  GOSSIP REPLICATION REPLICATION FORWARDING REGION FORWARDING  REGION A SERVER FOLLOWER SERVER SERVER LEADER FOLLOWER
  36. Copyright © 2017 HashiCorp @anubhavm  !36 Region is Isolation

    Domain 1-N Datacenters Per Region Flexibility to do 1:1 (Consul) Scheduling Boundary
  37. Installing and Configuring

  38. Copyright © 2017 HashiCorp @anubhavm  Installing Nomad !38 Client

    Server
  39. Copyright © 2017 HashiCorp @anubhavm  Fingerprinting !39 Type Examples

    Operating System Kernel, OS, Version Hardware CPU, Memory, Disk Apps (Capabilities) Docker, Java, Consul Environment AWS, GCE
  40. Copyright © 2017 HashiCorp @anubhavm  !40 Constrain Placement and

    Bin Pack
  41. Copyright © 2017 HashiCorp @anubhavm  !41 “Task Requires Linux,

    Docker, and PCI-Compliant Hardware” expressed as constraints in job file
  42. Copyright © 2017 HashiCorp @anubhavm  !42 “Task needs 512MB

    RAM and 1 Core” expressed as resource in job file
  43. Terminal $ nomad agent -config=/path/to/config.hcl

  44. Terminal $ echo "let’s try it?"

  45. Job File

  46. @anubhavm  !X job "redis" { datacenters = ["us-east-1"] task

    "redis" { driver = "docker" config { image = "redis:latest" } resources { cpu = 500 # Mhz memory = 256 # MB network { mbits = 10 port "redis" {} } } } }
  47. @anubhavm  !X job "webserver" { datacenters = ["us-east-1"] task

    "webserver" { driver = "exec" config { command = "yet-another-golang-webserver" } artifact { source = "https://github.com/download/../yet-another-golang-webserver" } resources { cpu = 500 # Mhz memory = 128 # MB network { port "http" { static = 8080 } } } } }
  48. Terminal $ echo "let’s try it?"

  49. Copyright © 2017 HashiCorp @anubhavm  !49 Job specification declares

    what to run
  50. Copyright © 2017 HashiCorp @anubhavm  !50 Nomad determines how

    and where to run
  51. Copyright © 2017 HashiCorp @anubhavm  !51 Nomad abstracts work

    from resources
  52. Job Types

  53. Copyright © 2017 HashiCorp @anubhavm  Nomad has three scheduler

    types that can be used when creating your job: service, batch, and system. Job Types !53
  54. Copyright © 2017 HashiCorp @anubhavm  Service Scheduler Job Type

    The service scheduler is designed for scheduling long-lived services that should never go down. The service scheduler ranks a large portion of the nodes that meet the jobs constraints and selects the optimal node to place a task group on. Examples: webapp, redis Job Types: Service !54
  55. Copyright © 2017 HashiCorp @anubhavm  Batch Scheduler Job Type

    Batch jobs are less sensitive to short-term performance fluctuations and are short lived, finishing after some period. Examples: billing, data replication Job Types: Batch !55
  56. Copyright © 2017 HashiCorp @anubhavm  System Scheduler Job Type

    The system scheduler is used to register jobs that should be run on all clients that meet the job's constraints. The system scheduler is also invoked when clients join the cluster or transition into the ready state. Examples: logging agent, security auditing tool Job Types: System !56
  57. Drivers

  58. @anubhavm  !58 Containerized Virtualized Standalone Docker Qemu / KVM

    Java Jar Static Binaries rkt LXC
  59. @anubhavm  !59 Containerized Virtualized Standalone Docker Qemu / KVM

    Java Jar Static Binaries rkt LXC Windows Server Containers Hyper-V Xen C#
  60. Service Discovery

  61. Copyright © 2017 HashiCorp @anubhavm  Consul is a free

    and open-source tool by HashiCorp that implements service discovery. It uses the RAFT and gossip protocols to reach massive scale. It has integrations with health checks, so unhealthy services are not added to the service discovery layer. Similar client-server model to Nomad. Service Discovery using Consul !61
  62. @anubhavm  !X job "redis" { datacenters = [“us-east-1”] service

    { name = "redis" port = "redis" check { type = "tcp" port = "redis" interval = "10s" timeout = "5s" } } }
  63. Terminal $ echo "let’s try it?"

  64. Advanced Job Strategies

  65. Copyright © 2017 HashiCorp @anubhavm  Nomad currently supports two

    advanced job strategies: Rolling Upgrades, Blue/Green & Canary Deployments Advanced Job Strategies !65
  66. @anubhavm  !X job "redis" { # Add an update

    stanza to enable rolling updates of the service update { max_parallel = 1 min_healthy_time = "30s" healthy_deadline = "10m" } ..... }
  67. @anubhavm  !X job "redis" { # Add an update

    stanza to enable rolling updates of the service update { max_parallel = 1 canary = 5 min_healthy_time = "30s" auto_revert = true } ..... }
  68. Terminal $ echo "let’s try it?"

  69. @anubhavm  !69 Multi-Datacenter Multi-Region Flexible Workloads Job Priorities Bin

    Packing Large Scale Operationally Simple
  70. @anubhavm  !70 Multi-Datacenter Multi-Region Flexible Workloads Job Priorities Bin

    Packing Large Scale Operationally Simple
  71. s Copyright © 2017 HashiCorp @anubhavm  !71 ONE MORE

    DEMO
  72. Nomad Million Container Challenge 1,000 Jobs 1,000 Tasks per Job

    5,000 Hosts on GCE 1,000,000 Containers
  73. Copyright © 2017 HashiCorp @anubhavm  !73

  74. None
  75. Copyright © 2017 HashiCorp @anubhavm  !75 “640 KB ought

    to be enough for anybody.” - Bill Gates
  76. Copyright © 2017 HashiCorp @anubhavm  !76 2nd Largest Hedge

    Fund 18K Cores 5 Hours 2,200 Containers/second
  77. Copyright © 2017 HashiCorp @anubhavm  !77 CircleCI 7+ Million

    Builds a Month Sustain 400-1000 Jobs a Minute Great Talk By Danielle Tomlinson: https://youtu.be/b8NQO_vFAYo
  78. Copyright © 2017 HashiCorp @anubhavm  !78 Globally Distributed Optimistically

    Concurrent Scheduler
  79. Copyright © 2017 HashiCorp @anubhavm  !79 Higher Resource Utilization

    Decouple Work from Resources Better Quality of Service
  80. www.hashicorp.com FOR EVERYONE, EVERYWHERE Thank You! I have stickers! Ask

    me anything. @anubhavm Anubhav Mishra