Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Nomad Overview and 0.6 New Features

Nomad Overview and 0.6 New Features

We provide an overview of HashiCorp Nomad and the new features coming in 0.6, which is a cluster manager and scheduler. We explore what a cluster scheduler is and the benefits they provide, including higher resource utilization, decoupling developers and operators, and better quality of service. The high level design of Nomad is explained and how it supports running over a million containers.

The new 0.6 release adds support for first class deployments. This includes rolling deploys, canaries, promotions, blue/green, and auto-revert. Dynamic config values from Consul and Vault can be used as environment variables, and much more.

The talk and demo are also recorded as a webinar with Caius from Citadel discussing their use case.

Armon Dadgar

July 12, 2017
Tweet

More Decks by Armon Dadgar

Other Decks in Technology

Transcript

  1. Nomad Overview +
    0.6 New Features

    View Slide

  2. Armon Dadgar
    Founder and CTO
    @armon

    View Slide

  3. Copyright © 2017 HashiCorp
    CONNECT
    Infrastructure & applications
    RUN
    SECURE
    PROVISION
    Applications
    Infrastructure & applications
    Infrastructure
    OSS TOOL SUITE PRODUCT SUITE
    Consul
    Nomad
    Terraform
    Vault
    Consul
    Enterprise
    Nomad
    Enterprise
    Vault Enterprise
    Terraform
    Enterprise
    FOR TEAMS
    FOR INDIVIDUALS
    Provision, secure, connect, and run any infrastructure
    3
    Copyright © 2017 HashiCorp

    View Slide

  4. Copyright © 2017 HashiCorp
    ▪Nomad Overview
    ▪New Features of 0.6
    ▪Demo!
    4
    Agenda

    View Slide

  5. Nomad
    Cluster Manager
    Scheduler

    View Slide

  6. Nomad
    Cluster Manager
    Scheduler

    View Slide

  7. Schedulers map a set of
    work to a set of resources

    View Slide

  8. CPU Scheduler
    Web Server -Thread 1
    CPU - Core 1
    CPU - Core 2
    Web Server -Thread 2
    Redis -Thread 1
    Kernel -Thread 1
    Work (Input) Resources
    CPU
    Scheduler

    View Slide

  9. CPU Scheduler
    Web Server -Thread 1
    CPU - Core 1
    CPU - Core 2
    Web Server -Thread 2
    Redis -Thread 1
    Kernel -Thread 1
    Work (Input) Resources
    CPU
    Scheduler

    View Slide

  10. Schedulers in the Wild
    Type Work Resources
    CPU Scheduler Threads Physical Cores
    AWS EC2 /
    OpenStack Nova
    Virtual Machines Hypervisors
    Hadoop YARN MapReduce Jobs Client Nodes
    Cluster Scheduler Applications Servers

    View Slide

  11. User Nomad
    Servers
    Nomad
    Clients

    View Slide

  12. User Nomad
    Servers
    Submits
    Job
    Nomad
    Clients

    View Slide

  13. User Nomad
    Servers
    Submits
    Job
    Nomad
    Clients
    Deploy App

    View Slide

  14. User Nomad
    Servers
    Submits
    Job
    Nomad
    Clients
    Deploy App
    Skip (Busy)

    View Slide

  15. User Nomad
    Servers
    Submits
    Job
    Nomad
    Clients
    Deploy App
    Skip (Busy)
    Deploy App

    View Slide

  16. Advantages
    Higher Resource U.liza.on
    Decouple Work from Resources
    Be:er Quality of Service

    View Slide

  17. Advantages
    Higher Resource U.liza.on
    Decouple Work from Resources
    Be:er Quality of Service
    Bin Packing
    Over-Subscrip.on
    Job Queueing

    View Slide

  18. Advantages
    Higher Resource U.liza.on
    Decouple Work from Resources
    Be:er Quality of Service
    Abstrac.on
    API Contracts
    Packaging

    View Slide

  19. Advantages
    Higher Resource U.liza.on
    Decouple Work from Resources
    Be:er Quality of Service
    Priori.es
    Resource Isola.on
    Pre-emp.on

    View Slide

  20. View Slide


  21. Nomad
    @armon

    View Slide

  22. Nomad
    Cluster Scheduler
    Easily Deploy Applications
    Operationally Simple
    Built for Scale

    View Slide

  23. job "redis" {
    datacenters = ["us-east-1"]
    task "redis" {
    driver = "docker"
    config { image = "redis:latest" }
    resources {
    cpu = 500 # Mhz
    memory = 256 # MB
    network {
    mbits = 10
    port “redis" {}
    }
    }
    }
    }
    example.nomad

    View Slide

  24. Declares what to run

    View Slide

  25. Nomad determines where
    and manages how to run

    View Slide

  26. Nomad abstracts
    work from resources

    View Slide

  27. OS Workloads Drivers
    Windows Long Running Service Docker / Rkt / LXC
    Linux Short Lived Batch Qemu / KVM
    BSD Periodic Cron
    “exec”
    cgroups+chroot
    Solaris System Agents
    Static Binaries /
    Fat JARs

    View Slide

  28. Nomad
    Declarative Jobs
    Infrastructure as Code
    Consul Integration
    Vault Integration
    Composable vs Platform

    View Slide

  29. Empowers developers by
    de-coupling operators

    View Slide

  30. Operationally Simple
    &
    Scalable

    View Slide

  31. Nomad
    Single Binary
    No Dependencies
    Highly Available
    Multi-DC/Region Support

    View Slide

  32. Built on Research & Experience
    GOSSIP CONSENSUS
    Mature Libraries Proven Design Patterns

    View Slide

  33. Nomad
    Inspired by Google Omega
    Optimistic Concurrency
    Service & Batch workloads
    Pluggable Architecture

    View Slide

  34. Single Region Architecture
    SERVER SERVER SERVER
    CLIENT CLIENT CLIENT
    DC1 DC2 DC3
    FOLLOWER LEADER FOLLOWER
    REPLICATION
    FORWARDING
    REPLICATION
    FORWARDING
    RPC RPC RPC

    View Slide

  35. Multi Region Architecture
    SERVER SERVER SERVER
    FOLLOWER LEADER FOLLOWER
    REPLICATION
    FORWARDING
    REPLICATION
    REGION B
     GOSSIP
    REPLICATION REPLICATION
    FORWARDING
    REGION FORWARDING
     REGION A
    SERVER
    FOLLOWER
    SERVER SERVER
    LEADER FOLLOWER

    View Slide

  36. 100’s of Regions
    10,000’s of Clients per Region
    1000’s of Jobs per Region

    View Slide

  37. Nomad
    Million Container
    Challenge
    1,000 Jobs
    1,000 Tasks per Job
    5,000 Hosts on GCE
    1,000,000 Containers

    View Slide

  38. View Slide

  39. “640 KB ought to be enough for anybody.”
    - Bill Gates

    View Slide

  40. 2nd Largest Hedge Fund
    18K Cores
    5 Hours
    2,200 Containers/second

    View Slide

  41. Nomad
    Cluster Scheduler
    Easily Deploy Applications
    Operationally Simple
    Built for Scale

    View Slide

  42. Copyright © 2017 HashiCorp
    ▪Service Updates
    ▪Rolling Deploys with Health Checks
    ▪Canaries
    ▪Blue/Green
    ▪Dynamic Environment Variables
    ▪Multiple Job Versions & Rollbacks
    ▪Docker Networking Plugins
    ▪Much More (see CHANGELOG)
    42
    New Features in 0.6

    View Slide

  43. Service Updates
    • Changing a Service Job from Version A to Version B
    • Upgrading WebApp from 1.1.0 to 1.2.0
    • Controlled by update block
    • Supports many different approaches
    • Rolling Deploys, Canaries, Blue/Green

    View Slide

  44. Copyright © 2017 HashiCorp 44
    Rolling Deploys
    job "webapp" {
    update {
    max_parallel = 2
    }
    group "web" {
    count = 6
    task "app" {
    driver = "docker"
    config {
    image = "webapp:1.0"
    }
    }
    }
    }
    webapp.nomad

    View Slide

  45. webapp 1.0

    View Slide

  46. webapp 1.0
    1.0 1.0 1.0 1.0 1.0
    1.0

    View Slide

  47. 1.0 1.0 1.0 1.0
    webapp 2.0
    2.0 2.0

    View Slide

  48. 1.0 1.0
    webapp 2.0
    2.0 2.0 2.0 2.0

    View Slide

  49. webapp 2.0
    2.0 2.0 2.0 2.0 2.0 2.0

    View Slide

  50. Copyright © 2017 HashiCorp 50
    Canaries
    job "webapp" {
    update {
    max_parallel = 2
    canary = 1
    }
    group "web" {
    count = 6
    task "app" {
    driver = "docker"
    config {
    image = "webapp:1.0"
    }
    }
    }
    }
    webapp.nomad

    View Slide

  51. webapp 1.0
    1.0 1.0 1.0 1.0 1.0
    1.0

    View Slide

  52. 1.0 1.0 1.0 1.0
    webapp 2.0
    2.0
    1.0 1.0
    Canary

    View Slide

  53. 1.0 1.0 1.0 1.0
    Promote
    webapp 2.0 2.0
    1.0

    View Slide

  54. 1.0 1.0 1.0
    Promote
    webapp 2.0 2.0 2.0 2.0

    View Slide

  55. 1.0
    Promote
    webapp 2.0 2.0 2.0 2.0 2.0 2.0

    View Slide

  56. Promote
    webapp 2.0 2.0 2.0 2.0 2.0 2.0 2.0

    View Slide

  57. Copyright © 2017 HashiCorp 57
    Blue/Green
    job "webapp" {
    update {
    max_parallel = 2
    canary = 6 # Match count of group
    }
    group "web" {
    count = 6
    task "app" {
    driver = "docker"
    config {
    image = "webapp:1.0"
    }
    }
    }
    }
    webapp.nomad

    View Slide

  58. webapp 1.0
    1.0 1.0 1.0 1.0 1.0
    1.0

    View Slide

  59. 1.0 1.0 1.0 1.0
    webapp 2.0
    2.0
    1.0 1.0
    Canary
    2.0 2.0 2.0 2.0 2.0
    Canary Canary Canary Canary Canary

    View Slide

  60. 2.0 2.0 2.0 2.0 2.0 2.0
    Promote
    webapp 2.0

    View Slide

  61. Multiple Job Versions & Rollbacks
    • Track Multiple Versions per Job
    • Rollbacks
    • “Auto Revert” failed deploys

    View Slide

  62. Copyright © 2017 HashiCorp 62
    Job Versions
    $ nomad job history example
    Version = 1
    Stable = true
    Submit Date = 07/08/17 23:29:31 UTC
    Version = 0
    Stable = true
    Submit Date = 07/08/17 23:27:54 UTC
    $ nomad job revert example 0
    ==> Monitoring evaluation "4e0662c7"

    ==> Evaluation "4e0662c7" finished with status “complete"
    $ nomad job history -p example
    Version = 2
    Stable = true
    Submit Date = 07/08/17 23:31:07 UTC
    Diff =
    +/- Job: "example"
    +/- Task Group: "cache"
    +/- Task: "redis"
    +/- Config {
    +/- image: "redis:3.0" => "redis:3.2"
    port_map[0][db]: "6379"
    }
    Terminal

    View Slide

  63. Copyright © 2017 HashiCorp 63
    Job Status with
    Versions
    $ nomad status example
    ID = example
    Name = example
    Submit Date = 07/08/17 23:31:07 UTC
    Type = service
    Priority = 50
    Datacenters = dc1
    Status = running
    Periodic = false
    Parameterized = false
    Summary
    Task Group Queued Starting Running Failed Complete Lost
    cache 0 0 1 0 2 0
    Latest Deployment
    ID = 07e88357
    Status = successful
    Description = Deployment completed successfully
    Deployed
    Task Group Desired Placed Healthy Unhealthy
    cache 1 1 1 0
    Allocations
    ID Node ID Task Group Version Desired Status Created At
    71c6f915 49a104fe cache 2 run running 07/08/17 23:31:07 UTC
    9b69e806 49a104fe cache 1 stop complete 07/08/17 23:29:31 UTC
    2e5550f0 49a104fe cache 0 stop complete 07/08/17 23:27:54 UTC
    Terminal

    View Slide

  64. Dynamic Environment Variables
    • The template block renders dynamic configuration
    • Integrates with Consul for app configuration
    • Integrates with Vault for secrets (DB credentials, TLS, etc)
    • New env parameter sets environment variables

    View Slide

  65. Copyright © 2017 HashiCorp 65
    Dynamic
    Environment
    Variables
    job "webapp" {
    task "app" {
    driver = "docker"
    config {
    image = "webapp:1.0"
    }
    template {
    data = <# Feature flag from Consul
    FEATURE_FLAG_FOO = {{ key "service/webapp/foo" }}
    # AWS credentials from Vault
    {{ with secret "service/webapp/aws_key" }}
    AWS_ACCESS_KEY = {{ .Data.access_key }}
    AWS_SECRET_KEY = {{ .Data.secret_key }}
    {{ end }}
    EOH
    destination = "local/file.env"
    env = true
    }
    webapp.nomad

    View Slide

  66. Demo!

    View Slide


  67. Thanks!
    @armon

    View Slide