Nomad Overview and 0.6 New Features

We provide an overview of HashiCorp Nomad and the new features coming in 0.6, which is a cluster manager and scheduler. We explore what a cluster scheduler is and the benefits they provide, including higher resource utilization, decoupling developers and operators, and better quality of service. The high level design of Nomad is explained and how it supports running over a million containers.

The new 0.6 release adds support for first class deployments. This includes rolling deploys, canaries, promotions, blue/green, and auto-revert. Dynamic config values from Consul and Vault can be used as environment variables, and much more.

The talk and demo are also recorded as a webinar with Caius from Citadel discussing their use case.

Transcript

1. Nomad Overview +
   0.6 New Features
   

   View Slide

2. Armon Dadgar
   Founder and CTO
   @armon
   

   View Slide

3. Copyright © 2017 HashiCorp
   CONNECT
   Infrastructure & applications
   RUN
   SECURE
   PROVISION
   Applications
   Infrastructure & applications
   Infrastructure
   OSS TOOL SUITE PRODUCT SUITE
   Consul
   Nomad
   Terraform
   Vault
   Consul
   Enterprise
   Nomad
   Enterprise
   Vault Enterprise
   Terraform
   Enterprise
   FOR TEAMS
   FOR INDIVIDUALS
   Provision, secure, connect, and run any infrastructure
   3
   Copyright © 2017 HashiCorp
   

   View Slide

4. Copyright © 2017 HashiCorp
   ▪Nomad Overview
   ▪New Features of 0.6
   ▪Demo!
   4
   Agenda
   

   View Slide

5. Nomad
   Cluster Manager
   Scheduler
   

   View Slide

6. Nomad
   Cluster Manager
   Scheduler
   

   View Slide

7. Schedulers map a set of
   work to a set of resources
   

   View Slide

8. CPU Scheduler
   Web Server -Thread 1
   CPU - Core 1
   CPU - Core 2
   Web Server -Thread 2
   Redis -Thread 1
   Kernel -Thread 1
   Work (Input) Resources
   CPU
   Scheduler
   

   View Slide

9. CPU Scheduler
   Web Server -Thread 1
   CPU - Core 1
   CPU - Core 2
   Web Server -Thread 2
   Redis -Thread 1
   Kernel -Thread 1
   Work (Input) Resources
   CPU
   Scheduler
   

   View Slide

10. Schedulers in the Wild
    Type Work Resources
    CPU Scheduler Threads Physical Cores
    AWS EC2 /
    OpenStack Nova
    Virtual Machines Hypervisors
    Hadoop YARN MapReduce Jobs Client Nodes
    Cluster Scheduler Applications Servers
    

    View Slide

11. User Nomad
    Servers
    Nomad
    Clients
    

    View Slide

12. User Nomad
    Servers
    Submits
    Job
    Nomad
    Clients
    

    View Slide

13. User Nomad
    Servers
    Submits
    Job
    Nomad
    Clients
    Deploy App
    

    View Slide

14. User Nomad
    Servers
    Submits
    Job
    Nomad
    Clients
    Deploy App
    Skip (Busy)
    

    View Slide

15. User Nomad
    Servers
    Submits
    Job
    Nomad
    Clients
    Deploy App
    Skip (Busy)
    Deploy App
    

    View Slide

16. Advantages
    Higher Resource U.liza.on
    Decouple Work from Resources
    Be:er Quality of Service
    

    View Slide

17. Advantages
    Higher Resource U.liza.on
    Decouple Work from Resources
    Be:er Quality of Service
    Bin Packing
    Over-Subscrip.on
    Job Queueing
    

    View Slide

18. Advantages
    Higher Resource U.liza.on
    Decouple Work from Resources
    Be:er Quality of Service
    Abstrac.on
    API Contracts
    Packaging
    

    View Slide

19. Advantages
    Higher Resource U.liza.on
    Decouple Work from Resources
    Be:er Quality of Service
    Priori.es
    Resource Isola.on
    Pre-emp.on
    

    View Slide

20. View Slide

21. 
    Nomad
    @armon
    

    View Slide

22. Nomad
    Cluster Scheduler
    Easily Deploy Applications
    Operationally Simple
    Built for Scale
    

    View Slide

23. job "redis" {
    datacenters = ["us-east-1"]
    task "redis" {
    driver = "docker"
    config { image = "redis:latest" }
    resources {
    cpu = 500 # Mhz
    memory = 256 # MB
    network {
    mbits = 10
    port “redis" {}
    }
    }
    }
    }
    example.nomad
    

    View Slide

24. Declares what to run
    

    View Slide

25. Nomad determines where
    and manages how to run
    

    View Slide

26. Nomad abstracts
    work from resources
    

    View Slide

27. OS Workloads Drivers
    Windows Long Running Service Docker / Rkt / LXC
    Linux Short Lived Batch Qemu / KVM
    BSD Periodic Cron
    “exec”
    cgroups+chroot
    Solaris System Agents
    Static Binaries /
    Fat JARs
    

    View Slide

28. Nomad
    Declarative Jobs
    Infrastructure as Code
    Consul Integration
    Vault Integration
    Composable vs Platform
    

    View Slide

29. Empowers developers by
    de-coupling operators
    

    View Slide

30. Operationally Simple
    &
    Scalable
    

    View Slide

31. Nomad
    Single Binary
    No Dependencies
    Highly Available
    Multi-DC/Region Support
    

    View Slide

32. Built on Research & Experience
    GOSSIP CONSENSUS
    Mature Libraries Proven Design Patterns
    

    View Slide

33. Nomad
    Inspired by Google Omega
    Optimistic Concurrency
    Service & Batch workloads
    Pluggable Architecture
    

    View Slide

34. Single Region Architecture
    SERVER SERVER SERVER
    CLIENT CLIENT CLIENT
    DC1 DC2 DC3
    FOLLOWER LEADER FOLLOWER
    REPLICATION
    FORWARDING
    REPLICATION
    FORWARDING
    RPC RPC RPC
    

    View Slide

35. Multi Region Architecture
    SERVER SERVER SERVER
    FOLLOWER LEADER FOLLOWER
    REPLICATION
    FORWARDING
    REPLICATION
    REGION B
     GOSSIP
    REPLICATION REPLICATION
    FORWARDING
    REGION FORWARDING
     REGION A
    SERVER
    FOLLOWER
    SERVER SERVER
    LEADER FOLLOWER
    

    View Slide

36. 100’s of Regions
    10,000’s of Clients per Region
    1000’s of Jobs per Region
    

    View Slide

37. Nomad
    Million Container
    Challenge
    1,000 Jobs
    1,000 Tasks per Job
    5,000 Hosts on GCE
    1,000,000 Containers
    

    View Slide

38. View Slide

39. “640 KB ought to be enough for anybody.”
    - Bill Gates
    

    View Slide

40. 2nd Largest Hedge Fund
    18K Cores
    5 Hours
    2,200 Containers/second
    

    View Slide

41. Nomad
    Cluster Scheduler
    Easily Deploy Applications
    Operationally Simple
    Built for Scale
    

    View Slide

42. Copyright © 2017 HashiCorp
    ▪Service Updates
    ▪Rolling Deploys with Health Checks
    ▪Canaries
    ▪Blue/Green
    ▪Dynamic Environment Variables
    ▪Multiple Job Versions & Rollbacks
    ▪Docker Networking Plugins
    ▪Much More (see CHANGELOG)
    42
    New Features in 0.6
    

    View Slide

43. Service Updates
    • Changing a Service Job from Version A to Version B
    • Upgrading WebApp from 1.1.0 to 1.2.0
    • Controlled by update block
    • Supports many different approaches
    • Rolling Deploys, Canaries, Blue/Green
    

    View Slide

44. Copyright © 2017 HashiCorp 44
    Rolling Deploys
    job "webapp" {
    update {
    max_parallel = 2
    }
    group "web" {
    count = 6
    task "app" {
    driver = "docker"
    config {
    image = "webapp:1.0"
    }
    }
    }
    }
    webapp.nomad
    

    View Slide

45. webapp 1.0
    

    View Slide

46. webapp 1.0
    1.0 1.0 1.0 1.0 1.0
    1.0
    

    View Slide

47. 1.0 1.0 1.0 1.0
    webapp 2.0
    2.0 2.0
    

    View Slide

48. 1.0 1.0
    webapp 2.0
    2.0 2.0 2.0 2.0
    

    View Slide

49. webapp 2.0
    2.0 2.0 2.0 2.0 2.0 2.0
    

    View Slide

50. Copyright © 2017 HashiCorp 50
    Canaries
    job "webapp" {
    update {
    max_parallel = 2
    canary = 1
    }
    group "web" {
    count = 6
    task "app" {
    driver = "docker"
    config {
    image = "webapp:1.0"
    }
    }
    }
    }
    webapp.nomad
    

    View Slide

51. webapp 1.0
    1.0 1.0 1.0 1.0 1.0
    1.0
    

    View Slide

52. 1.0 1.0 1.0 1.0
    webapp 2.0
    2.0
    1.0 1.0
    Canary
    

    View Slide

53. 1.0 1.0 1.0 1.0
    Promote
    webapp 2.0 2.0
    1.0
    

    View Slide

54. 1.0 1.0 1.0
    Promote
    webapp 2.0 2.0 2.0 2.0
    

    View Slide

55. 1.0
    Promote
    webapp 2.0 2.0 2.0 2.0 2.0 2.0
    

    View Slide

56. Promote
    webapp 2.0 2.0 2.0 2.0 2.0 2.0 2.0
    

    View Slide

57. Copyright © 2017 HashiCorp 57
    Blue/Green
    job "webapp" {
    update {
    max_parallel = 2
    canary = 6 # Match count of group
    }
    group "web" {
    count = 6
    task "app" {
    driver = "docker"
    config {
    image = "webapp:1.0"
    }
    }
    }
    }
    webapp.nomad
    

    View Slide

58. webapp 1.0
    1.0 1.0 1.0 1.0 1.0
    1.0
    

    View Slide

59. 1.0 1.0 1.0 1.0
    webapp 2.0
    2.0
    1.0 1.0
    Canary
    2.0 2.0 2.0 2.0 2.0
    Canary Canary Canary Canary Canary
    

    View Slide

60. 2.0 2.0 2.0 2.0 2.0 2.0
    Promote
    webapp 2.0
    

    View Slide

61. Multiple Job Versions & Rollbacks
    • Track Multiple Versions per Job
    • Rollbacks
    • “Auto Revert” failed deploys
    

    View Slide

62. Copyright © 2017 HashiCorp 62
    Job Versions
    $ nomad job history example
    Version = 1
    Stable = true
    Submit Date = 07/08/17 23:29:31 UTC
    Version = 0
    Stable = true
    Submit Date = 07/08/17 23:27:54 UTC
    $ nomad job revert example 0
    ==> Monitoring evaluation "4e0662c7"
    …
    ==> Evaluation "4e0662c7" finished with status “complete"
    $ nomad job history -p example
    Version = 2
    Stable = true
    Submit Date = 07/08/17 23:31:07 UTC
    Diff =
    +/- Job: "example"
    +/- Task Group: "cache"
    +/- Task: "redis"
    +/- Config {
    +/- image: "redis:3.0" => "redis:3.2"
    port_map[0][db]: "6379"
    }
    Terminal
    

    View Slide

63. Copyright © 2017 HashiCorp 63
    Job Status with
    Versions
    $ nomad status example
    ID = example
    Name = example
    Submit Date = 07/08/17 23:31:07 UTC
    Type = service
    Priority = 50
    Datacenters = dc1
    Status = running
    Periodic = false
    Parameterized = false
    Summary
    Task Group Queued Starting Running Failed Complete Lost
    cache 0 0 1 0 2 0
    Latest Deployment
    ID = 07e88357
    Status = successful
    Description = Deployment completed successfully
    Deployed
    Task Group Desired Placed Healthy Unhealthy
    cache 1 1 1 0
    Allocations
    ID Node ID Task Group Version Desired Status Created At
    71c6f915 49a104fe cache 2 run running 07/08/17 23:31:07 UTC
    9b69e806 49a104fe cache 1 stop complete 07/08/17 23:29:31 UTC
    2e5550f0 49a104fe cache 0 stop complete 07/08/17 23:27:54 UTC
    Terminal
    

    View Slide

64. Dynamic Environment Variables
    • The template block renders dynamic configuration
    • Integrates with Consul for app configuration
    • Integrates with Vault for secrets (DB credentials, TLS, etc)
    • New env parameter sets environment variables
    

    View Slide

65. Copyright © 2017 HashiCorp 65
    Dynamic
    Environment
    Variables
    job "webapp" {
    task "app" {
    driver = "docker"
    config {
    image = "webapp:1.0"
    }
    template {
    data = <# Feature flag from Consul
    FEATURE_FLAG_FOO = {{ key "service/webapp/foo" }}
    # AWS credentials from Vault
    {{ with secret "service/webapp/aws_key" }}
    AWS_ACCESS_KEY = {{ .Data.access_key }}
    AWS_SECRET_KEY = {{ .Data.secret_key }}
    {{ end }}
    EOH
    destination = "local/file.env"
    env = true
    }
    webapp.nomad
    

    View Slide

66. Demo!
    

    View Slide

67. 
    Thanks!
    @armon
    

    View Slide