Scheduling Applications at Scale with Nomad

Session details on Linux Foundation OSS Summit Website: https://ossna18.sched.com/event/FAP0/scheduling-applications-at-scale-with-nomad-anubhav-mishra-hashicorp

Scheduler frameworks enable reliable and repeatable application deploys. In this session, attendees will use Nomad, a single binary cluster scheduler, to build a multi-region, self-healing production environment that runs a diverse set of workloads. They will also get hands on experience in writing and submitting job specifications, interacting with the API, and deployment strategies. This session will cover the following:
- Nomad Overview
- Installing and Configuring Nomad
- Creating, Running, and Inspecting Jobs
- Service Registration
- Interacting via the HTTP API
- Advanced Job Strategies (rolling updates, blue-green)

Transcript

1. Scheduling Applications at Scale with Nomad Photo by Sašo Tušar on Unsplash @anubhavm 
2. @anubhavm Anubhav Mishra Developer Advocate, HashiCorp
3. @anubhavm Anubhav Mishra Developer Advocate, HashiCorp has stickers Atlan&s Gopher Artwork by Ashley McNamara
4. PROVISION, SECURE AND RUN ANY INFRASTRUCTURE Nomad Consul Vault Vagrant Packer Terraform Consul Enterprise Terraform Enterprise Vault Enterprise PRODUCT SUITE OSS TOOL SUITE RUN Applications SECURE Application Infrastructure PROVISION Infrastructure FOR INDIVIDUALS FOR TEAMS Nomad Enterprise
6. Copyright © 2017 HashiCorp @anubhavm  !6 Globally Distributed Optimistically Concurrent Scheduler
7. Copyright © 2017 HashiCorp @anubhavm  !7 Globally Distributed Optimistically Concurrent Scheduler
8. Scheduler Photo by Emma Matthews on Unsplash
9. Copyright © 2017 HashiCorp @anubhavm  !9 A person or machine that helps scheduling during each day of work. Scheduler
10. Copyright © 2017 HashiCorp @anubhavm  !10
11. Copyright © 2017 HashiCorp @anubhavm  !11 A computer program that controls or manages the execution of jobs / processes / operations. Scheduler (Computing)
12. Copyright © 2017 HashiCorp @anubhavm  !12
13. Copyright © 2017 HashiCorp @anubhavm  !13 Schedulers map a set of work to a set of resources
14. Copyright © 2017 HashiCorp @anubhavm  CPU Scheduler !14 CORE CORE CORE CORE CPU SCHEDULER KERNEL APACHE REDIS BASH
15. Copyright © 2017 HashiCorp @anubhavm  CPU Scheduler !15 CORE CORE CPU SCHEDULER KERNEL APACHE REDIS BASH
16. Copyright © 2017 HashiCorp @anubhavm  CPU Scheduler !16 CORE CORE CPU SCHEDULER KERNEL APACHE REDIS BASH
17. Copyright © 2017 HashiCorp @anubhavm  CPU Scheduler !17 CORE CORE CPU SCHEDULER KERNEL APACHE REDIS BASH
18. Copyright © 2017 HashiCorp @anubhavm  Scheduler Advantages !18 Higher Resource Utilization Decouple Work from Resources Better Quality of Service
19. Copyright © 2017 HashiCorp @anubhavm  Nop! Schedulers Aren’t New Concept !19
20. Copyright © 2017 HashiCorp @anubhavm  Landscape !20
22. Designing
23. @anubhavm  !23 Multi-Datacenter Multi-Region Flexible Workloads Job Priorities Bin Packing Large Scale Operationally Simple
24. Copyright © 2017 HashiCorp @anubhavm  Thousands of regions Tens of thousands of clients per region Thousands of jobs per region Scaling Requirements !24
25. Copyright © 2017 HashiCorp @anubhavm  Our Past Experience !25 GOSSIP CONSENSUS
26. @anubhavm  !26 Cluster Management Gossip Based (P2P) Membership Failure Detection Event System Serf
27. @anubhavm  !27 Service Mesh Service Discovery Configuration Coordination (Locking) Central Servers + Distributed Clients
28. Copyright © 2017 HashiCorp @anubhavm  Our Past Experience !28 GOSSIP CONSENSUS Mature Libraries Proven Design Patterns
29. Copyright © 2017 HashiCorp @anubhavm  Our Past Experience !29 GOSSIP CONSENSUS Mature Libraries Proven Design Patterns ?
30. Copyright © 2017 HashiCorp @anubhavm  Our Past Experience !30 GOSSIP CONSENSUS
32. @anubhavm  !32 Inspired by Google Omega Optimistic Concurrency State Coordination Service & Batch workloads Pluggable Architecture
33. Copyright © 2017 HashiCorp @anubhavm  Consul Cluster !33 CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT SERVER SERVER SERVER REPLICATION REPLICATION RPC RPC LAN GOSSIP SERVER SERVER SERVER REPLICATION REPLICATION WAN GOSSIP
34. Copyright © 2017 HashiCorp @anubhavm  Single Region Architecture !34 SERVER SERVER SERVER CLIENT CLIENT CLIENT DC1 DC2 DC3 FOLLOWER LEADER FOLLOWER REPLICATION FORWARDING REPLICATION FORWARDING RPC RPC RPC
35. Copyright © 2017 HashiCorp @anubhavm  Multi Region Architecture !35 SERVER SERVER SERVER FOLLOWER LEADER FOLLOWER REPLICATION FORWARDING REPLICATION REGION B  GOSSIP REPLICATION REPLICATION FORWARDING REGION FORWARDING  REGION A SERVER FOLLOWER SERVER SERVER LEADER FOLLOWER
36. Copyright © 2017 HashiCorp @anubhavm  !36 Region is Isolation Domain 1-N Datacenters Per Region Flexibility to do 1:1 (Consul) Scheduling Boundary
37. Installing and Configuring
38. Copyright © 2017 HashiCorp @anubhavm  Installing Nomad !38 Client Server
39. Copyright © 2017 HashiCorp @anubhavm  Fingerprinting !39 Type Examples Operating System Kernel, OS, Version Hardware CPU, Memory, Disk Apps (Capabilities) Docker, Java, Consul Environment AWS, GCE
40. Copyright © 2017 HashiCorp @anubhavm  !40 Constrain Placement and Bin Pack
41. Copyright © 2017 HashiCorp @anubhavm  !41 “Task Requires Linux, Docker, and PCI-Compliant Hardware” expressed as constraints in job file
42. Copyright © 2017 HashiCorp @anubhavm  !42 “Task needs 512MB RAM and 1 Core” expressed as resource in job file
43. Terminal $ nomad agent -config=/path/to/config.hcl
44. Terminal $ echo "let’s try it?"
45. Job File
46. @anubhavm  !X job "redis" { datacenters = ["us-east-1"] task "redis" { driver = "docker" config { image = "redis:latest" } resources { cpu = 500 # Mhz memory = 256 # MB network { mbits = 10 port "redis" {} } } } }
47. @anubhavm  !X job "webserver" { datacenters = ["us-east-1"] task "webserver" { driver = "exec" config { command = "yet-another-golang-webserver" } artifact { source = "https://github.com/download/../yet-another-golang-webserver" } resources { cpu = 500 # Mhz memory = 128 # MB network { port "http" { static = 8080 } } } } }
48. Terminal $ echo "let’s try it?"
49. Copyright © 2017 HashiCorp @anubhavm  !49 Job specification declares what to run
50. Copyright © 2017 HashiCorp @anubhavm  !50 Nomad determines how and where to run
51. Copyright © 2017 HashiCorp @anubhavm  !51 Nomad abstracts work from resources
52. Job Types
53. Copyright © 2017 HashiCorp @anubhavm  Nomad has three scheduler types that can be used when creating your job: service, batch, and system. Job Types !53
54. Copyright © 2017 HashiCorp @anubhavm  Service Scheduler Job Type The service scheduler is designed for scheduling long-lived services that should never go down. The service scheduler ranks a large portion of the nodes that meet the jobs constraints and selects the optimal node to place a task group on. Examples: webapp, redis Job Types: Service !54
55. Copyright © 2017 HashiCorp @anubhavm  Batch Scheduler Job Type Batch jobs are less sensitive to short-term performance fluctuations and are short lived, finishing after some period. Examples: billing, data replication Job Types: Batch !55
56. Copyright © 2017 HashiCorp @anubhavm  System Scheduler Job Type The system scheduler is used to register jobs that should be run on all clients that meet the job's constraints. The system scheduler is also invoked when clients join the cluster or transition into the ready state. Examples: logging agent, security auditing tool Job Types: System !56
57. Drivers
58. @anubhavm  !58 Containerized Virtualized Standalone Docker Qemu / KVM Java Jar Static Binaries rkt LXC
59. @anubhavm  !59 Containerized Virtualized Standalone Docker Qemu / KVM Java Jar Static Binaries rkt LXC Windows Server Containers Hyper-V Xen C#
60. Service Discovery
61. Copyright © 2017 HashiCorp @anubhavm  Consul is a free and open-source tool by HashiCorp that implements service discovery. It uses the RAFT and gossip protocols to reach massive scale. It has integrations with health checks, so unhealthy services are not added to the service discovery layer. Similar client-server model to Nomad. Service Discovery using Consul !61
62. @anubhavm  !X job "redis" { datacenters = [“us-east-1”] service { name = "redis" port = "redis" check { type = "tcp" port = "redis" interval = "10s" timeout = "5s" } } }
63. Terminal $ echo "let’s try it?"
64. Advanced Job Strategies
65. Copyright © 2017 HashiCorp @anubhavm  Nomad currently supports two advanced job strategies: Rolling Upgrades, Blue/Green & Canary Deployments Advanced Job Strategies !65
66. @anubhavm  !X job "redis" { # Add an update stanza to enable rolling updates of the service update { max_parallel = 1 min_healthy_time = "30s" healthy_deadline = "10m" } ..... }
67. @anubhavm  !X job "redis" { # Add an update stanza to enable rolling updates of the service update { max_parallel = 1 canary = 5 min_healthy_time = "30s" auto_revert = true } ..... }
68. Terminal $ echo "let’s try it?"
69. @anubhavm  !69 Multi-Datacenter Multi-Region Flexible Workloads Job Priorities Bin Packing Large Scale Operationally Simple
70. @anubhavm  !70 Multi-Datacenter Multi-Region Flexible Workloads Job Priorities Bin Packing Large Scale Operationally Simple
71. s Copyright © 2017 HashiCorp @anubhavm  !71 ONE MORE DEMO
72. Nomad Million Container Challenge 1,000 Jobs 1,000 Tasks per Job 5,000 Hosts on GCE 1,000,000 Containers
73. Copyright © 2017 HashiCorp @anubhavm  !73
75. Copyright © 2017 HashiCorp @anubhavm  !75 “640 KB ought to be enough for anybody.” - Bill Gates
76. Copyright © 2017 HashiCorp @anubhavm  !76 2nd Largest Hedge Fund 18K Cores 5 Hours 2,200 Containers/second
77. Copyright © 2017 HashiCorp @anubhavm  !77 CircleCI 7+ Million Builds a Month Sustain 400-1000 Jobs a Minute Great Talk By Danielle Tomlinson: https://youtu.be/b8NQO_vFAYo
78. Copyright © 2017 HashiCorp @anubhavm  !78 Globally Distributed Optimistically Concurrent Scheduler
79. Copyright © 2017 HashiCorp @anubhavm  !79 Higher Resource Utilization Decouple Work from Resources Better Quality of Service
80. www.hashicorp.com FOR EVERYONE, EVERYWHERE Thank You! I have stickers! Ask me anything. @anubhavm Anubhav Mishra