Scheduling Applications at Scale with Nomad

Scheduling Applications at Scale with
Nomad
Photo by Sašo Tušar on Unsplash
@anubhavm


View Slide

@anubhavm
Anubhav Mishra
Developer Advocate, HashiCorp

View Slide

@anubhavm
Anubhav Mishra
Developer Advocate, HashiCorp
has stickers
Atlan&s
Gopher Artwork by Ashley McNamara

View Slide

PROVISION, SECURE AND RUN ANY INFRASTRUCTURE
Nomad Consul
Vault
Vagrant Packer Terraform
Consul Enterprise
Terraform Enterprise
Vault Enterprise
PRODUCT SUITE
OSS TOOL SUITE
RUN
Applications
SECURE
Application Infrastructure
PROVISION
Infrastructure
FOR INDIVIDUALS FOR TEAMS
Nomad Enterprise

View Slide

View Slide

Copyright © 2017 HashiCorp
@anubhavm
 !6
Globally Distributed
Optimistically Concurrent
Scheduler

View Slide

@anubhavm
 !7
Scheduler

View Slide

Scheduler
Photo by Emma Matthews on Unsplash

View Slide

@anubhavm
 !9
A person or machine that helps scheduling during
each day of work.
Scheduler

View Slide

@anubhavm
 !10

View Slide

@anubhavm
 !11
A computer program that controls or manages the
execution of jobs / processes / operations.
Scheduler (Computing)

View Slide

@anubhavm
 !12

View Slide

@anubhavm
 !13
Schedulers map a set of work to a set of resources

View Slide

@anubhavm

CPU Scheduler
!14
CORE
CORE
CORE
CORE
CPU
SCHEDULER
KERNEL
APACHE
REDIS
BASH

View Slide

@anubhavm

CPU Scheduler
!15
CORE
CORE
CPU
SCHEDULER
KERNEL
APACHE
REDIS
BASH

View Slide

@anubhavm

CPU Scheduler
!16
CORE
CORE
CPU
SCHEDULER
KERNEL
APACHE
REDIS
BASH

View Slide

@anubhavm

CPU Scheduler
!17
CORE
CORE
CPU
SCHEDULER
KERNEL
APACHE
REDIS
BASH

View Slide

@anubhavm

Scheduler Advantages
!18
Higher Resource Utilization
Decouple Work from Resources
Better Quality of Service

View Slide

@anubhavm

Nop! Schedulers Aren’t New Concept
!19

View Slide

@anubhavm

Landscape
!20

View Slide

View Slide

Designing

View Slide

@anubhavm
 !23
Multi-Datacenter
Multi-Region
Flexible Workloads
Job Priorities
Bin Packing
Large Scale
Operationally Simple

View Slide

@anubhavm

Thousands of regions
Tens of thousands of clients per region
Thousands of jobs per region
Scaling Requirements
!24

View Slide

@anubhavm

Our Past Experience
!25
GOSSIP CONSENSUS

View Slide

@anubhavm
 !26
Cluster Management
Gossip Based (P2P)
Membership
Failure Detection
Event System
Serf

View Slide

@anubhavm
 !27
Service Mesh
Service Discovery
Conﬁguration
Coordination (Locking)
Central Servers +
Distributed Clients

View Slide

@anubhavm

Our Past Experience
!28
GOSSIP CONSENSUS
Mature Libraries Proven Design Patterns

View Slide

@anubhavm

Our Past Experience
!29
GOSSIP CONSENSUS
Mature Libraries Proven Design Patterns
?

View Slide

@anubhavm

Our Past Experience
!30
GOSSIP CONSENSUS

View Slide

View Slide

@anubhavm
 !32
Inspired by Google Omega
Optimistic Concurrency
State Coordination
Service & Batch workloads
Pluggable Architecture

View Slide

@anubhavm

Consul Cluster
!33
CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT
SERVER SERVER SERVER
REPLICATION REPLICATION
RPC
RPC
LAN GOSSIP
WAN GOSSIP

View Slide

@anubhavm

Single Region Architecture
!34
CLIENT CLIENT CLIENT
DC1 DC2 DC3
FOLLOWER LEADER FOLLOWER
REPLICATION
FORWARDING
REPLICATION
FORWARDING
RPC RPC RPC

View Slide

@anubhavm

Multi Region Architecture
!35
FOLLOWER LEADER FOLLOWER
REPLICATION
FORWARDING
REPLICATION
REGION B

GOSSIP
FORWARDING
REGION FORWARDING
 REGION A
SERVER
FOLLOWER
SERVER SERVER
LEADER FOLLOWER

View Slide

@anubhavm
 !36
Region is Isolation Domain
1-N Datacenters Per Region
Flexibility to do 1:1 (Consul)
Scheduling Boundary

View Slide

Installing and
Configuring

View Slide

@anubhavm

Installing Nomad
!38
Client Server

View Slide

@anubhavm

Fingerprinting
!39
Type Examples
Operating System Kernel, OS, Version
Hardware CPU, Memory, Disk
Apps (Capabilities) Docker, Java, Consul
Environment AWS, GCE

View Slide

@anubhavm
 !40
Constrain Placement and Bin Pack

View Slide

@anubhavm
 !41
“Task Requires Linux, Docker, and PCI-Compliant Hardware”
expressed as constraints in job ﬁle

View Slide

@anubhavm
 !42
“Task needs 512MB RAM and 1 Core”
expressed as resource in job ﬁle

View Slide

Terminal
$ nomad agent -config=/path/to/config.hcl

View Slide

Terminal
$ echo "let’s try it?"

View Slide

Job File

View Slide

@anubhavm
 !X
job "redis" {
datacenters = ["us-east-1"]
task "redis" {
driver = "docker"
config { image = "redis:latest" }
resources {
cpu = 500 # Mhz
memory = 256 # MB
network {
mbits = 10
port "redis" {}
}
}
}
}

View Slide

@anubhavm
 !X
job "webserver" {
datacenters = ["us-east-1"]
task "webserver" {
driver = "exec"
config {
command = "yet-another-golang-webserver"
}
artifact {
source = "https://github.com/download/../yet-another-golang-webserver"
}
resources {
cpu = 500 # Mhz
memory = 128 # MB
network {
port "http" {
static = 8080
}
}
}
}
}

View Slide

Terminal

View Slide

@anubhavm
 !49
Job speciﬁcation declares what to run

View Slide

@anubhavm
 !50
Nomad determines how and where to run

View Slide

@anubhavm
 !51
Nomad abstracts work from resources

View Slide

Job Types

View Slide

@anubhavm

Nomad has three scheduler types that can be used when creating
your job: service, batch, and system.
Job Types
!53

View Slide

@anubhavm

Service Scheduler Job Type
The service scheduler is designed for scheduling long-lived services
that should never go down. The service scheduler ranks a large
portion of the nodes that meet the jobs constraints and selects the
optimal node to place a task group on.
Examples: webapp, redis
Job Types: Service
!54

View Slide

@anubhavm

Batch Scheduler Job Type
Batch jobs are less sensitive to short-term performance ﬂuctuations
and are short lived, ﬁnishing after some period.
Examples: billing, data replication
Job Types: Batch
!55

View Slide

@anubhavm

System Scheduler Job Type
The system scheduler is used to register jobs that should be run on
all clients that meet the job's constraints. The system scheduler is
also invoked when clients join the cluster or transition into the ready
state.
Examples: logging agent, security auditing tool
Job Types: System
!56

View Slide

Drivers

View Slide

@anubhavm
 !58
Containerized
Virtualized
Standalone
Docker
Qemu / KVM
Java Jar
Static Binaries
rkt
LXC

View Slide

@anubhavm
 !59
Containerized
Virtualized
Standalone
Docker
Qemu / KVM
Java Jar
Static Binaries
rkt
LXC
Windows Server Containers
Hyper-V
Xen
C#

View Slide

Service Discovery

View Slide

@anubhavm

Consul is a free and open-source tool by HashiCorp that implements
service discovery.
It uses the RAFT and gossip protocols to reach massive scale.
It has integrations with health checks, so unhealthy services are not
added to the service discovery layer.
Similar client-server model to Nomad.
Service Discovery using Consul
!61

View Slide

@anubhavm
 !X
job "redis" {
datacenters = [“us-east-1”]
service {
name = "redis"
port = "redis"
check {
type = "tcp"
port = "redis"
interval = "10s"
timeout = "5s"
}
}
}

View Slide

Terminal

View Slide

Advanced Job
Strategies

View Slide

@anubhavm

Nomad currently supports two advanced job strategies:
Rolling Upgrades, Blue/Green & Canary Deployments
Advanced Job Strategies
!65

View Slide

@anubhavm
 !X
job "redis" {
# Add an update stanza to enable rolling updates of the service
update {
max_parallel = 1
min_healthy_time = "30s"
healthy_deadline = "10m"
}
.....
}

View Slide

@anubhavm
 !X
job "redis" {
# Add an update stanza to enable rolling updates of the service
update {
max_parallel = 1
canary = 5
min_healthy_time = "30s"
auto_revert = true
}
.....
}

View Slide

Terminal

View Slide

@anubhavm
 !69
Multi-Datacenter
Multi-Region
Flexible Workloads
Job Priorities
Bin Packing
Large Scale

View Slide

@anubhavm
 !70
Multi-Datacenter
Multi-Region
Flexible Workloads
Job Priorities
Bin Packing
Large Scale

View Slide

Nomad
Million Container
Challenge
1,000 Jobs
1,000 Tasks per Job
5,000 Hosts on GCE
1,000,000 Containers

View Slide

@anubhavm
 !73

View Slide

View Slide

@anubhavm
 !75
“640 KB ought to be enough for anybody.”
- Bill Gates

View Slide

@anubhavm
 !76
2nd Largest Hedge Fund
18K Cores
5 Hours
2,200 Containers/second

View Slide

@anubhavm
 !77
CircleCI 7+ Million Builds a Month
Sustain 400-1000 Jobs a Minute
Great Talk By Danielle Tomlinson: https://youtu.be/b8NQO_vFAYo

View Slide

@anubhavm
 !78
Scheduler

View Slide

@anubhavm
 !79
Higher Resource Utilization
Decouple Work from Resources
Better Quality of Service

View Slide

www.hashicorp.com
FOR EVERYONE, EVERYWHERE
Thank You!
I have stickers!
Ask me anything. @anubhavm
Anubhav Mishra

View Slide

Scheduling Applications at Scale with Nomad

Scheduling Applications at Scale with Nomad

Anubhav Mishra

More Decks by Anubhav Mishra

Other Decks in Technology

Featured

Transcript