Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Towards Running Stateful Applications on Nomad
Search
Diptanu Choudhury
August 03, 2016
0
970
Towards Running Stateful Applications on Nomad
Diptanu Choudhury
August 03, 2016
Tweet
Share
More Decks by Diptanu Choudhury
See All by Diptanu Choudhury
Chaos Engineering and design principles for building Highly Available services on the cloud
diptanu
0
440
Taming the public and private clouds with Nomad
diptanu
1
200
Distributed Scheduling with Apache Mesos in the Cloud
diptanu
1
430
Reliably shipping containers in a resource rich world using Titan
diptanu
8
2.3k
Distributed Scheduling with Apache Mesos in the Cloud
diptanu
0
390
Featured
See All Featured
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
12
1.2k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
132
19k
Fireside Chat
paigeccino
40
3.7k
How To Stay Up To Date on Web Technology
chriscoyier
791
250k
Art, The Web, and Tiny UX
lynnandtonic
303
21k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
4k
The Art of Programming - Codeland 2020
erikaheidi
56
14k
Scaling GitHub
holman
463
140k
Raft: Consensus for Rubyists
vanstee
139
7.1k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
19
1.2k
A Modern Web Designer's Workflow
chriscoyier
697
190k
YesSQL, Process and Tooling at Scale
rocio
173
14k
Transcript
HASHICORP Towards Running Stateful Applications on Nomad Diptanu Gon Choudhury
@diptanu HashiConf US Sep 2016
Quest to increase agility and reliability Develop Build Deploy Production
Fast turnaround High Availability Control Plane for operations
Quest to increase agility and reliability Develop Build Deploy Region
1 Region 2 DC 1 DC 2 DC 3 DC 3 DC 3 DC 3
Nomad Multi DC Multi Region Diverse workload Diverse platform Built
for scale Operationally simple
Nomad Optimistic Concurrency State Coordination Service, Batch and System workloads
Application Configuration Constraints job "redis" { datacenters = ["us-east-1"] group
“cache” { count = 100 task "redis" { driver = "docker" config { image = "redis:latest" } resources { cpu = 500 memory = 256 network { mbits = 10 dynamic_ports = ["redis"] } } } }
job "redis" { datacenters = ["us-east-1"] task "redis" { driver
= "docker" config { image = "redis:latest" } resources { cpu = 500 memory = 256 network { mbits = 10 dynamic_ports = ["redis"] } } } }
Batch Scheduler Service Scheduler Restart Policies System Scheduler Consul Integration
Log Management Runtime Stats Job Plans Vault Integration TLS Sticky Volumes Disk Watchers Volume Plugins Network Plugins ACL Priorities Quotas
Nomad is a great platform for stateless workloads
Improved support and features for Stateful Applications
Nomad Jobs Task Group Task Group Task A Task B
Task C Task D Allocation Allocation Job
Allocations Allocations are instances of a task group on a
compute node Ephemeral in nature Allocations manage the life cycle of tasks within them Provides the environment and file system for the tasks
Allocation Resources CPU Shares Memory Network Ports and IPs Task
Resources Shared Resources Disk Resources
Allocation Directory Allocation directory provides a shared data directory for
tasks stdout and stderr streams of tasks are written in logs dir Each Task has a task local directory
Allocation Directory Ephemeral Disk Allocation Directory Task Dir Task Dir
Allocation Directory
Allocation Directory Allocation Directory data logs tmp local tmp Task
Dir Task Dir local tmp
Allocations Allocations are ephemeral in nature Allocations can be restarted
on a different node Allocation directory not preserved when restarted on same node They are garbage collected by Nomad after they transition to terminal state
Sticky Volumes
Sticky Volumes Prefers to restart allocation on the same nodes
to avoid replication Replicate the shared data dir and task local dirs of allocations Best effort replication, not a Distributed File Systems
group “redis” { ephemeral_disk { sticky = true size =
20000 } task “redis” { … } task “backup-agent” { … } } cache.nomad
Ephemeral Disk Shared Allocation Directory Task A Task B
Ephemeral Disk Shared Allocation Directory Task A Task B Shared
Allocation Directory Task A Task B Move
Node A Allocation Allocation Node B P2P Replication
Disk Watcher Watch dog process to monitor disk usage Allocations
get killed if they use more disk resources than they are allocated If an allocation exceeds it’s disk quota, the allocation fails permanently
The Future Data Volumes to add additional volumes to allocations
Volume plugins for materializing volumes on storage services like EBS, EFS, NFS, etc File system drivers for supporting file systems such as ZFS
Thanks! https://www.nomadproject.io/ https://github.com/hashicorp/nomad