HASHICORP Towards Running Stateful Applications on Nomad Diptanu Gon Choudhury @diptanu HashiConf US Sep 2016

Quest to increase agility and reliability Develop Build Deploy Production Fast turnaround High Availability
 Control Plane for operations

Quest to increase agility and reliability Develop Build Deploy Region 1 Region 2 DC 1 DC 2 DC 3 DC 3 DC 3 DC 3

Nomad Multi DC Multi Region Diverse workload Diverse platform Built for scale Operationally simple

Nomad Optimistic Concurrency State Coordination Service, Batch and System workloads

Application Configuration Constraints job "redis" { datacenters = ["us-east-1"] group “cache” { count = 100 task "redis" { driver = "docker" config { image = "redis:latest" } resources { cpu = 500 memory = 256 network { mbits = 10 dynamic_ports = ["redis"] } } } }

job "redis" { datacenters = ["us-east-1"] task "redis" { driver = "docker" config { image = "redis:latest" } resources { cpu = 500 memory = 256 network { mbits = 10 dynamic_ports = ["redis"] } } } }

Batch Scheduler
 Service Scheduler Restart Policies
 System Scheduler Consul Integration
 Log Management
 Runtime Stats
 Job Plans Vault Integration
 TLS Sticky Volumes
 Disk Watchers Volume Plugins
 Network Plugins ACL Priorities
 Quotas

Nomad is a great platform for stateless workloads

Improved support and features
 for Stateful Applications

Nomad Jobs Task Group Task Group Task A Task B Task C Task D Allocation Allocation Job

Allocations Allocations are instances of a task group on a compute node Ephemeral in nature Allocations manage the life cycle of tasks within them Provides the environment and file system for the tasks

Allocation Resources CPU Shares Memory Network Ports and IPs Task Resources Shared Resources Disk Resources

Allocation Directory Allocation directory provides a shared data directory for tasks stdout and stderr streams of tasks are written in logs dir Each Task has a task local directory

Allocation Directory Ephemeral Disk Allocation Directory Task Dir Task Dir Allocation Directory

Allocation Directory Allocation Directory data logs tmp local tmp Task Dir Task Dir local tmp

Allocations Allocations are ephemeral in nature Allocations can be restarted on a different node Allocation directory not preserved when restarted on same node They are garbage collected by Nomad after they transition to terminal state

Sticky Volumes

Sticky Volumes Prefers to restart allocation on the same nodes to avoid replication Replicate the shared data dir and task local dirs of allocations Best effort replication, not a Distributed File Systems

group “redis” { ephemeral_disk { sticky = true
 size = 20000 } task “redis” { … } 
 task “backup-agent” { … } } cache.nomad

Ephemeral Disk Shared Allocation Directory Task A Task B

Ephemeral Disk Shared Allocation Directory Task A Task B Shared Allocation Directory Task A Task B Move

Node A Allocation Allocation Node B P2P Replication

Disk Watcher Watch dog process to monitor disk usage Allocations get killed if they use more disk resources than they are allocated If an allocation exceeds it’s disk quota, the allocation fails permanently

The Future Data Volumes to add additional volumes to allocations Volume plugins for materializing volumes on storage services like EBS, EFS, NFS, etc File system drivers for supporting file systems such as ZFS

Thanks! https://www.nomadproject.io/
 https://github.com/hashicorp/nomad