The Life of a Placement

A deep dive into how Nomad makes placement decisions.

Transcript

1. @adadgar Place or place not, there is no try
2. Alex Dadgar Nomad Team Lead, HashiCorp @adadgar
3. Nomad @adadgar
4. @adadgar Jobs Nodes
5. Scheduler: Map Work to Resources @adadgar Jobs and Nodes … … Placements …
6. @adadgar Jobs Nodes
7. Scheduler: Map Work to Resources @adadgar Jobs and Nodes … … Placements …
8. Job File @adadgar
9. HASHICORP example.nomad # Define our simple redis job job "redis" { # Run only in us-east-1 datacenters = ["us-east-1"] # Define the single redis task using Docker task "redis" { driver = "docker" config { image = "redis:latest" } resources { cpu = 500 # Mhz memory = 256 # MB network { mbits = 10 port “redis” {} } } } }
10. HASHICORP Job Specification Declares what to run
11. HASHICORP Job Specification Nomad determines where and manages how to run
12. HASHICORP # Define our simple redis job job "redis" { # Run only in us-east-1 datacenters = ["us-east-1"] # Define the single redis task using Docker task "redis" { driver = "docker" config { image = "redis:latest" } resources { cpu = 500 # Mhz memory = 256 # MB network { mbits = 10 port “redis” {} } } } }
13. HASHICORP # Define our simple redis job job "redis" { ... group "redis" { # Ensure reliable network speed constraint { attribute = "${attr.platform.aws.instance-type}" value = "m4.xlarge" } task "redis" { driver = "docker" config { image = "redis:latest" } resources { cpu = 500 # Mhz memory = 256 # MB network { mbits = 10 port “redis” {} } } } } }
14. HASHICORP # Define our simple redis job job "redis" { ... group "redis" { count = 3 constraint { distinct_hosts = true } task "redis" { driver = "docker" config { image = "redis:latest" } resources { cpu = 500 # Mhz memory = 256 # MB network { mbits = 10 port “redis” {} } } } } }
15. HASHICORP # Define our simple redis job job "redis" { ... group "redis" { count = 3 constraint { distinct_property = "${meta.rack}" } task "redis" { driver = "docker" config { image = "redis:latest" } resources { cpu = 500 # Mhz memory = 256 # MB network { mbits = 10 port “redis” {} } } } } }
16. Impact Placement? • Allowed datacenters • Driver type • Constraints • Resource Request • CPU, Memory, Disk, Network (Bandwidth and Ports) • Future: GPU @adadgar
17. HASHICORP Job Specification Powerful yet simple
18. Scheduler: Map Work to Resources @adadgar Jobs and Nodes … … Placements …
19. Scheduler • Reconcile with what already exists • Stop any unneeded allocations • In-place upgrade where possible • Rolling updates • Place the required new allocations @adadgar
20. Stack @adadgar Node Source Filter 1 Filter 2 Ranker 1 Ranker 2 Limit Selector • Chain multiple iterators together • Start with all possibly eligible nodes • Final output is eligible and highly ranked node
21. HASHICORP job "redis" { datacenters = [“us-east-1”] group "redis" { count = 3 constraint { attribute = "${attr.platform.aws.instance-type}" value = "m4.xlarge" } task "redis" { driver = “docker" config { image = "redis:latest" } resources { cpu = 500 # Mhz memory = 256 # MB network { mbits = 10 port “redis” {} } } } } }
22. @adadgar us-east-1 us-east-2 Node Source
23. @adadgar us-east-1 Node Source
24. @adadgar us-east-1 Node Source
25. @adadgar Node Source
26. HASHICORP job "redis" { datacenters = [“us-east-1”] group "redis" { count = 3 constraint { attribute = "${attr.platform.aws.instance-type}" value = "m4.xlarge" } task "redis" { driver = “docker" config { image = "redis:latest" } resources { cpu = 500 # Mhz memory = 256 # MB network { mbits = 10 port “redis” {} } } } } }
27. @adadgar Node Source Driver Filter
28. HASHICORP job "redis" { datacenters = [“us-east-1”] group "redis" { count = 3 constraint { attribute = "${attr.platform.aws.instance-type}" value = "m4.xlarge" } task "redis" { driver = “docker" config { image = "redis:latest" } resources { cpu = 500 # Mhz memory = 256 # MB network { mbits = 10 port “redis” {} } } } } }
29. @adadgar Driver Filter Constraint Filter
30. Types of Constraint Filters • Regex Matcher • AWS instance type matches “m4.[0-9]+xlarge • Version • Is Java version >= 1.8.0 • Distinct Hosts • Spread across unique hosts • Distinct Property • Spread across unique values of attribute (e.g. different racks) @adadgar
31. @adadgar Distinct Property Filter Seen: [] 11 12 13 11 13
32. @adadgar Distinct Property Filter Seen: [11] 11 12 13 13 11 8
33. @adadgar Distinct Property Filter Seen: [11, 13] 11 12 13 11 13 8 6
34. @adadgar Distinct Property Filter Seen: [11, 12, 13] 11 13 11 12 8 6 4 3
35. @adadgar Driver Filter Constraint Filter
36. @adadgar Driver Filter Constraint Filter ?
37. @adadgar Driver Filter Constraint Filter ?
38. @adadgar Driver Filter Constraint Filter ?
39. Stack @adadgar Node Source Filter 1 Filter 2 Ranker 1 Ranker 2 Limit Selector • Ranker can filter and add a score to each node
40. Rankers • Bin Packer • Job Anti-Affinity @adadgar
41. Bin Packer • Filter Nodes that can’t fit the placement (not enough resources) • Score nodes based upon the density @adadgar
42. @adadgar Node 1 Node 3 Place Node 2
43. @adadgar Node 1 Node 3 Place Node 2 Place
44. @adadgar Node 1 Node 3 Place Node 2 Place
45. @adadgar Node 1 Node 3 Place Node 2 Place Place Place
46. @adadgar Node 1 Node 3 Place Node 2 Place Place
47. How and Why? @adadgar
48. How? • When Nodes join they fingerprint host and determine resources • Filtering step: Do already placed allocations + new allocation fit on node • Rank: Produce score between 0-18. • Score = 20 - 10^x - 10^y, where x is free CPU and y is free memory @adadgar
49. @adadgar
50. Why? Bin Packing versus Spread • Accommodate heterogeneous workloads • Easier capacity planning • Future: Autoscaling @adadgar
51. Heterogeneous Workloads @adadgar 3 GB RAM 4 GB RAM 3 GB RAM Place Following: Bin Pack Spread 4 GB RAM
52. Heterogeneous Workloads @adadgar 3 GB RAM 4 GB RAM 3 GB RAM Bin Pack Spread 4 GB RAM 4 GB RAM 4 GB RAM 3 GB RAM 3 GB RAM Place Following:
53. Heterogeneous Workloads @adadgar 3 GB RAM 4 GB RAM 3 GB RAM Bin Pack Spread 4 GB RAM 4 GB RAM 4 GB RAM 3 GB RAM 3 GB RAM 4 GB RAM 4 GB RAM 3 GB RAM 3 GB RAM Place Following:
54. Heterogeneous Workloads @adadgar 3 GB RAM 4 GB RAM 3 GB RAM Bin Pack Spread 4 GB RAM 6 GB RAM 4 GB RAM 4 GB RAM 3 GB RAM 3 GB RAM 4 GB RAM 4 GB RAM 3 GB RAM 3 GB RAM Place Following:
55. Heterogeneous Workloads @adadgar 3 GB RAM 4 GB RAM 3 GB RAM Bin Pack Spread 4 GB RAM 6 GB RAM 4 GB RAM 4 GB RAM 3 GB RAM 3 GB RAM 4 GB RAM 4 GB RAM 3 GB RAM 3 GB RAM 6 GB RAM Place Following:
56. Heterogeneous Workloads @adadgar 3 GB RAM 4 GB RAM 3 GB RAM Bin Pack Spread 4 GB RAM 6 GB RAM 4 GB RAM 4 GB RAM 3 GB RAM 3 GB RAM 4 GB RAM 4 GB RAM 3 GB RAM 3 GB RAM 6 GB RAM Place Following:
57. Why? Bin Packing versus Spread • Accommodate heterogeneous workloads • Easier capacity planning • Future: Autoscaling @adadgar
58. Capacity Planning • Assume 10 nodes • Scheduling many jobs that take roughly half of the overall resources @adadgar
59. Spread @adadgar Utilization % 0% 25% 50% 75% 100% Nodes 1 2 3 4 5 6 7 8 9 10
60. Bin Packing @adadgar Utilization % 0% 25% 50% 75% 100% Nodes 1 2 3 4 5 6 7 8 9 10
61. Bin Packing @adadgar Utilization % 0% 25% 50% 75% 100% Nodes 1 2 3 4 5 6 7 8 9 10
62. Why? Bin Packing versus Spread • Accommodate heterogeneous workloads • Easier capacity planning • Future: Autoscaling @adadgar
63. Auto Scaling - Scaling Up @adadgar 3 GB RAM 4 GB RAM 3 GB RAM Bin Pack Spread 4 GB RAM 4 GB RAM 4 GB RAM 3 GB RAM 3 GB RAM 6 GB RAM Place Following: 4 GB RAM 3 GB RAM 3 GB RAM 4 GB RAM 6 GB RAM 6 GB RAM
64. Auto Scaling - Scaling Up @adadgar 3 GB RAM 4 GB RAM 3 GB RAM Bin Pack Spread 4 GB RAM 4 GB RAM 4 GB RAM 3 GB RAM 3 GB RAM Place Following: 4 GB RAM 3 GB RAM 3 GB RAM 4 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM
65. Auto Scaling - Scaling Up @adadgar 3 GB RAM 4 GB RAM 3 GB RAM Bin Pack Spread 4 GB RAM 4 GB RAM 4 GB RAM 3 GB RAM 3 GB RAM Place Following: 4 GB RAM 3 GB RAM 3 GB RAM 4 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM
66. Auto Scaling - Scaling Up @adadgar 3 GB RAM 4 GB RAM 3 GB RAM Bin Pack Wasted = 5 / 32 = 15.6% Spread Wasted = 35% and Extra Node 4 GB RAM 4 GB RAM 4 GB RAM 3 GB RAM 3 GB RAM Place Following: 4 GB RAM 3 GB RAM 3 GB RAM 4 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM
67. Auto Scaling - Scaling Down @adadgar 3 GB RAM 4 GB RAM 3 GB RAM Bin Pack Spread 4 GB RAM 4 GB RAM 4 GB RAM 3 GB RAM 3 GB RAM Place Following: 4 GB RAM 3 GB RAM 3 GB RAM 4 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM
68. Auto Scaling - Scaling Down @adadgar 3 GB RAM 4 GB RAM 3 GB RAM Bin Pack Spread 4 GB RAM 4 GB RAM 4 GB RAM 3 GB RAM 3 GB RAM Place Following: 4 GB RAM 3 GB RAM 3 GB RAM 4 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM
69. Auto Scaling - Scaling Down @adadgar 3 GB RAM 4 GB RAM 3 GB RAM Bin Pack Spread 4 GB RAM 4 GB RAM 4 GB RAM 3 GB RAM 3 GB RAM Place Following: 4 GB RAM 3 GB RAM 3 GB RAM 4 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM
70. Auto Scaling - Scaling Down @adadgar 3 GB RAM 4 GB RAM 3 GB RAM Bin Pack Spread 4 GB RAM 4 GB RAM 3 GB RAM Place Following: 4 GB RAM 3 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM
71. Auto Scaling - Scaling Down @adadgar 3 GB RAM 4 GB RAM 3 GB RAM Bin Pack Spread 4 GB RAM 4 GB RAM 3 GB RAM Place Following: 4 GB RAM 3 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM
72. Auto Scaling - Scaling Down @adadgar 3 GB RAM 4 GB RAM 3 GB RAM Bin Pack Spread 4 GB RAM 4 GB RAM 3 GB RAM Place Following: 4 GB RAM 3 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM
73. Auto Scaling - Scaling Down @adadgar 3 GB RAM 4 GB RAM 3 GB RAM Bin Pack Spread 4 GB RAM 4 GB RAM 3 GB RAM Place Following: 4 GB RAM 3 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM
74. Auto Scaling - Scaling Down @adadgar 3 GB RAM 4 GB RAM 3 GB RAM Bin Pack Spread 4 GB RAM 4 GB RAM 3 GB RAM Place Following: 4 GB RAM 3 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM 6 GB RAM
75. Why Spread? Less impact when node dies! @adadgar
76. Rankers • Bin Packer • Job Anti-Affinity @adadgar
77. Job Anti-Affinity • Does not filter nodes • Applies a penalty to collocating multiple placements onto the same node @adadgar
78. Job Anti-Affinity @adadgar Score: 13 Score: 9 Place Place Anti-Affinity
79. Job Anti-Affinity @adadgar Score: 13 - 5 Score: 9 Place Place Anti-Affinity
80. Job Anti-Affinity @adadgar Anti-Affinity Score: 13 - 5 Score: 9 Place Place
81. Stack @adadgar Node Source Filter 1 Filter 2 Ranker 1 Ranker 2 Limit Selector • Limit Selector picks the highest scored node using sampling
82. Limit Selector @adadgar Limit Selector 17 1 3 11 8 16 12 5 3 7 14
83. Limit Selector @adadgar Limit Selector 17 17 1 3 11 8 16 12 5 3 7 14
84. Limit Selector @adadgar Limit Selector 17 1 3 11 8 16 12 5 3 7 14 Ranking Is Costly 17
85. Why is Ranking expense? • Each placement must go through all iterators. • Must be done sequentially • Bin-packing and job anti-affinity score may change @adadgar
86. Limit Selector - Batch @adadgar Limit Selector 7 14 14 Power of Two Choices
87. Power of Two Choices @adadgar • Imagine n balls that are placed into n buckets • Maximum load in any bucket: • Chosen at random: ~ log(n)/loglog(n) • Chosen sequentially, picking least full of d bins: ~ loglog(n)/log(d) +O(1) • d=2 yields a exponential improvement
88. Power of Two Choices @adadgar • Inspired from the Berkley Sparrow Scheduler • Originally from Load Balancing Research • Talk: https://people.eecs.berkeley.edu/~keo/talks/sparrow- sosp-talk.pdf • Paper: https://www.eecs.harvard.edu/~michaelm/postscripts/ mythesis.pdf
89. Limit Selector - Service @adadgar Limit Selector 16 12 5 3 7 14 Log(all eligible nodes) 16
90. All that for one allocation! @adadgar Node Source Filter 1 Filter 2 Ranker 1 Ranker 2 Limit Selector Allocation
91. Nomad 0.6.0 Highlights • Rolling Deploys using health checks • Dynamic Environment variables @adadgar
92. @adadgar Thank You!