AI and ML with HashiCorp Nomad

In this presentation, I explain how to use the NVIDIA Device Driver for HashiCorp Nomad to deploy AI and ML workloads.

This version of the talk was given at HashiConf Boston, in October 2024.

Transcript

1. Kerim Satirli Sr. Developer Advocate Infrastructure & Orchestration AI workloads with Nomad and NVIDIA at Scale
2. Senior Developer Advocate, Infrastructure & Orchestration he / him @ksatirli Kerim Satirli
3. Rate Time Interest in AI Price of GPUs
4. Cost of Hardware Cost Time $ 4.50 $ 8.10 $ 1.85
5. amazon.com/dp/B0CGL976H5 Cost of Hardware
6. amazon.com/dp/B0CGL976H5 Cost of Hardware
7. H100 4000 TFLOPS US$ 28.500 A100 600 - 1200 TOPS US$ 22.000 Jetson Orin series 270 TOPS $ 500 - 2,000 GeForce RTX 3090 230 TOPS US$ 1.000 Jetson Xavier series 20 - 30 TOPS $ 500 - 800 Raspberry Pi 5 with AI Accelerator 10 TOPS US$ 100 Cost of Hardware
8. H100 4000 TFLOPS US$ 28.500 A100 600 - 1200 TOPS US$ 22.000 Jetson Orin series 270 TOPS $ 500 - 2,000 GeForce RTX 3090 230 TOPS US$ 1.000 Jetson Xavier series 20 - 30 TOPS $ 500 - 800 Raspberry Pi 5 with AI Accelerator 10 TOPS US$ 100 Cost of Hardware
9. Workload 01100 10110 11110 Edge Virt NVIDIA
10. 01100 10110 11110 Edge Virt NVIDIA Risk
11. 5GB 5GB 5GB 5GB 5GB 5GB 5GB 5GB 1 compute 1 compute 1 compute 1 compute 1 compute 1 compute 1 compute
12. job "gapcloser-scrapers" { datacenters = ["*"] namespace = "gapcloser" node_pool = "gpu_instances" type = "batch" group "scrapers" { count = 1 task "code-scraper" { driver = "docker" config { image = "nvidia/cuda:12.6.0-base" command = "nvidia-smi" } resources { cores = 16 memory = 8192 Targeting Partitions gapcloser.nomad.hcl
13. driver = "docker" config { image = "nvidia/cuda:12.6.0-base" command = "nvidia-smi" } resources { cores = 16 memory = 8192 device "nvidia/gpu/NVIDIA A100-SXM4-40GB MIG 1g.5gb" { count = 3 } device "nvidia/gpu/NVIDIA A100-SXM4-40GB MIG 3g.20gb" { count = 1 } } } } } Targeting Partitions gapcloser.nomad.hcl
14. driver = "docker" config { image = "nvidia/cuda:12.6.0-base" command = "nvidia-smi" } resources { cores = 16 memory = 8192 device "nvidia/gpu/NVIDIA A100-SXM4-40GB MIG 1g.5gb" { count = 3 } device "nvidia/gpu/NVIDIA A100-SXM4-40GB MIG 3g.20gb" { count = 1 } } } } } Targeting Partitions gapcloser.nomad.hcl
15. driver = "docker" config { image = "nvidia/cuda:12.6.0-base" command = "nvidia-smi" } resources { cores = 16 memory = 8192 device "nvidia/gpu/NVIDIA A100-SXM4-40GB MIG 1g.5gb" { count = 3 } device "nvidia/gpu/NVIDIA A100-SXM4-40GB MIG 3g.20gb" { count = 1 } } } } } Targeting Partitions gapcloser.nomad.hcl
16. Memory MiB Power Watt BAR1 size MiB Driver version (Semantic) version string PCI bandwidth MB/s Core and memory clock speed MHz Fingerprinting Options
17. Next steps hashi.co/nomad-nvidia-plugin NVIDIA GPU Device Plugin
18. Thank you