IBM Dev Day: Running AI Workloads on Nomad

In this presentation, I dive into how to think about AI workloads and why HashiCorp Nomad is the right foundation for an AI platform.

This version of the talk was given at IBM Dev Day: AI Demystified in January 2026.

Transcript

1. Running AI Workloads on Nomad IBM Dev Day: AI Demystified January 29, 2026 Kerim Satirli Senior Developer Advocate II
2. Kerim Satirli Senior Developer Advocate II HashiCorp he / him @ksatirli
3. GPUs are scarce and expensive (and oddly shaped)
4. 2010 - 2025 Interest in GPU Scheduling (Source: Google Trends)
5. ©2026 HASHICORP AI workloads are not special snowflakes
6. ©2026 HASHICORP • resource hungry • state aware • hardware constrained AI workloads are just
7. Nomad is exceptionally great at these jobs.
8. Workload Orchestration with HashiCorp Nomad
9. Infrastructure Provisioning build images with Packer and Ansible provision infrastructure with Terraform orchestrate workloads with Nomad secure access and data with Vault
10. Infrastructure Landscape Local / Edge Cloud NVIDIA Jetson (100 TOPS) NVIDIA Jetson (100 TOPS) Windows Server (1300 TOPS) other connected clients (not GPU accelerated) IBM Cloud Compute AWS EC2 Azure VMs other supporting infrastructure (file and data storage, etc.)
12. Job Specification Overview Task Group Job
13. defines basic job properties • datacenter and region • type of job • update strategy defines atomic units of work • driver selection • task environment • resource requirements defines how to co-locate tasks • network config • volume config • service discovery Job Specification Overview Task Job Group
14. Job Specification Overview Affinity and Constraints defines basic job properties • datacenter and region • type of job • update strategy defines atomic units of work • driver selection • task environment • resource requirements defines how to co-locate tasks • network config • volume config • service discovery Task Job Group
15. Writing Job Specifications for HashiCorp Nomad
16. Basic Job Spec job "raw_exec" { datacenters = ["dc1"] type = "batch" group "user" { task "whoami" { driver = "raw_exec" user = "nomad" config { command = "/usr/bin/whoami" args = [] } resources { cpu = 100 memory = 100 } } } } raw-exec.nomad.hcl
17. Basic Job Spec job "raw_exec" { datacenters = ["dc1"] type = "batch" group "user" { task "whoami" { driver = "raw_exec" user = "nomad" config { command = "/usr/bin/whoami" args = [] } resources { cpu = 100 memory = 100 } } } } raw-exec.nomad.hcl
18. Basic Job Spec job "raw_exec" { datacenters = ["dc1"] type = "batch" group "user" { task "whoami" { driver = "raw_exec" user = "nomad" config { command = "/usr/bin/whoami" args = [] } resources { cpu = 100 memory = 100 } } } } raw-exec.nomad.hcl
19. OS-constrained Job Spec job "run_on_windows" { datacenters = ["dc1"] group "windows" { constraint { attribute = "${attr.kernel.name}" operator = "=" value = "windows" } service { name = "windows-iis" tags = ["windows","iis"] port = "www" check { name = "alive" type = "tcp" interval = "10s" timeout = "2s" } } run-on-windows.nomad.hcl
20. Inject Secure Data into Job Spec job "certs_from_vault" { datacenters = ["dc1"] group "group" { task "sleepy" { driver = "exec" vault { policies = ["nomad-client"] change_mode = "signal" change_signal = "SIGUSR1" } template { destination = "${NOMAD_SECRETS_DIR}/certificate.crt" change_mode = "restart" data = <<EOH {{ with pkiCert "pki/issue/svcs-dev" "common_name=svcs.dev" "ttl=24h" "ip_sans=127.0.0.1" }} {{- .Data.certificate -}} {{ end }} EOH } certs-from-vault.nomad.hcl
21. Inject Secure Data into Job Spec job "certs_from_vault" { datacenters = ["dc1"] group "group" { task "sleepy" { driver = "exec" vault { policies = ["nomad-client"] change_mode = "signal" change_signal = "SIGUSR1" } template { destination = "${NOMAD_SECRETS_DIR}/certificate.crt" change_mode = "restart" data = <<EOH {{ with pkiCert "pki/issue/svcs-dev" "common_name=svcs.dev" "ttl=24h" "ip_sans=127.0.0.1" }} {{- .Data.certificate -}} {{ end }} EOH } certs-from-vault.nomad.hcl
22. GPU Workloads with Nomad plugin "nomad-device-nvidia" { config { enabled = true # find GPU IDs by running `nvidia-smi -L` ignored_gpu_ids = [ "GPU-4f707ad8-2c9a-3b1f-9a58-1c2e9f0b7c3d", ] fingerprint_period = "1m" } } nomad-gpu-config.hcl
23. GPU Workloads with Nomad job "gpu_workload" { datacenters = ["dc1"] type = "batch" group "smi" { task "smi" { driver = "docker" config { # see https://hub.docker.com/r/nvidia/cuda/tags image = "nvidia/cuda:13.1.1-base-ubuntu24.04" command = "nvidia-smi" } resources { device "nvidia/gpu" { count = 1 } } } } } nvidia-smi.nomad.hcl
24. GPU Workloads with Nomad job "gpu_workload" { datacenters = ["dc1"] type = "batch" group "smi" { task "smi" { driver = "docker" config { # see https://hub.docker.com/r/nvidia/cuda/tags image = "nvidia/cuda:13.1.1-base-ubuntu24.04" command = "nvidia-smi" } resources { device "nvidia/gpu" { count = 1 } } } } } nvidia-smi.nomad.hcl
25. GPU Workloads with Nomad job "gpu_workload" { # <other config hidden> group "smi" { task "smi" { # <other config hidden> resources { device "nvidia/gpu" { count = 1 } } } } nvidia-smi.nomad.hcl
26. GPU Workloads with Nomad job "gpu_workload" { # <other config hidden> group "smi" { task "smi" { # <other config hidden> resources { device "nvidia/gpu" { count = 1 affinity { attribute = "${device.model}" # covers both "H100 PCIe" and "H100 SXM" type of GPUs operator = "regexp" value = "H100" weight = 50 } } } } } nvidia-smi.nomad.hcl
27. GPU Workloads with Nomad 5GB 5GB 5GB 5GB 5GB 5GB 5GB 5GB 1 compute 1 compute 1 compute 1 compute 1 compute 1 compute 1 compute
28. 5GB 5GB 5GB 5GB 5GB 5GB 5GB 1 compute 1 compute 1 compute 1 compute 1 compute 1 compute GPU Workloads with Nomad 5GB 1 compute
29. 5GB 5GB 5GB 5GB 1 compute 1 compute 1 compute GPU Workloads with Nomad 5GB 5GB 5GB 5GB 1 compute 1 compute 1 compute 1 compute
30. GPU Workloads with Nomad job "gpu_workload" { datacenters = ["dc1"] node_pool = "gpu_instances" # <other config hidden> group "smi" { task "smi" { # <other config hidden> resources { device "nvidia/gpu/NVIDIA A100-SXM4-40GB MIG 1g.5gb" { count = 1 } } } } } nvidia-smi.nomad.hcl
31. GPU Workloads with Nomad job "gpu_workload" { datacenters = ["dc1"] node_pool = "gpu_instances" # <other config hidden> group "smi" { task "smi" { # <other config hidden> resources { device "nvidia/gpu/NVIDIA A100-SXM4-40GB MIG 1g.5gb" { count = 1 } } } } } nvidia-smi.nomad.hcl
32. IBM Granite Docling Langflow Cassandra Open WebUI (raw) execution containerized exec. GPU-accelerated Scheduling Suggestions
33. IBM Granite Docling Langflow Cassandra (raw) execution containerized exec. GPU-accelerated Scheduling Suggestions Open WebUI
34. IBM Granite Langflow Cassandra (raw) execution containerized exec. GPU-accelerated Scheduling Suggestions Open WebUI Docling
35. IBM Granite Langflow Cassandra (raw) execution containerized exec. GPU-accelerated Scheduling Suggestions Open WebUI Docling
36. IBM Granite Langflow Cassandra (raw) execution containerized exec. GPU-accelerated Scheduling Suggestions Open WebUI Docling
37. IBM Granite Langflow (raw) execution containerized exec. GPU-accelerated Scheduling Suggestions Open WebUI Docling Cassandra
38. IBM Granite (raw) execution containerized exec. GPU-accelerated Scheduling Suggestions Open WebUI Docling Cassandra Langflow
39. ©2026 HASHICORP IB M Dev ay: y Ams DtifdDt AtM D(nt:sDasd A ovasl Dh:p
40. ©2026 HASHICORP IB)M Dev ay:)DtifdDt) otDas D:) s )f:t)trd:sdAG)ovasl Dh:)PtsstDp
41. speakerdeck.com'ksatirli Thank you