Paris container Day 2017 - Living the Nomadic life - Nic Jackson, HashiCorp

Transcript

1. www.hashicorp.com O @hashicor p hello@hashicorp.co m
2. nic@hashicorp.com @sheriffjackson NIC JACKSON
3. 3 AGENDA HASHICORP SCHEDULING NOMAD Overview Fundamentals Job Configuration Scheduling Demo
4. HASHICORP OVERVIEW
5. 5 FOUNDED 2012 by Mitchell Hashimoto and Armon Dadgar MISSION We enable organizations to provision, secure, and run any infrastructure for any application INVESTORS Mayfield Fund, GGV Capital, Redpoint and True Ventures KEY PRODUCTS Vagrant, Packer, Terraform, Vault, Nomad, Consul COMPANY OVERVIEW
6. 6 OSS TO ENTERPRISE SOFTWARE INNOVATORS TECHNOLOGY PARTNERS
7. 7 PRODUCT SUITE
8. 8 NOMAD Nomad
9. SCHEDULING OVERVIEW
10. Schedulers map a set of work to a set of resources
11. 11 CPU SCHEDULER 11 CORE CORE CORE CORE CPU SCHEDULER KERNEL APACHE REDIS BASH
12. 12 CPU SCHEDULER 12 CORE CORE CPU SCHEDULER KERNEL APACHE REDIS BASH
13. 13 SCHEDULERS IN THE WILD 13 Type Work Resources CPU Scheduler Threads Physical Cores EC2 / Nova Virtual Machines Hypervisors Hadoop YARN MapReduce Jobs Client Nodes Cluster Scheduler Applications Machines
14. 14 SCHEDULER ADVANTAGES 14 Higher Resource Utilization Decouple Work from Resources Better Quality of Service
15. 15 SCHEDULER ADVANTAGES 15 Bin Packing Over-Subscription Job Queueing Higher Resource Utilization Decouple Work from Resources Better Quality of Service
16. 16 SCHEDULER ADVANTAGES 16 Abstraction API Contracts Standardization Higher Resource Utilization Decouple Work from Resources Better Quality of Service
17. 17 SCHEDULER ADVANTAGES 17 Priorities Resource Isolation Pre-emption Higher Resource Utilization Decouple Work from Resources Better Quality of Service
18. 18 NOT A NEW CONCEPT 18
19. 19 BASED ON RESEARCH 19
20. NOMAD OVERVIEW
21. 21 NOMAD DESIGN PRINCIPLES 21 HashiCorp confidential do not distribute Integrated scheduler and cluster manager Distributed, shared state, optimistically concurrent Agent-based, client/server No dependencies
22. 22 NOMAD CHARACTERISTICS 22 HashiCorp confidential do not distribute Multi-datacenter and multi-region Highly performant and highly available Hybrid workloads with multiple schedulers and drivers Seamlessly integrates with HashiCorp ecosystem
23. NOMAD FUNDAMENTALS
24. 24 SINGLE REGION DEPLOYMENT 24 SERVER SERVER SERVER CLIENT CLIENT CLIENT DC1 DC2 DC3 FOLLOWER LEADER FOLLOWER REPLICATION FORWARDING REPLICATION FORWARDING RPC RPC RPC
25. 25 MULTI REGION DEPLOYMENT 25 SERVER SERVER SERVER FOLLOWER LEADER FOLLOWER REPLICATION FORWARDING REPLICATION REGION B GOSSIP REPLICATION REPLICATION FORWARDING REGION FORWARDING REGION A SERVER FOLLOWER SERVER SERVER LEADER FOLLOWER
26. 26 SERVER ARCHITECTURE 26 Omega Class Scheduler Pluggable Logic Internal Coordination and State Multi-Region / Multi-Datacenter
27. 27 CLIENT ARCHITECTURE 27 Broad OS Support Host Fingerprinting Pluggable Drivers Job restarts and lifecycle management
28. 28 CLIENT DRIVERS 28 Containerized Docker rkt Windows Server Containers Virtualized Qemu / KVM Hyper-V Xen Standalone Java Jar C# Static Binaries
29. 29 CLIENT FINGERPRINTING 29 Type Examples Operating System Kernel, OS, Version Hardware CPU, Memory, Disk Apps (Capabilities) Docker, Java, Consul Environment AWS, GCE
30. NOMAD JOB CONFIGURATION
31. 31 JOB FILE 31 Declarative Scheduler, driver, and resource needs Lifecycle behavior Constraints Versioned
32. 32 redis.nomad JOB FILE job "redis" { datacenters = ["us-east-1"] task "redis" { driver = "docker" config { image = "redis:v13" } resources { cpu = 500 # Mhz memory = 256 # MB network { mbits = 10 dynamic_ports = ["redis"] } } } }
33. 33 redis.nomad JOB FILE: TASK GROUPS job "app" { group "app" { task "redis" { # ... } task "app" { # ... } } }
34. 34 redis.nomad JOB FILE: CONSTRAINTS job "redis" { constraint { attribute = "${attr.kernel.version}" operator = "version" value = "> 3.19" } constraint { attribute = "${attr.platform.aws.instance-type}" value = "p2.16xlarge" } task "redis" { # ... } }
35. 35 redis.nomad JOB FILE: CONSUL SERVICE DISCOVERY job "redis" { task "redis" { # ... service { port = “redis” check { type = “tcp” interval = “10s” } } } }
36. 36 redis.nomad JOB FILE: CONSUL CONFIGURATION job "redis" { task "redis" { # ... template { data = <<EOH bind_port: {{ env "NOMAD_PORT_db" }} scratch_dir: {{ env "NOMAD_TASK_DIR" }} service_key: {{ key "service/my-key" }} EOH destination = "local/file.yml" } } }
37. 37 redis.nomad JOB FILE: VAULT INTEGRATION job "redis" { task "redis" { # ... template { data = <<EOH {{ with secret "secret/credentials" }} username: {{ .Data.username }} password: {{ .Data.password }} {{ end }} EOH destination = "local/file.yml" } } }
38. 38 redis.nomad JOB FILE: PARAMETERIZED job "encode" { type = "batch" parameterized { payload = "required" meta_required = ["s3-input", "s3-output", ...] } # ... task "ffmpeg" { driver = "exec" config { command = "ffmpeg" # When dispatched, the payload is written to a file that is then # read by the created task upon startup args = ["-config=${NOMAD_TASK_DIR}/config.json"] # ... }
39. 39 $ nomad job dispatch encode video-config.json $ $ cat video-config.json { "s3-input": "https://s3-us-west-1.com/video-bucket/cb31dabb1", "s3-output": "https://s3-us-west-1.com/video-bucket/a149adbe3", "input-codec": "mp4", "output-codec": "webm", "quality": "1080p" } Text JOB FILE: PARAMETERIZED
40. NOMAD MULTI-CLOUD
41. Why Multi-Cloud? • High Availability • Redundancy • Burstable Workload • Cloud Migration • Because we can
42. 42 CONSUL NOMAD SERVER LEADER SERVER FOLLOWER SERVER LEADER SERVER FOLLOWER SERVER FOLLOWER SERVER FOLLOWER NODE A NODE B GOOGLE CLOUD NATS CLOUD MESSAGING REPLICATION FORWARDING REPLICATION FORWARDING REPLICATION FORWARDING REPLICATION FORWARDING LOAD BALANCER LOAD BALANCER CONSUL NOMAD SERVER LEADER SERVER FOLLOWER SERVER LEADER SERVER FOLLOWER SERVER FOLLOWER SERVER FOLLOWER NODE A NODE B AWS REPLICATION FORWARDING REPLICATION FORWARDING REPLICATION FORWARDING REPLICATION FORWARDING LOAD BALANCER REGION FORWARDING (VPN) REGION FORWARDING (VPN)
43. NOMAD SCHEDULING
44. 44 SCHEDULING 44 Schedulers process evaluations and generate allocation plans. Placement is determined using the relevant scheduler. Scheduling involves feasibility checking and ranking. Feasibility filters out nodes missing necessary drivers and those failing the specified constraints. Ranking score feasible nodes to find the best fit (bin packing).
45. 45 SCHEDULER TYPES 45 HashiCorp confidential do not distribute Service Long-running applications and services Batch Short-lived data processing jobs (benefit from fast placement) System Lower level jobs that run on all clients (logging, monitoring)
46. 46 $ nomad plan example.nomad + Job: "example" + Task Group: "cache" (1 create) + Task: "redis" (forces create) Scheduler dry-run: - All tasks successfully allocated. $ Text SCHEDULING: PLAN
47. 47 $ nomad plan example.nomad.java + Job: "example" + Task Group: "web" (1 create) + Task: "tomcat" (forces create) Scheduler dry-run: - WARNING: Failed to place all allocations. Task Group "web" (failed to place 1 allocation): * Constraint "missing drivers" filtered 2 nodes $ Text SCHEDULING: PLAN
48. 48 $ nomad run example.nomad ==> Monitoring evaluation "4b8b7779" Evaluation triggered by job "example" Allocation "38720b8e" created: node "ec2f0830", group "cache" Evaluation status changed: "pending" -> "complete" ==> Evaluation "4b8b7779" finished with status "complete" $ Text SCHEDULING: RUN
49. 49 $ nomad run -region=gcp events.nomad ==> Monitoring evaluation "e2a8dfe6" !On branch master Evaluation triggered by job "events" !Your branch is up-to-date with 'origin/master'. Allocation "6615b39f" modified: node "0d6a6103", group "pubsub" !nothing to commit, working tree clean Evaluation status changed: "pending" -> "complete" ! ==> Evaluation "e2a8dfe6" finished with status "complete" $ Text SCHEDULING: RUN DIFFERENT REGION
50. 50 $ nomad status example ID = example Name = example Type = service Priority = 50 Datacenters = us-west-1 Status = running Summary Task Group Queued Starting Running Failed Complete Lost cache 0 0 1 0 0 0 Allocations ID Eval ID Node ID Task Group Desired Status Created At 38720b8e 4b8b7779 ec2f0830 cache run running 04/26/17 ... $ Text SCHEDULING: STATUS
51. DEMO!
52. Nomad Million Container Challenge 1,000 Jobs 1,000 Tasks per Job 5,000 Hosts on GCE 1,000,000 Containers
53. 53 MILLION CONTAINER CHALLENGE 53
54. 54 MILLION CONTAINER CHALLENGE 54 – Bill Gates 640 KB ought to be enough for anybody. “
55. 55 REAL WORLD SCALE 55 2nd Largest Hedge Fund 18K Cores 5 Hours 2,200 Containers/ second
56. Q/A AND HASHICONF
57. SEPTEMBER 18-20 AUSTIN, TEXAS www.hashiconf.com #hashiconf
58. #hashiconf Links: https://www.nomadproject.io https://github.com/nicholasjackson/terraform-nomad-multi-cloud