Save 37% off PRO during our Black Friday Sale! »

I Just Want to Ship My Code: Waypoint, Nomad, and Other Things

B598a16a9e9cc49bcfbfc771bfa0bbc1?s=47 Michael Lange
February 18, 2021

I Just Want to Ship My Code: Waypoint, Nomad, and Other Things

Follow a nodejs application through Kubernetes, Nomad, Docker, and Pkg. Maybe learn a thing about how to think about systems along the way!

B598a16a9e9cc49bcfbfc771bfa0bbc1?s=128

Michael Lange

February 18, 2021
Tweet

Transcript

  1. Hi, I’m Michael Lange. I work at HashiCorp. he/him

  2. None
  3. 😡🧊🌧

  4. None
  5. You may know me from the Nomad UI.

  6. I write JavaScript for the browser ▪ A high-level language

    ▪ The loosest types with judicious coercion ▪ No memory management ▪ No filesystem ▪ Computers are an implementation detail
  7. Copyright © 2021 HashiCorp I just want to ship my

    code. Waypoint, Nomad, and other things.
  8. Netlify Vercel Just push your code and receive a website.

    A+ dev experience
  9. Black box What happens inside? None of our business.

  10. Black box ▪ Data plane (customer sites) ▪ Control plane

    ▪ Some sorta scheduler ▪ Lotsa DNS for static files ▪ CDN, file propagation ▪ Cgi-bin? Lol? ▪ A bunch of engineers of all varieties
  11. That’s just business.

  12. What happens when you have your own internal platform?

  13. An extreme example ✨ externalities ✨

  14. Consider the whole system

  15. Ops Friendly

  16. It’s possible to go too far the other way.

  17. Enter Waypoint Consistent, dev-friendly workflows decoupled from platforms.

  18. With Waypoint, you get a single config file CODE EDITOR

    project = "example-nodejs" app "example-nodejs" { labels = { "service" = "example-nodejs", "env" = "dev" } build { use "pack" {} registry { use "docker" { image = "example-nodejs" tag = "1" local = true } } } deploy { use "kubernetes" { probe_path = "/" } } release { use "kubernetes" { } } }
  19. Demo One NodeJS + Kubernetes

  20. What you didn’t see is what’s interesting 1. No Dockerfile

    or docker-compose.yml 2. No ReplicaSet 3. No Deployment 4. No Service 5. No kubectl 6. No YAML
  21. The Power of Abstractions 💪 Build Registry Deploy Release package.json

    *.js *.ejs *.png Source Code Buildpacks Container Image Hub/ECR/GCR ReplicaSet Service kubectl apply Deployment
  22. Let’s Re-platform! Nomad Kubernetes

  23. Before… Build Registry Deploy Release package.json *.js *.ejs *.png

  24. …After Build Registry Deploy Release package.json *.js *.ejs *.png HCL

    Built in to service jobs
  25. And the Waypoint File CODE EDITOR project = "example-nodejs" app

    "example-nodejs" { build { use "pack" {} registry { use "docker" { image = "nodejs-example" tag = "1" local = true } } } deploy { use "nomad" { datacenter = "dc1" } } }
  26. And the Waypoint File CODE EDITOR project = "example-nodejs" app

    "example-nodejs" { build { use "pack" {} registry { use "docker" { image = "nodejs-example" tag = "1" local = true } } } deploy { use "nomad" { datacenter = "dc1" } } } project = "example-nodejs" app "example-nodejs" { labels = { "service" = "example-nodejs", "env" = "dev" } build { use "pack" {} registry { use "docker" { image = "example-nodejs" tag = "1" local = true } } } deploy { use "kubernetes" { probe_path = "/" } } release { use "kubernetes" { } } }
  27. Demo Two NodeJS + Nomad

  28. Takeaways From the perspective of a product engineer 1. No

    workflow change!!! 1. waypoint init && waypoint up 2. Very little code change 3. Minor conceptual changes to output 4. Both Kubernetes and Nomad are built for common abstractions
  29. Bigger Picture Abstractions All The Way Down

  30. Deprecated? How could this be???

  31. Anatomy of Docker From https://docs.docker.com/get-started/overview/

  32. The Power of Abstractions!! From https://docs.docker.com/get-started/overview/ Container Runtime Interface

  33. Want to go deeper? From https://containerd.io/ :)

  34. Wait a second… From https://containerd.io/ >:(

  35. Copyright © 2021 HashiCorp I just want to ship my

    code. Waypoint, Nomad, and other things.
  36. What else can be removed? Yertle the Turtle by Dr.

    Seuss
  37. What will it take? Yertle the Turtle by Dr. Seuss

  38. First, why containers? Workflows, not technologies 1. Immutability 2. Portability

    3. Isolation 4. Resource limits
  39. A look at Nomad’s Task Driver Model A loosely abstracted,

    unopinionated, pluggable runtime model Ranging from raw processes to VMs to containers. Many are builtin, many are community owned.
  40. The Nomad exec driver 1. Builds a chroot 2. And

    a cgroup 3. Configurable resource limits 4. All on demand
  41. Immutability Portability Isolation Resource limits

  42. Immutability Portability Isolation Resource limits

  43. How do we get immutability?

  44. How do we get immutability? We make a binary! https://github.com/vercel/pkg

  45. But what about non- executable files? Annotate your package.json file

    to bake files into your binary. CODE EDITOR { "name": "node-js-getting-started", "version": "0.3.0", "description": "A sample Node.js app using Express 4", "engines": { "node": "12.x" }, "main": "index.js", "scripts": { "start": "node index.js", "test": "node test.js" }, "dependencies": { "directory-tree": "^2.2.5", "ejs": "^2.5.6", "express": "^4.15.2" }, "devDependencies": { "got": "^11.3.0", "tape": "^4.7.0" }, "keywords": [ "node", "express" ], "license": "MIT", "bin": "index.js", "pkg": { "assets": [ "views/**/*", "public/**/*" ] } }
  46. Immutability Portability Isolation Resource limits

  47. File Servers are tried and true.

  48. Immutability Portability Isolation Resource limits

  49. Putting it all together ▪ Buildpacks (Dockerfile) ▪ Container Registry

    ▪ docker driver pkg (binary) File server (probably S3) exec Driver
  50. Demo Three NodeJS + Pkg + Nomad

  51. Nomad with Docker Build Registry Deploy Release package.json *.js *.ejs

    *.png HCL Built in to service jobs
  52. The Power of ABSTRACTIONS!!! 💪 Build Registry Deploy Release package.json

    *.js *.ejs *.png HCL Built in to service jobs pkg BIN :)
  53. No workflow change.

  54. Docker Pkg Let’s review some stats Build Time 55.50s Artifact

    Size 1075MB Build Time 4.67s Artifact Size 46MB
  55. Safety NOT Guaranteed ▪ Not ready for production ▪ Check

    out the code though! https://github.com/dingoeatingfuzz/waypoint-plugin-pkg
  56. The moral of the story 1. Consider the whole system

    2. Well defined interfaces help us all 3. Abstractions are all around us 4. Challenge those abstractions from time to time Make time and space for this R&D
  57. None