Nelson: Functional programming in system design (SBTB 2018)

Nelson: Functional programming in system design (SBTB 2018)

As functional programmers we work hard to keep our code compositional and modular. Unfortunately, these noble pursuits often stop with our code, the resulting program flung over the wall to be deployed in complex and prescriptive systems. In this talk we will look at Nelson, a deployment orchestration system that applies functional programming not only in its implementation, but also in its system behavior in managing the messy world of deployment infrastructure. Where Free algebras and streams allow Nelson to be extended to support different VCS platforms, schedulers, and health checkers, Nelson's emphasis on immutable deployments inform system-level decisions such as deployment workflows and service discovery. Having been run successfully at different companies under different configurations, Nelson serves as a prime example of how functional programming can help not just with the software we write, but also with the systems we maintain.

Fb8e986500c5059b2a6c0b2184bb0faf?s=128

Adelbert Chang

November 16, 2018
Tweet

Transcript

  1. Nelson Functional programming in system design Adelbert Chang @adelbertchang Lead

    Data Engineer @ Target Distinguished YAML
  2. https://youtu.be/t8KRo-DXnEo

  3. None
  4. None
  5. None
  6. None
  7. None
  8. None
  9. None
  10. None
  11. None
  12. None
  13. None
  14. Why are people scared of deployments?

  15. Why are people scared of deployments? Deployments are often manual,

    happen infrequently, and have cascading effects.
  16. B 5.4.5 D 2.1.8 F 1.2.3 G 1.0.9 C 2.3.6

    A 1.1.5 Mutable deployments E 4.12.7
  17. B 5.4.5 D 2.1.8 F 1.2.3 G 1.0.9 C 2.3.6

    A 1.1.5 Mutable deployments E 4.12.7 D 2.2.0
  18. B 5.4.5 D 2.2.0 F 1.2.3 G 1.0.9 C 2.3.6

    A 1.1.5 Mutable deployments E 4.12.7
  19. B 5.4.5 D 2.2.0 F 1.2.3 G 1.0.9 C 2.3.6

    A 1.1.5 Mutable deployments E 4.12.7
  20. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5

    E 4.12.7 D 2.1.8 B 5.4.5
  21. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5

    E 4.12.7 D 2.1.8 B 5.4.5 D 2.2.0
  22. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5

    E 4.12.7 D 2.1.8 B 5.4.5 D 2.2.0
  23. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5

    E 4.12.7 D 2.1.8 B 5.4.5 D 2.2.0 E 4.12.8
  24. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5

    D 2.1.8 B 5.4.5 D 2.2.0 E 4.12.8
  25. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5

    D 2.1.8 B 5.4.5 D 2.2.0 E 4.12.8 B 5.4.6
  26. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5

    D 2.1.8 B 5.4.5 D 2.2.0 E 4.12.8 B 5.4.6 A 1.1.6
  27. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments D 2.1.8

    B 5.4.5 D 2.2.0 E 4.12.8 B 5.4.6 A 1.1.6
  28. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments D 2.1.8

    D 2.2.0 E 4.12.8 B 5.4.6 A 1.1.6
  29. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments D 2.2.0

    E 4.12.8 B 5.4.6 A 1.1.6
  30. Reify the call graph B 5.4.5 D 2.1.8 F 1.2.3

    G 1.0.9 C 2.3.6 A 1.1.5 E 4.12.7
  31. Reify the call graph B 5.4.5 D 2.1.8 F 1.2.3

    G 1.0.9 C 2.3.6 A 1.1.5 E 4.12.7
  32. None
  33. Nelson: User's perspective

  34. .nelson.yml (checked-in) Nelson: User's perspective

  35. .nelson.yml (checked-in) Nelson: User's perspective What to deploy (logical)

  36. .nelson.yml (checked-in) Nelson: User's perspective What to deploy (logical) How

    to deploy
  37. .nelson.yml (checked-in) Nelson: User's perspective What to deploy (logical) How

    to deploy Where to deploy
  38. .nelson.yml (checked-in) Nelson: User's perspective What to deploy (logical) How

    to deploy foo.deployable.yml (per-deploy) Where to deploy
  39. .nelson.yml (checked-in) Nelson: User's perspective What to deploy (logical) How

    to deploy What to deploy (physical) foo.deployable.yml (per-deploy) Where to deploy
  40. Nelson: User's perspective Cloud image taken from https://en.wikipedia.org/wiki/File:Cartoon_cloud.svg

  41. Nelson: User's perspective 1. Make a deployment Cloud image taken

    from https://en.wikipedia.org/wiki/File:Cartoon_cloud.svg
  42. Nelson: User's perspective 1. Make a deployment Cloud image taken

    from https://en.wikipedia.org/wiki/File:Cartoon_cloud.svg 2. Webhook calls Nelson
  43. Nelson: User's perspective 1. Make a deployment Cloud image taken

    from https://en.wikipedia.org/wiki/File:Cartoon_cloud.svg 2. Webhook calls Nelson 3. Nelson fetches + parses manifest
  44. Nelson: User's perspective 1. Make a deployment Cloud image taken

    from https://en.wikipedia.org/wiki/File:Cartoon_cloud.svg 2. Webhook calls Nelson 3. Nelson fetches + parses manifest 4. Nelson deploys to scheduler + enters deployment into graph
  45. Nelson: User's perspective 1. Make a deployment Cloud image taken

    from https://en.wikipedia.org/wiki/File:Cartoon_cloud.svg 2. Webhook calls Nelson 3. Nelson fetches + parses manifest 4. Nelson deploys to scheduler + enters deployment into graph 5. User uses Nelson CLI and other tools to interact with deployment
  46. Nelson: User's perspective - CLI

  47. Nelson: User's perspective - CLI

  48. Nelson: User's perspective - CLI

  49. Nelson: User's perspective - CLI

  50. Nelson: Operator's perspective

  51. Nelson: Operator's perspective

  52. Nelson: Operator's perspective 1. Nelson receives + parses manifests

  53. Nelson: Operator's perspective 1. Nelson receives + parses manifests 2.

    Dynamically provision credentials
  54. Nelson: Operator's perspective 1. Nelson receives + parses manifests 2.

    Dynamically provision credentials 3. Write service discovery information
  55. Nelson: Operator's perspective 1. Nelson receives + parses manifests 2.

    Dynamically provision credentials 3. Write service discovery information 4. Deploy
  56. Nelson: Blueprints

  57. Nelson: Blueprints

  58. Nelson: Blueprints

  59. Nelson: Blueprints Manifest contains blueprint

  60. Nelson workflow

  61. Nelson workflow 1. Make a deployment

  62. Nelson workflow 1. Make a deployment 2. Webhook calls Nelson

  63. Nelson workflow 1. Make a deployment 2. Webhook calls Nelson

    3. Nelson fetches + parses manifest
  64. Nelson workflow 1. Make a deployment 2. Webhook calls Nelson

    3. Nelson fetches + parses manifest 4. Dynamically provision credentials
  65. Nelson workflow 1. Make a deployment 2. Webhook calls Nelson

    3. Nelson fetches + parses manifest 4. Dynamically provision credentials 5. Write service discovery information
  66. Nelson workflow 1. Make a deployment 2. Webhook calls Nelson

    3. Nelson fetches + parses manifest 4. Dynamically provision credentials 5. Write service discovery information 6. Deploy
  67. Nelson workflow 1. Make a deployment 2. Webhook calls Nelson

    3. Nelson fetches + parses manifest 4. Dynamically provision credentials 5. Write service discovery information 6. Deploy In the background: • User uses Nelson CLI and other tools to interact with deployment • Traffic shifting, health checks, cleanup ...
  68. Nelson: Roadmap • Kubernetes backend: no one-size-fits-all answer for service-to-service

    right now • Labeled services? Envoy sidecar? Istio? • Test deployments off of branches or pull requests • Actual protocols: Protocol Buffers and gRPC API • Nelson UI • "Getting Started" improvements
  69. https://getnelson.io/

  70. https://getnelson.io/ https://github.com/getnelson/ https://gitter.im/getnelson/nelson Find me for Nelson stickers! Questions? Tim

    Perrett and I are around today and tomorrow!