Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Nelson: Functional programming in system design (SBTB 2018)

Nelson: Functional programming in system design (SBTB 2018)

As functional programmers we work hard to keep our code compositional and modular. Unfortunately, these noble pursuits often stop with our code, the resulting program flung over the wall to be deployed in complex and prescriptive systems. In this talk we will look at Nelson, a deployment orchestration system that applies functional programming not only in its implementation, but also in its system behavior in managing the messy world of deployment infrastructure. Where Free algebras and streams allow Nelson to be extended to support different VCS platforms, schedulers, and health checkers, Nelson's emphasis on immutable deployments inform system-level decisions such as deployment workflows and service discovery. Having been run successfully at different companies under different configurations, Nelson serves as a prime example of how functional programming can help not just with the software we write, but also with the systems we maintain.

Adelbert Chang

November 16, 2018
Tweet

More Decks by Adelbert Chang

Other Decks in Programming

Transcript

  1. Why are people scared of deployments? Deployments are often manual,

    happen infrequently, and have cascading effects.
  2. B 5.4.5 D 2.1.8 F 1.2.3 G 1.0.9 C 2.3.6

    A 1.1.5 Mutable deployments E 4.12.7
  3. B 5.4.5 D 2.1.8 F 1.2.3 G 1.0.9 C 2.3.6

    A 1.1.5 Mutable deployments E 4.12.7 D 2.2.0
  4. B 5.4.5 D 2.2.0 F 1.2.3 G 1.0.9 C 2.3.6

    A 1.1.5 Mutable deployments E 4.12.7
  5. B 5.4.5 D 2.2.0 F 1.2.3 G 1.0.9 C 2.3.6

    A 1.1.5 Mutable deployments E 4.12.7
  6. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5

    E 4.12.7 D 2.1.8 B 5.4.5 D 2.2.0 E 4.12.8
  7. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5

    D 2.1.8 B 5.4.5 D 2.2.0 E 4.12.8 B 5.4.6
  8. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5

    D 2.1.8 B 5.4.5 D 2.2.0 E 4.12.8 B 5.4.6 A 1.1.6
  9. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments D 2.1.8

    B 5.4.5 D 2.2.0 E 4.12.8 B 5.4.6 A 1.1.6
  10. Reify the call graph B 5.4.5 D 2.1.8 F 1.2.3

    G 1.0.9 C 2.3.6 A 1.1.5 E 4.12.7
  11. Reify the call graph B 5.4.5 D 2.1.8 F 1.2.3

    G 1.0.9 C 2.3.6 A 1.1.5 E 4.12.7
  12. .nelson.yml (checked-in) Nelson: User's perspective What to deploy (logical) How

    to deploy foo.deployable.yml (per-deploy) Where to deploy
  13. .nelson.yml (checked-in) Nelson: User's perspective What to deploy (logical) How

    to deploy What to deploy (physical) foo.deployable.yml (per-deploy) Where to deploy
  14. Nelson: User's perspective 1. Make a deployment Cloud image taken

    from https://en.wikipedia.org/wiki/File:Cartoon_cloud.svg
  15. Nelson: User's perspective 1. Make a deployment Cloud image taken

    from https://en.wikipedia.org/wiki/File:Cartoon_cloud.svg 2. Webhook calls Nelson
  16. Nelson: User's perspective 1. Make a deployment Cloud image taken

    from https://en.wikipedia.org/wiki/File:Cartoon_cloud.svg 2. Webhook calls Nelson 3. Nelson fetches + parses manifest
  17. Nelson: User's perspective 1. Make a deployment Cloud image taken

    from https://en.wikipedia.org/wiki/File:Cartoon_cloud.svg 2. Webhook calls Nelson 3. Nelson fetches + parses manifest 4. Nelson deploys to scheduler + enters deployment into graph
  18. Nelson: User's perspective 1. Make a deployment Cloud image taken

    from https://en.wikipedia.org/wiki/File:Cartoon_cloud.svg 2. Webhook calls Nelson 3. Nelson fetches + parses manifest 4. Nelson deploys to scheduler + enters deployment into graph 5. User uses Nelson CLI and other tools to interact with deployment
  19. Nelson: Operator's perspective 1. Nelson receives + parses manifests 2.

    Dynamically provision credentials 3. Write service discovery information
  20. Nelson: Operator's perspective 1. Nelson receives + parses manifests 2.

    Dynamically provision credentials 3. Write service discovery information 4. Deploy
  21. Nelson workflow 1. Make a deployment 2. Webhook calls Nelson

    3. Nelson fetches + parses manifest 4. Dynamically provision credentials
  22. Nelson workflow 1. Make a deployment 2. Webhook calls Nelson

    3. Nelson fetches + parses manifest 4. Dynamically provision credentials 5. Write service discovery information
  23. Nelson workflow 1. Make a deployment 2. Webhook calls Nelson

    3. Nelson fetches + parses manifest 4. Dynamically provision credentials 5. Write service discovery information 6. Deploy
  24. Nelson workflow 1. Make a deployment 2. Webhook calls Nelson

    3. Nelson fetches + parses manifest 4. Dynamically provision credentials 5. Write service discovery information 6. Deploy In the background: • User uses Nelson CLI and other tools to interact with deployment • Traffic shifting, health checks, cleanup ...
  25. Nelson: Roadmap • Kubernetes backend: no one-size-fits-all answer for service-to-service

    right now • Labeled services? Envoy sidecar? Istio? • Test deployments off of branches or pull requests • Actual protocols: Protocol Buffers and gRPC API • Nelson UI • "Getting Started" improvements