Nelson: Functional programming in system design (SBTB 2018)

As functional programmers we work hard to keep our code compositional and modular. Unfortunately, these noble pursuits often stop with our code, the resulting program flung over the wall to be deployed in complex and prescriptive systems. In this talk we will look at Nelson, a deployment orchestration system that applies functional programming not only in its implementation, but also in its system behavior in managing the messy world of deployment infrastructure. Where Free algebras and streams allow Nelson to be extended to support different VCS platforms, schedulers, and health checkers, Nelson's emphasis on immutable deployments inform system-level decisions such as deployment workflows and service discovery. Having been run successfully at different companies under different configurations, Nelson serves as a prime example of how functional programming can help not just with the software we write, but also with the systems we maintain.

Transcript

1. Nelson Functional programming in system design Adelbert Chang @adelbertchang Lead Data Engineer @ Target Distinguished YAML
2. https://youtu.be/t8KRo-DXnEo
14. Why are people scared of deployments?
15. Why are people scared of deployments? Deployments are often manual, happen infrequently, and have cascading effects.
16. B 5.4.5 D 2.1.8 F 1.2.3 G 1.0.9 C 2.3.6 A 1.1.5 Mutable deployments E 4.12.7
17. B 5.4.5 D 2.1.8 F 1.2.3 G 1.0.9 C 2.3.6 A 1.1.5 Mutable deployments E 4.12.7 D 2.2.0
18. B 5.4.5 D 2.2.0 F 1.2.3 G 1.0.9 C 2.3.6 A 1.1.5 Mutable deployments E 4.12.7
19. B 5.4.5 D 2.2.0 F 1.2.3 G 1.0.9 C 2.3.6 A 1.1.5 Mutable deployments E 4.12.7
20. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5 E 4.12.7 D 2.1.8 B 5.4.5
21. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5 E 4.12.7 D 2.1.8 B 5.4.5 D 2.2.0
22. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5 E 4.12.7 D 2.1.8 B 5.4.5 D 2.2.0
23. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5 E 4.12.7 D 2.1.8 B 5.4.5 D 2.2.0 E 4.12.8
24. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5 D 2.1.8 B 5.4.5 D 2.2.0 E 4.12.8
25. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5 D 2.1.8 B 5.4.5 D 2.2.0 E 4.12.8 B 5.4.6
26. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5 D 2.1.8 B 5.4.5 D 2.2.0 E 4.12.8 B 5.4.6 A 1.1.6
27. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments D 2.1.8 B 5.4.5 D 2.2.0 E 4.12.8 B 5.4.6 A 1.1.6
28. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments D 2.1.8 D 2.2.0 E 4.12.8 B 5.4.6 A 1.1.6
29. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments D 2.2.0 E 4.12.8 B 5.4.6 A 1.1.6
30. Reify the call graph B 5.4.5 D 2.1.8 F 1.2.3 G 1.0.9 C 2.3.6 A 1.1.5 E 4.12.7
31. Reify the call graph B 5.4.5 D 2.1.8 F 1.2.3 G 1.0.9 C 2.3.6 A 1.1.5 E 4.12.7
33. Nelson: User's perspective
34. .nelson.yml (checked-in) Nelson: User's perspective
35. .nelson.yml (checked-in) Nelson: User's perspective What to deploy (logical)
36. .nelson.yml (checked-in) Nelson: User's perspective What to deploy (logical) How to deploy
37. .nelson.yml (checked-in) Nelson: User's perspective What to deploy (logical) How to deploy Where to deploy
38. .nelson.yml (checked-in) Nelson: User's perspective What to deploy (logical) How to deploy foo.deployable.yml (per-deploy) Where to deploy
39. .nelson.yml (checked-in) Nelson: User's perspective What to deploy (logical) How to deploy What to deploy (physical) foo.deployable.yml (per-deploy) Where to deploy
40. Nelson: User's perspective Cloud image taken from https://en.wikipedia.org/wiki/File:Cartoon_cloud.svg
41. Nelson: User's perspective 1. Make a deployment Cloud image taken from https://en.wikipedia.org/wiki/File:Cartoon_cloud.svg
42. Nelson: User's perspective 1. Make a deployment Cloud image taken from https://en.wikipedia.org/wiki/File:Cartoon_cloud.svg 2. Webhook calls Nelson
43. Nelson: User's perspective 1. Make a deployment Cloud image taken from https://en.wikipedia.org/wiki/File:Cartoon_cloud.svg 2. Webhook calls Nelson 3. Nelson fetches + parses manifest
44. Nelson: User's perspective 1. Make a deployment Cloud image taken from https://en.wikipedia.org/wiki/File:Cartoon_cloud.svg 2. Webhook calls Nelson 3. Nelson fetches + parses manifest 4. Nelson deploys to scheduler + enters deployment into graph
45. Nelson: User's perspective 1. Make a deployment Cloud image taken from https://en.wikipedia.org/wiki/File:Cartoon_cloud.svg 2. Webhook calls Nelson 3. Nelson fetches + parses manifest 4. Nelson deploys to scheduler + enters deployment into graph 5. User uses Nelson CLI and other tools to interact with deployment
46. Nelson: User's perspective - CLI
47. Nelson: User's perspective - CLI
48. Nelson: User's perspective - CLI
49. Nelson: User's perspective - CLI
50. Nelson: Operator's perspective
51. Nelson: Operator's perspective
52. Nelson: Operator's perspective 1. Nelson receives + parses manifests
53. Nelson: Operator's perspective 1. Nelson receives + parses manifests 2. Dynamically provision credentials
54. Nelson: Operator's perspective 1. Nelson receives + parses manifests 2. Dynamically provision credentials 3. Write service discovery information
55. Nelson: Operator's perspective 1. Nelson receives + parses manifests 2. Dynamically provision credentials 3. Write service discovery information 4. Deploy
56. Nelson: Blueprints
57. Nelson: Blueprints
58. Nelson: Blueprints
59. Nelson: Blueprints Manifest contains blueprint
60. Nelson workflow
61. Nelson workflow 1. Make a deployment
62. Nelson workflow 1. Make a deployment 2. Webhook calls Nelson
63. Nelson workflow 1. Make a deployment 2. Webhook calls Nelson 3. Nelson fetches + parses manifest
64. Nelson workflow 1. Make a deployment 2. Webhook calls Nelson 3. Nelson fetches + parses manifest 4. Dynamically provision credentials
65. Nelson workflow 1. Make a deployment 2. Webhook calls Nelson 3. Nelson fetches + parses manifest 4. Dynamically provision credentials 5. Write service discovery information
66. Nelson workflow 1. Make a deployment 2. Webhook calls Nelson 3. Nelson fetches + parses manifest 4. Dynamically provision credentials 5. Write service discovery information 6. Deploy
67. Nelson workflow 1. Make a deployment 2. Webhook calls Nelson 3. Nelson fetches + parses manifest 4. Dynamically provision credentials 5. Write service discovery information 6. Deploy In the background: • User uses Nelson CLI and other tools to interact with deployment • Traffic shifting, health checks, cleanup ...
68. Nelson: Roadmap • Kubernetes backend: no one-size-fits-all answer for service-to-service right now • Labeled services? Envoy sidecar? Istio? • Test deployments off of branches or pull requests • Actual protocols: Protocol Buffers and gRPC API • Nelson UI • "Getting Started" improvements
69. https://getnelson.io/
70. https://getnelson.io/ https://github.com/getnelson/ https://gitter.im/getnelson/nelson Find me for Nelson stickers! Questions? Tim Perrett and I are around today and tomorrow!