Nelson: Functional programming in system design

As functional programmers we work hard to keep things immutable and referentially transparent. However, these noble pursuits often stop with our text editors, the resulting code flung over the wall to be built and deployed in mutable, imperative systems. In this talk we will take a look at Nelson, a deployment orchestration system that applies functional programming not only in its implementation, but also in its system behavior in managing the messy world of deployment infrastructure. Where Free algebras and streams allow the code to be easily extended to support different source repositories, schedulers, and health checkers, Nelson's strict stance on immutable deployments inform important system level decisions such as deployment workflows and service discovery. Having now been run at a couple of companies under different configurations, Nelson serves as a prime example of how functional programming is for the real world and can influence not just code, but systems as well.

Transcript

1. Nelson Functional programming in system design Adelbert Chang @adelbertchang Target
7. B 5.4.5 D 2.1.8 F 1.2.3 G 1.0.9 C 2.3.6 A 1.1.5 Mutable deployments E 4.12.7
8. B 5.4.5 D 2.1.8 F 1.2.3 G 1.0.9 C 2.3.6 A 1.1.5 Mutable deployments E 4.12.7 D 2.2.0
9. B 5.4.5 D 2.2.0 F 1.2.3 G 1.0.9 C 2.3.6 A 1.1.5 Mutable deployments E 4.12.7
10. B 5.4.5 D 2.2.0 F 1.2.3 G 1.0.9 C 2.3.6 A 1.1.5 Mutable deployments E 4.12.7
11. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5 E 4.12.7 D 2.1.8 B 5.4.5
12. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5 E 4.12.7 D 2.1.8 B 5.4.5 D 2.2.0
13. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5 E 4.12.7 B 5.4.5 D 2.1.8 D 2.2.0
14. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5 E 4.12.7 B 5.4.5 D 2.1.8 D 2.2.0 E 4.12.8
15. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5 B 5.4.5 D 2.1.8 D 2.2.0 E 4.12.8
16. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5 B 5.4.5 D 2.1.8 D 2.2.0 E 4.12.8 B 5.4.6
17. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5 B 5.4.5 D 2.1.8 D 2.2.0 E 4.12.8 B 5.4.6 A 1.1.6
18. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments B 5.4.5 D 2.1.8 D 2.2.0 E 4.12.8 B 5.4.6 A 1.1.6
19. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments D 2.2.0 E 4.12.8 B 5.4.6 A 1.1.6
20. Reify the call graph B 5.4.5 D 2.1.8 F 1.2.3 G 1.0.9 C 2.3.6 A 1.1.5 E 4.12.7
21. Reify the call graph B 5.4.5 D 2.1.8 F 1.2.3 G 1.0.9 C 2.3.6 A 1.1.5 E 4.12.7
22. Reify the call graph B 5.4.5 D 2.1.8 F 1.2.3 G 1.0.9 C 2.3.6 A 1.1.5 E 4.12.7
23. Reify the call graph B 5.4.5 D 2.1.8 F 1.2.3 G 1.0.9 C 2.3.6 A 1.1.5 E 4.12.7
24. Reify the call graph B 5.4.5 D 2.1.8 F 1.2.3 G 1.0.9 C 2.3.6 A 1.1.5 E 4.12.7
25. Reify the call graph B 5.4.5 D 2.1.8 F 1.2.3 G 1.0.9 C 2.3.6 A 1.1.5 E 4.12.7
28. Name and version
29. Name and version Dependencies
30. Nelson workflow
31. Nelson workflow
32. Nelson workflow
33. Nelson workflow
34. Nelson workflow
35. Graph management: traffic shifting C 2.3.6 A 1.1.5 E 4.12.7 F 1.2.3 G 1.0.9 D 2.1.8 B 5.4.5
36. Graph management: traffic shifting C 2.3.6 A 1.1.5 E 4.12.7 F 1.2.3 G 1.0.9 D 2.1.8 B 5.4.5 C 2.3.7
37. Graph management: traffic shifting C 2.3.6 A 1.1.5 E 4.12.7 F 1.2.3 G 1.0.9 D 2.1.8 B 5.4.5 C 2.3.7 B 5.4.6
38. Graph management: traffic shifting C 2.3.6 A 1.1.5 E 4.12.7 F 1.2.3 G 1.0.9 D 2.1.8 B 5.4.5 C 2.3.7 B 5.4.6 A 1.1.6
39. Graph management: traffic shifting C 2.3.6 A 1.1.5 E 4.12.7 F 1.2.3 G 1.0.9 D 2.1.8 B 5.4.5 C 2.3.7
40. Graph management: traffic shifting C 2.3.6 A 1.1.5 E 4.12.7 F 1.2.3 G 1.0.9 D 2.1.8 B 5.4.5 C 2.3.7
41. Graph management: traffic shifting D 2.1.8 C 2.3.6 A 1.1.5 B 5.4.5 C 2.3.7 E 4.12.7 F 1.2.3 G 1.0.9
42. Graph management: traffic shifting D 2.1.8 A 1.1.5 B 5.4.5 C 2.3.7 E 4.12.7 F 1.2.3 G 1.0.9
43. Graph management: pruning A 1.1.5 E 4.12.7 F 1.2.3 G 1.0.9 D 2.1.8 B 5.4.5 C 2.3.7
44. Graph management: pruning A 1.1.5 E 4.12.7 F 1.2.3 G 1.0.9 D 2.1.8 B 5.4.5 C 2.3.7
45. Graph management: pruning A 1.1.5 E 4.12.7 F 1.2.3 G 1.0.9 D 2.1.8 B 5.4.5 F 1.3.0 C 2.3.7
46. Graph management: pruning A 1.1.5 E 4.12.7 F 1.2.3 G 1.0.9 D 2.1.8 B 5.4.5 F 1.3.0 E 4.12.8 C 2.3.7
47. Graph management: pruning A 1.1.5 F 1.2.3 G 1.0.9 D 2.1.8 B 5.4.5 F 1.3.0 E 4.12.8 C 2.3.7
48. Graph management: pruning A 1.1.5 D 2.1.8 B 5.4.5 F 1.3.0 E 4.12.8 C 2.3.7
49. D 2.1.8 A 1.1.5 F 1.3.0 E 4.12.8 C 2.3.7 B 5.4.5 Graph management: pruning
50. D 2.1.8 A 1.1.5 F 1.3.0 E 4.12.8 C 2.3.7 B 5.4.5 Graph management: pruning B 5.4.6
51. D 2.1.8 A 1.1.5 F 1.3.0 E 4.12.8 C 2.3.7 B 5.4.5 Graph management: pruning B 5.4.6
52. D 2.1.8 A 1.1.5 F 1.3.0 E 4.12.8 C 2.3.7 B 5.4.5 Graph management: pruning B 5.4.6
53. D 2.1.8 A 1.1.5 F 1.3.0 E 4.12.8 C 2.3.7 Graph management: pruning B 5.4.6
54. D 2.1.8 A 1.1.5 F 1.3.0 E 4.12.8 C 2.3.7 Graph management: pruning B 5.4.6
55. D 2.1.8 A 1.1.5 F 1.3.0 E 4.12.8 C 2.3.7 Job X 1.8.5 Graph management: pruning B 5.4.6
56. Time
57. Time retain-latest
58. X 3.2.0 Time retain-latest
59. X 3.2.0 X 3.2.1 Time retain-latest
60. X 3.2.0 X 3.2.1 Time retain-latest
61. X 3.2.0 X 3.2.1 X 3.3.0 Time retain-latest
62. X 3.2.0 X 3.2.1 X 3.3.0 Time retain-latest
63. X 3.2.0 X 3.2.1 X 3.3.0 X 4.0.0 Time retain-latest
64. X 3.2.0 X 3.2.1 X 3.3.0 X 4.0.0 Time retain-latest
65. X 3.2.0 X 3.2.1 X 3.3.0 X 4.0.0 X 4.1.0 Time retain-latest
66. X 3.2.0 X 3.2.1 X 3.3.0 X 4.0.0 X 4.1.0 Time retain-latest
67. X 3.2.0 X 3.2.1 X 3.3.0 X 4.0.0 X 4.1.0 Time retain-latest X 3.2.0 X 3.2.1 X 3.3.0 X 4.0.0 X 4.1.0 retain-latest-two-major
68. X 3.2.0 X 3.2.1 X 3.3.0 X 4.0.0 X 4.1.0 Time retain-latest X 3.2.0 X 3.2.1 X 3.3.0 X 4.0.0 X 4.1.0 retain-latest-two-major X 3.2.0 X 3.2.1 X 3.3.0 X 4.0.0 X 4.1.0 retain-latest-two-feature
69. D 2.1.8 A 1.1.5 F 1.2.4 E 4.12.8 C 2.3.7 Job X 1.8.5 Graph management: pruning B 5.4.6
70. D 2.1.8 A 1.1.5 F 1.2.4 E 4.12.8 C 2.3.7 Job X 1.8.5 Job X 1.9.0 Graph management: pruning B 5.4.6
71. D 2.1.8 A 1.1.5 F 1.2.4 E 4.12.8 C 2.3.7 Job X 1.8.5 Job X 1.9.0 Job X 1.8.6 Graph management: pruning B 5.4.6
72. D 2.1.8 A 1.1.5 F 1.2.4 E 4.12.8 C 2.3.7 Job X 1.9.0 Job X 1.8.6 Graph management: pruning B 5.4.6
73. D 2.1.8 A 1.1.5 F 1.2.4 E 4.12.8 C 2.3.7 Job X 1.9.0 Job X 1.8.6 Graph management: pruning B 5.4.6
74. D 2.1.8 A 1.1.5 F 1.2.4 E 4.12.8 C 2.3.7 Job X 1.9.0 Job X 1.8.6 Job X 1.9.1 Graph management: pruning B 5.4.6
75. D 2.1.8 A 1.1.5 F 1.2.4 E 4.12.8 C 2.3.7 Job X 1.8.6 Job X 1.9.1 Graph management: pruning B 5.4.6
77. for { _ <- status(id, Pending, "workflow about to start") i <- dockerOps(id, unit, dc.docker.registry) Launch workflow
78. for { _ <- status(id, Pending, "workflow about to start") i <- dockerOps(id, unit, dc.docker.registry) _ <- status(id, Deploying, s"writing alert definitions to ${dc.name}'s consul") _ <- writeAlertsToConsul(sn, ns.name, p.name, unit, p.environment.alertOptOuts) Launch workflow
79. for { _ <- status(id, Pending, "workflow about to start") i <- dockerOps(id, unit, dc.docker.registry) _ <- status(id, Deploying, s"writing alert definitions to ${dc.name}'s consul") _ <- writeAlertsToConsul(sn, ns.name, p.name, unit, p.environment.alertOptOuts) Launch workflow _ <- logToFile(...) _ <- writePolicyToVault(cfg = dc.policy, sn = sn, ns = ns.name, rs = rs)
80. for { _ <- status(id, Pending, "workflow about to start") i <- dockerOps(id, unit, dc.docker.registry) _ <- status(id, Deploying, s"writing alert definitions to ${dc.name}'s consul") _ <- writeAlertsToConsul(sn, ns.name, p.name, unit, p.environment.alertOptOuts) _ <- logToFile(...) _ <- writeDiscoveryToConsul(id, sn, ns.name, dc) Launch workflow _ <- logToFile(...) _ <- writePolicyToVault(cfg = dc.policy, sn = sn, ns = ns.name, rs = rs)
81. for { _ <- status(id, Pending, "workflow about to start") i <- dockerOps(id, unit, dc.docker.registry) _ <- status(id, Deploying, s"writing alert definitions to ${dc.name}'s consul") _ <- writeAlertsToConsul(sn, ns.name, p.name, unit, p.environment.alertOptOuts) _ <- logToFile(...) _ <- writeDiscoveryToConsul(id, sn, ns.name, dc) _ <- getTrafficShift.cata(...) _ <- logToFile(...) Launch workflow _ <- logToFile(...) _ <- writePolicyToVault(cfg = dc.policy, sn = sn, ns = ns.name, rs = rs)
82. for { _ <- status(id, Pending, "workflow about to start") i <- dockerOps(id, unit, dc.docker.registry) _ <- status(id, Deploying, s"writing alert definitions to ${dc.name}'s consul") _ <- writeAlertsToConsul(sn, ns.name, p.name, unit, p.environment.alertOptOuts) _ <- logToFile(...) _ <- writeDiscoveryToConsul(id, sn, ns.name, dc) _ <- getTrafficShift.cata(...) _ <- logToFile(...) Launch workflow _ <- logToFile(...) _ <- writePolicyToVault(cfg = dc.policy, sn = sn, ns = ns.name, rs = rs) l <- launch(i, dc, ns.name, vunit, p, hash) _ <- debug(s"response from scheduler $l")
83. for { _ <- status(id, Pending, "workflow about to start") i <- dockerOps(id, unit, dc.docker.registry) _ <- status(id, Deploying, s"writing alert definitions to ${dc.name}'s consul") _ <- writeAlertsToConsul(sn, ns.name, p.name, unit, p.environment.alertOptOuts) _ <- logToFile(...) _ <- writeDiscoveryToConsul(id, sn, ns.name, dc) _ <- getTrafficShift.cata(...) _ <- logToFile(...) _ <- status(id, getStatus(unit, p), "======> workflow completed <======") } yield () Launch workflow _ <- logToFile(...) _ <- writePolicyToVault(cfg = dc.policy, sn = sn, ns = ns.name, rs = rs) l <- launch(i, dc, ns.name, vunit, p, hash) _ <- debug(s"response from scheduler $l")
84. logToFile(d.id, s"removing policy from vault: ${vaultLoggingFields(...)}") >> deletePolicyFromVault(d.stackName, ns.name) >> Delete workflow
85. logToFile(d.id, s"removing policy from vault: ${vaultLoggingFields(...)}") >> deletePolicyFromVault(d.stackName, ns.name) >> Delete workflow logToFile(d.id, s"removing alerts from consul ${alerts.alertingKey(sn)}") >> deleteAlertsFromConsul(d.stackName) >>
86. logToFile(d.id, s"removing policy from vault: ${vaultLoggingFields(...)}") >> deletePolicyFromVault(d.stackName, ns.name) >> Delete workflow logToFile(d.id, ...) >> deleteDiscoveryInfoFromConsul(sn) >> logToFile(d.id, s"removing alerts from consul ${alerts.alertingKey(sn)}") >> deleteAlertsFromConsul(d.stackName) >>
87. logToFile(d.id, s"removing policy from vault: ${vaultLoggingFields(...)}") >> deletePolicyFromVault(d.stackName, ns.name) >> Delete workflow logToFile(d.id, ...) >> deleteDiscoveryInfoFromConsul(sn) >> logToFile(d.id, s"removing alerts from consul ${alerts.alertingKey(sn)}") >> deleteAlertsFromConsul(d.stackName) >> logToFile(d.id, s"instructing ${dc.name}'s scheduler to decommission ${sn}") >> delete(dc,d) >>
88. logToFile(d.id, s"removing policy from vault: ${vaultLoggingFields(...)}") >> deletePolicyFromVault(d.stackName, ns.name) >> Delete workflow logToFile(d.id, ...) >> deleteDiscoveryInfoFromConsul(sn) >> logToFile(d.id, s"removing alerts from consul ${alerts.alertingKey(sn)}") >> deleteAlertsFromConsul(d.stackName) >> logToFile(d.id, s"instructing ${dc.name}'s scheduler to decommission ${sn}") >> delete(dc,d) >> status(d.id, Terminated, s"Decommissioning deployment ${sn} in ${dc.name}")
89. runBackgroundJob("auditor", cfg.auditor.process(cfg.storage)) Background processes
90. runBackgroundJob("auditor", cfg.auditor.process(cfg.storage)) runBackgroundJob("pipeline_processor", Process.eval(Pipeline.task(cfg)(Pipeline.sinks.runAction(cfg)))) Background processes
91. runBackgroundJob("auditor", cfg.auditor.process(cfg.storage)) runBackgroundJob("pipeline_processor", Process.eval(Pipeline.task(cfg)(Pipeline.sinks.runAction(cfg)))) runBackgroundJob("workflow_logger", cfg.workflowLogger.process) Background processes
92. runBackgroundJob("auditor", cfg.auditor.process(cfg.storage)) runBackgroundJob("pipeline_processor", Process.eval(Pipeline.task(cfg)(Pipeline.sinks.runAction(cfg)))) runBackgroundJob("workflow_logger", cfg.workflowLogger.process) runBackgroundJob("routing_cron", routing.cron.consulRefresh(cfg) to Http4sConsul.consulSink) Background processes
93. runBackgroundJob("auditor", cfg.auditor.process(cfg.storage)) runBackgroundJob("pipeline_processor", Process.eval(Pipeline.task(cfg)(Pipeline.sinks.runAction(cfg)))) runBackgroundJob("workflow_logger", cfg.workflowLogger.process) runBackgroundJob("routing_cron", routing.cron.consulRefresh(cfg) to Http4sConsul.consulSink) runBackgroundJob("cleanup_pipeline", cleanup.CleanupCron.pipeline(cfg)) Background processes
94. runBackgroundJob("auditor", cfg.auditor.process(cfg.storage)) runBackgroundJob("pipeline_processor", Process.eval(Pipeline.task(cfg)(Pipeline.sinks.runAction(cfg)))) runBackgroundJob("workflow_logger", cfg.workflowLogger.process) runBackgroundJob("routing_cron", routing.cron.consulRefresh(cfg) to Http4sConsul.consulSink) runBackgroundJob("cleanup_pipeline", cleanup.CleanupCron.pipeline(cfg)) runBackgroundJob("sweeper", cleanup.Sweeper.process(cfg)) Background processes
95. runBackgroundJob("auditor", cfg.auditor.process(cfg.storage)) runBackgroundJob("pipeline_processor", Process.eval(Pipeline.task(cfg)(Pipeline.sinks.runAction(cfg)))) runBackgroundJob("workflow_logger", cfg.workflowLogger.process) runBackgroundJob("routing_cron", routing.cron.consulRefresh(cfg) to Http4sConsul.consulSink) runBackgroundJob("cleanup_pipeline", cleanup.CleanupCron.pipeline(cfg)) runBackgroundJob("sweeper", cleanup.Sweeper.process(cfg)) runBackgroundJob("deployment_monitor", DeploymentMonitor.loop(cfg)) Background processes
96. type Op0[A] = Coproduct[DockerOp, ConsulOp, A] type Op1[A] = Coproduct[LoggingOp, Op0,A] type Op2[A] = Coproduct[StoreOp, Op1, A] type Op3[A] = Coproduct[WorkflowControlOp, Op2,A] type Op4[A] = Coproduct[Vault, Op3, A] type WorkflowOp[A] = Coproduct[SchedulerOp, Op4, A] type WorkflowF[A] = Free.FreeC[WorkflowOp, A] Workflow algebra
97. final case class Interpreters( scheduler: SchedulerOp ~> Task, consul: ConsulOp ~> Task, vault: Vault ~> Task, storage: StoreOp ~> Task, logger: LoggingOp ~> Task, docker: DockerOp ~> Task, control: WorkflowControlOp ~> Task, health: HealthCheckOp ~> Task ) Workflow interpreter(s)
108. https://getnelson.github.io/ https://github.com/getnelson/ https://gitter.im/getnelson/nelson