Nelson: Functional programming in system design

Nelson: Functional programming in system design

As functional programmers we work hard to keep things immutable and referentially transparent. However, these noble pursuits often stop with our text editors, the resulting code flung over the wall to be built and deployed in mutable, imperative systems. In this talk we will take a look at Nelson, a deployment orchestration system that applies functional programming not only in its implementation, but also in its system behavior in managing the messy world of deployment infrastructure. Where Free algebras and streams allow the code to be easily extended to support different source repositories, schedulers, and health checkers, Nelson's strict stance on immutable deployments inform important system level decisions such as deployment workflows and service discovery. Having now been run at a couple of companies under different configurations, Nelson serves as a prime example of how functional programming is for the real world and can influence not just code, but systems as well.

Fb8e986500c5059b2a6c0b2184bb0faf?s=128

Adelbert Chang

March 19, 2018
Tweet

Transcript

  1. Nelson Functional programming in system design Adelbert Chang @adelbertchang Target

  2. None
  3. None
  4. None
  5. None
  6. None
  7. B 5.4.5 D 2.1.8 F 1.2.3 G 1.0.9 C 2.3.6

    A 1.1.5 Mutable deployments E 4.12.7
  8. B 5.4.5 D 2.1.8 F 1.2.3 G 1.0.9 C 2.3.6

    A 1.1.5 Mutable deployments E 4.12.7 D 2.2.0
  9. B 5.4.5 D 2.2.0 F 1.2.3 G 1.0.9 C 2.3.6

    A 1.1.5 Mutable deployments E 4.12.7
  10. B 5.4.5 D 2.2.0 F 1.2.3 G 1.0.9 C 2.3.6

    A 1.1.5 Mutable deployments E 4.12.7
  11. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5

    E 4.12.7 D 2.1.8 B 5.4.5
  12. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5

    E 4.12.7 D 2.1.8 B 5.4.5 D 2.2.0
  13. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5

    E 4.12.7 B 5.4.5 D 2.1.8 D 2.2.0
  14. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5

    E 4.12.7 B 5.4.5 D 2.1.8 D 2.2.0 E 4.12.8
  15. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5

    B 5.4.5 D 2.1.8 D 2.2.0 E 4.12.8
  16. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5

    B 5.4.5 D 2.1.8 D 2.2.0 E 4.12.8 B 5.4.6
  17. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5

    B 5.4.5 D 2.1.8 D 2.2.0 E 4.12.8 B 5.4.6 A 1.1.6
  18. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments B 5.4.5

    D 2.1.8 D 2.2.0 E 4.12.8 B 5.4.6 A 1.1.6
  19. F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments D 2.2.0

    E 4.12.8 B 5.4.6 A 1.1.6
  20. Reify the call graph B 5.4.5 D 2.1.8 F 1.2.3

    G 1.0.9 C 2.3.6 A 1.1.5 E 4.12.7
  21. Reify the call graph B 5.4.5 D 2.1.8 F 1.2.3

    G 1.0.9 C 2.3.6 A 1.1.5 E 4.12.7
  22. Reify the call graph B 5.4.5 D 2.1.8 F 1.2.3

    G 1.0.9 C 2.3.6 A 1.1.5 E 4.12.7
  23. Reify the call graph B 5.4.5 D 2.1.8 F 1.2.3

    G 1.0.9 C 2.3.6 A 1.1.5 E 4.12.7
  24. Reify the call graph B 5.4.5 D 2.1.8 F 1.2.3

    G 1.0.9 C 2.3.6 A 1.1.5 E 4.12.7
  25. Reify the call graph B 5.4.5 D 2.1.8 F 1.2.3

    G 1.0.9 C 2.3.6 A 1.1.5 E 4.12.7
  26. None
  27. None
  28. Name and version

  29. Name and version Dependencies

  30. Nelson workflow

  31. Nelson workflow

  32. Nelson workflow

  33. Nelson workflow

  34. Nelson workflow

  35. Graph management: traffic shifting C 2.3.6 A 1.1.5 E 4.12.7

    F 1.2.3 G 1.0.9 D 2.1.8 B 5.4.5
  36. Graph management: traffic shifting C 2.3.6 A 1.1.5 E 4.12.7

    F 1.2.3 G 1.0.9 D 2.1.8 B 5.4.5 C 2.3.7
  37. Graph management: traffic shifting C 2.3.6 A 1.1.5 E 4.12.7

    F 1.2.3 G 1.0.9 D 2.1.8 B 5.4.5 C 2.3.7 B 5.4.6
  38. Graph management: traffic shifting C 2.3.6 A 1.1.5 E 4.12.7

    F 1.2.3 G 1.0.9 D 2.1.8 B 5.4.5 C 2.3.7 B 5.4.6 A 1.1.6
  39. Graph management: traffic shifting C 2.3.6 A 1.1.5 E 4.12.7

    F 1.2.3 G 1.0.9 D 2.1.8 B 5.4.5 C 2.3.7
  40. Graph management: traffic shifting C 2.3.6 A 1.1.5 E 4.12.7

    F 1.2.3 G 1.0.9 D 2.1.8 B 5.4.5 C 2.3.7
  41. Graph management: traffic shifting D 2.1.8 C 2.3.6 A 1.1.5

    B 5.4.5 C 2.3.7 E 4.12.7 F 1.2.3 G 1.0.9
  42. Graph management: traffic shifting D 2.1.8 A 1.1.5 B 5.4.5

    C 2.3.7 E 4.12.7 F 1.2.3 G 1.0.9
  43. Graph management: pruning A 1.1.5 E 4.12.7 F 1.2.3 G

    1.0.9 D 2.1.8 B 5.4.5 C 2.3.7
  44. Graph management: pruning A 1.1.5 E 4.12.7 F 1.2.3 G

    1.0.9 D 2.1.8 B 5.4.5 C 2.3.7
  45. Graph management: pruning A 1.1.5 E 4.12.7 F 1.2.3 G

    1.0.9 D 2.1.8 B 5.4.5 F 1.3.0 C 2.3.7
  46. Graph management: pruning A 1.1.5 E 4.12.7 F 1.2.3 G

    1.0.9 D 2.1.8 B 5.4.5 F 1.3.0 E 4.12.8 C 2.3.7
  47. Graph management: pruning A 1.1.5 F 1.2.3 G 1.0.9 D

    2.1.8 B 5.4.5 F 1.3.0 E 4.12.8 C 2.3.7
  48. Graph management: pruning A 1.1.5 D 2.1.8 B 5.4.5 F

    1.3.0 E 4.12.8 C 2.3.7
  49. D 2.1.8 A 1.1.5 F 1.3.0 E 4.12.8 C 2.3.7

    B 5.4.5 Graph management: pruning
  50. D 2.1.8 A 1.1.5 F 1.3.0 E 4.12.8 C 2.3.7

    B 5.4.5 Graph management: pruning B 5.4.6
  51. D 2.1.8 A 1.1.5 F 1.3.0 E 4.12.8 C 2.3.7

    B 5.4.5 Graph management: pruning B 5.4.6
  52. D 2.1.8 A 1.1.5 F 1.3.0 E 4.12.8 C 2.3.7

    B 5.4.5 Graph management: pruning B 5.4.6
  53. D 2.1.8 A 1.1.5 F 1.3.0 E 4.12.8 C 2.3.7

    Graph management: pruning B 5.4.6
  54. D 2.1.8 A 1.1.5 F 1.3.0 E 4.12.8 C 2.3.7

    Graph management: pruning B 5.4.6
  55. D 2.1.8 A 1.1.5 F 1.3.0 E 4.12.8 C 2.3.7

    Job X 1.8.5 Graph management: pruning B 5.4.6
  56. Time

  57. Time retain-latest

  58. X 3.2.0 Time retain-latest

  59. X 3.2.0 X 3.2.1 Time retain-latest

  60. X 3.2.0 X 3.2.1 Time retain-latest

  61. X 3.2.0 X 3.2.1 X 3.3.0 Time retain-latest

  62. X 3.2.0 X 3.2.1 X 3.3.0 Time retain-latest

  63. X 3.2.0 X 3.2.1 X 3.3.0 X 4.0.0 Time retain-latest

  64. X 3.2.0 X 3.2.1 X 3.3.0 X 4.0.0 Time retain-latest

  65. X 3.2.0 X 3.2.1 X 3.3.0 X 4.0.0 X 4.1.0

    Time retain-latest
  66. X 3.2.0 X 3.2.1 X 3.3.0 X 4.0.0 X 4.1.0

    Time retain-latest
  67. X 3.2.0 X 3.2.1 X 3.3.0 X 4.0.0 X 4.1.0

    Time retain-latest X 3.2.0 X 3.2.1 X 3.3.0 X 4.0.0 X 4.1.0 retain-latest-two-major
  68. X 3.2.0 X 3.2.1 X 3.3.0 X 4.0.0 X 4.1.0

    Time retain-latest X 3.2.0 X 3.2.1 X 3.3.0 X 4.0.0 X 4.1.0 retain-latest-two-major X 3.2.0 X 3.2.1 X 3.3.0 X 4.0.0 X 4.1.0 retain-latest-two-feature
  69. D 2.1.8 A 1.1.5 F 1.2.4 E 4.12.8 C 2.3.7

    Job X 1.8.5 Graph management: pruning B 5.4.6
  70. D 2.1.8 A 1.1.5 F 1.2.4 E 4.12.8 C 2.3.7

    Job X 1.8.5 Job X 1.9.0 Graph management: pruning B 5.4.6
  71. D 2.1.8 A 1.1.5 F 1.2.4 E 4.12.8 C 2.3.7

    Job X 1.8.5 Job X 1.9.0 Job X 1.8.6 Graph management: pruning B 5.4.6
  72. D 2.1.8 A 1.1.5 F 1.2.4 E 4.12.8 C 2.3.7

    Job X 1.9.0 Job X 1.8.6 Graph management: pruning B 5.4.6
  73. D 2.1.8 A 1.1.5 F 1.2.4 E 4.12.8 C 2.3.7

    Job X 1.9.0 Job X 1.8.6 Graph management: pruning B 5.4.6
  74. D 2.1.8 A 1.1.5 F 1.2.4 E 4.12.8 C 2.3.7

    Job X 1.9.0 Job X 1.8.6 Job X 1.9.1 Graph management: pruning B 5.4.6
  75. D 2.1.8 A 1.1.5 F 1.2.4 E 4.12.8 C 2.3.7

    Job X 1.8.6 Job X 1.9.1 Graph management: pruning B 5.4.6
  76. None
  77. for { _ <- status(id, Pending, "workflow about to start")

    i <- dockerOps(id, unit, dc.docker.registry) Launch workflow
  78. for { _ <- status(id, Pending, "workflow about to start")

    i <- dockerOps(id, unit, dc.docker.registry) _ <- status(id, Deploying, s"writing alert definitions to ${dc.name}'s consul") _ <- writeAlertsToConsul(sn, ns.name, p.name, unit, p.environment.alertOptOuts) Launch workflow
  79. for { _ <- status(id, Pending, "workflow about to start")

    i <- dockerOps(id, unit, dc.docker.registry) _ <- status(id, Deploying, s"writing alert definitions to ${dc.name}'s consul") _ <- writeAlertsToConsul(sn, ns.name, p.name, unit, p.environment.alertOptOuts) Launch workflow _ <- logToFile(...) _ <- writePolicyToVault(cfg = dc.policy, sn = sn, ns = ns.name, rs = rs)
  80. for { _ <- status(id, Pending, "workflow about to start")

    i <- dockerOps(id, unit, dc.docker.registry) _ <- status(id, Deploying, s"writing alert definitions to ${dc.name}'s consul") _ <- writeAlertsToConsul(sn, ns.name, p.name, unit, p.environment.alertOptOuts) _ <- logToFile(...) _ <- writeDiscoveryToConsul(id, sn, ns.name, dc) Launch workflow _ <- logToFile(...) _ <- writePolicyToVault(cfg = dc.policy, sn = sn, ns = ns.name, rs = rs)
  81. for { _ <- status(id, Pending, "workflow about to start")

    i <- dockerOps(id, unit, dc.docker.registry) _ <- status(id, Deploying, s"writing alert definitions to ${dc.name}'s consul") _ <- writeAlertsToConsul(sn, ns.name, p.name, unit, p.environment.alertOptOuts) _ <- logToFile(...) _ <- writeDiscoveryToConsul(id, sn, ns.name, dc) _ <- getTrafficShift.cata(...) _ <- logToFile(...) Launch workflow _ <- logToFile(...) _ <- writePolicyToVault(cfg = dc.policy, sn = sn, ns = ns.name, rs = rs)
  82. for { _ <- status(id, Pending, "workflow about to start")

    i <- dockerOps(id, unit, dc.docker.registry) _ <- status(id, Deploying, s"writing alert definitions to ${dc.name}'s consul") _ <- writeAlertsToConsul(sn, ns.name, p.name, unit, p.environment.alertOptOuts) _ <- logToFile(...) _ <- writeDiscoveryToConsul(id, sn, ns.name, dc) _ <- getTrafficShift.cata(...) _ <- logToFile(...) Launch workflow _ <- logToFile(...) _ <- writePolicyToVault(cfg = dc.policy, sn = sn, ns = ns.name, rs = rs) l <- launch(i, dc, ns.name, vunit, p, hash) _ <- debug(s"response from scheduler $l")
  83. for { _ <- status(id, Pending, "workflow about to start")

    i <- dockerOps(id, unit, dc.docker.registry) _ <- status(id, Deploying, s"writing alert definitions to ${dc.name}'s consul") _ <- writeAlertsToConsul(sn, ns.name, p.name, unit, p.environment.alertOptOuts) _ <- logToFile(...) _ <- writeDiscoveryToConsul(id, sn, ns.name, dc) _ <- getTrafficShift.cata(...) _ <- logToFile(...) _ <- status(id, getStatus(unit, p), "======> workflow completed <======") } yield () Launch workflow _ <- logToFile(...) _ <- writePolicyToVault(cfg = dc.policy, sn = sn, ns = ns.name, rs = rs) l <- launch(i, dc, ns.name, vunit, p, hash) _ <- debug(s"response from scheduler $l")
  84. logToFile(d.id, s"removing policy from vault: ${vaultLoggingFields(...)}") >> deletePolicyFromVault(d.stackName, ns.name) >>

    Delete workflow
  85. logToFile(d.id, s"removing policy from vault: ${vaultLoggingFields(...)}") >> deletePolicyFromVault(d.stackName, ns.name) >>

    Delete workflow logToFile(d.id, s"removing alerts from consul ${alerts.alertingKey(sn)}") >> deleteAlertsFromConsul(d.stackName) >>
  86. logToFile(d.id, s"removing policy from vault: ${vaultLoggingFields(...)}") >> deletePolicyFromVault(d.stackName, ns.name) >>

    Delete workflow logToFile(d.id, ...) >> deleteDiscoveryInfoFromConsul(sn) >> logToFile(d.id, s"removing alerts from consul ${alerts.alertingKey(sn)}") >> deleteAlertsFromConsul(d.stackName) >>
  87. logToFile(d.id, s"removing policy from vault: ${vaultLoggingFields(...)}") >> deletePolicyFromVault(d.stackName, ns.name) >>

    Delete workflow logToFile(d.id, ...) >> deleteDiscoveryInfoFromConsul(sn) >> logToFile(d.id, s"removing alerts from consul ${alerts.alertingKey(sn)}") >> deleteAlertsFromConsul(d.stackName) >> logToFile(d.id, s"instructing ${dc.name}'s scheduler to decommission ${sn}") >> delete(dc,d) >>
  88. logToFile(d.id, s"removing policy from vault: ${vaultLoggingFields(...)}") >> deletePolicyFromVault(d.stackName, ns.name) >>

    Delete workflow logToFile(d.id, ...) >> deleteDiscoveryInfoFromConsul(sn) >> logToFile(d.id, s"removing alerts from consul ${alerts.alertingKey(sn)}") >> deleteAlertsFromConsul(d.stackName) >> logToFile(d.id, s"instructing ${dc.name}'s scheduler to decommission ${sn}") >> delete(dc,d) >> status(d.id, Terminated, s"Decommissioning deployment ${sn} in ${dc.name}")
  89. runBackgroundJob("auditor", cfg.auditor.process(cfg.storage)) Background processes

  90. runBackgroundJob("auditor", cfg.auditor.process(cfg.storage)) runBackgroundJob("pipeline_processor", Process.eval(Pipeline.task(cfg)(Pipeline.sinks.runAction(cfg)))) Background processes

  91. runBackgroundJob("auditor", cfg.auditor.process(cfg.storage)) runBackgroundJob("pipeline_processor", Process.eval(Pipeline.task(cfg)(Pipeline.sinks.runAction(cfg)))) runBackgroundJob("workflow_logger", cfg.workflowLogger.process) Background processes

  92. runBackgroundJob("auditor", cfg.auditor.process(cfg.storage)) runBackgroundJob("pipeline_processor", Process.eval(Pipeline.task(cfg)(Pipeline.sinks.runAction(cfg)))) runBackgroundJob("workflow_logger", cfg.workflowLogger.process) runBackgroundJob("routing_cron", routing.cron.consulRefresh(cfg) to Http4sConsul.consulSink)

    Background processes
  93. runBackgroundJob("auditor", cfg.auditor.process(cfg.storage)) runBackgroundJob("pipeline_processor", Process.eval(Pipeline.task(cfg)(Pipeline.sinks.runAction(cfg)))) runBackgroundJob("workflow_logger", cfg.workflowLogger.process) runBackgroundJob("routing_cron", routing.cron.consulRefresh(cfg) to Http4sConsul.consulSink)

    runBackgroundJob("cleanup_pipeline", cleanup.CleanupCron.pipeline(cfg)) Background processes
  94. runBackgroundJob("auditor", cfg.auditor.process(cfg.storage)) runBackgroundJob("pipeline_processor", Process.eval(Pipeline.task(cfg)(Pipeline.sinks.runAction(cfg)))) runBackgroundJob("workflow_logger", cfg.workflowLogger.process) runBackgroundJob("routing_cron", routing.cron.consulRefresh(cfg) to Http4sConsul.consulSink)

    runBackgroundJob("cleanup_pipeline", cleanup.CleanupCron.pipeline(cfg)) runBackgroundJob("sweeper", cleanup.Sweeper.process(cfg)) Background processes
  95. runBackgroundJob("auditor", cfg.auditor.process(cfg.storage)) runBackgroundJob("pipeline_processor", Process.eval(Pipeline.task(cfg)(Pipeline.sinks.runAction(cfg)))) runBackgroundJob("workflow_logger", cfg.workflowLogger.process) runBackgroundJob("routing_cron", routing.cron.consulRefresh(cfg) to Http4sConsul.consulSink)

    runBackgroundJob("cleanup_pipeline", cleanup.CleanupCron.pipeline(cfg)) runBackgroundJob("sweeper", cleanup.Sweeper.process(cfg)) runBackgroundJob("deployment_monitor", DeploymentMonitor.loop(cfg)) Background processes
  96. type Op0[A] = Coproduct[DockerOp, ConsulOp, A] type Op1[A] = Coproduct[LoggingOp,

    Op0,A] type Op2[A] = Coproduct[StoreOp, Op1, A] type Op3[A] = Coproduct[WorkflowControlOp, Op2,A] type Op4[A] = Coproduct[Vault, Op3, A] type WorkflowOp[A] = Coproduct[SchedulerOp, Op4, A] type WorkflowF[A] = Free.FreeC[WorkflowOp, A] Workflow algebra
  97. final case class Interpreters( scheduler: SchedulerOp ~> Task, consul: ConsulOp

    ~> Task, vault: Vault ~> Task, storage: StoreOp ~> Task, logger: LoggingOp ~> Task, docker: DockerOp ~> Task, control: WorkflowControlOp ~> Task, health: HealthCheckOp ~> Task ) Workflow interpreter(s)
  98. None
  99. None
  100. None
  101. None
  102. None
  103. None
  104. None
  105. None
  106. None
  107. None
  108. https://getnelson.github.io/ https://github.com/getnelson/ https://gitter.im/getnelson/nelson