Nelson: Functional programming in system design

Nelson: Functional programming in system design

As functional programmers we work hard to keep things immutable and referentially transparent. However, these noble pursuits often stop with our text editors, the resulting code flung over the wall to be built and deployed in mutable, imperative systems. In this talk we will take a look at Nelson, a deployment orchestration system that applies functional programming not only in its implementation, but also in its system behavior in managing the messy world of deployment infrastructure. Where Free algebras and streams allow the code to be easily extended to support different source repositories, schedulers, and health checkers, Nelson's strict stance on immutable deployments inform important system level decisions such as deployment workflows and service discovery. Having now been run at a couple of companies under different configurations, Nelson serves as a prime example of how functional programming is for the real world and can influence not just code, but systems as well.

Fb8e986500c5059b2a6c0b2184bb0faf?s=128

Adelbert Chang

March 19, 2018
Tweet

Transcript

  1. 2.
  2. 3.
  3. 4.
  4. 5.
  5. 6.
  6. 7.

    B 5.4.5 D 2.1.8 F 1.2.3 G 1.0.9 C 2.3.6

    A 1.1.5 Mutable deployments E 4.12.7
  7. 8.

    B 5.4.5 D 2.1.8 F 1.2.3 G 1.0.9 C 2.3.6

    A 1.1.5 Mutable deployments E 4.12.7 D 2.2.0
  8. 9.

    B 5.4.5 D 2.2.0 F 1.2.3 G 1.0.9 C 2.3.6

    A 1.1.5 Mutable deployments E 4.12.7
  9. 10.

    B 5.4.5 D 2.2.0 F 1.2.3 G 1.0.9 C 2.3.6

    A 1.1.5 Mutable deployments E 4.12.7
  10. 12.
  11. 13.
  12. 14.

    F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5

    E 4.12.7 B 5.4.5 D 2.1.8 D 2.2.0 E 4.12.8
  13. 15.
  14. 16.

    F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5

    B 5.4.5 D 2.1.8 D 2.2.0 E 4.12.8 B 5.4.6
  15. 17.

    F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments A 1.1.5

    B 5.4.5 D 2.1.8 D 2.2.0 E 4.12.8 B 5.4.6 A 1.1.6
  16. 18.

    F 1.2.3 G 1.0.9 C 2.3.6 Immutable deployments B 5.4.5

    D 2.1.8 D 2.2.0 E 4.12.8 B 5.4.6 A 1.1.6
  17. 20.

    Reify the call graph B 5.4.5 D 2.1.8 F 1.2.3

    G 1.0.9 C 2.3.6 A 1.1.5 E 4.12.7
  18. 21.

    Reify the call graph B 5.4.5 D 2.1.8 F 1.2.3

    G 1.0.9 C 2.3.6 A 1.1.5 E 4.12.7
  19. 22.

    Reify the call graph B 5.4.5 D 2.1.8 F 1.2.3

    G 1.0.9 C 2.3.6 A 1.1.5 E 4.12.7
  20. 23.

    Reify the call graph B 5.4.5 D 2.1.8 F 1.2.3

    G 1.0.9 C 2.3.6 A 1.1.5 E 4.12.7
  21. 24.

    Reify the call graph B 5.4.5 D 2.1.8 F 1.2.3

    G 1.0.9 C 2.3.6 A 1.1.5 E 4.12.7
  22. 25.

    Reify the call graph B 5.4.5 D 2.1.8 F 1.2.3

    G 1.0.9 C 2.3.6 A 1.1.5 E 4.12.7
  23. 26.
  24. 27.
  25. 36.

    Graph management: traffic shifting C 2.3.6 A 1.1.5 E 4.12.7

    F 1.2.3 G 1.0.9 D 2.1.8 B 5.4.5 C 2.3.7
  26. 37.

    Graph management: traffic shifting C 2.3.6 A 1.1.5 E 4.12.7

    F 1.2.3 G 1.0.9 D 2.1.8 B 5.4.5 C 2.3.7 B 5.4.6
  27. 38.

    Graph management: traffic shifting C 2.3.6 A 1.1.5 E 4.12.7

    F 1.2.3 G 1.0.9 D 2.1.8 B 5.4.5 C 2.3.7 B 5.4.6 A 1.1.6
  28. 39.

    Graph management: traffic shifting C 2.3.6 A 1.1.5 E 4.12.7

    F 1.2.3 G 1.0.9 D 2.1.8 B 5.4.5 C 2.3.7
  29. 40.

    Graph management: traffic shifting C 2.3.6 A 1.1.5 E 4.12.7

    F 1.2.3 G 1.0.9 D 2.1.8 B 5.4.5 C 2.3.7
  30. 41.

    Graph management: traffic shifting D 2.1.8 C 2.3.6 A 1.1.5

    B 5.4.5 C 2.3.7 E 4.12.7 F 1.2.3 G 1.0.9
  31. 43.
  32. 44.
  33. 45.

    Graph management: pruning A 1.1.5 E 4.12.7 F 1.2.3 G

    1.0.9 D 2.1.8 B 5.4.5 F 1.3.0 C 2.3.7
  34. 46.

    Graph management: pruning A 1.1.5 E 4.12.7 F 1.2.3 G

    1.0.9 D 2.1.8 B 5.4.5 F 1.3.0 E 4.12.8 C 2.3.7
  35. 47.

    Graph management: pruning A 1.1.5 F 1.2.3 G 1.0.9 D

    2.1.8 B 5.4.5 F 1.3.0 E 4.12.8 C 2.3.7
  36. 49.

    D 2.1.8 A 1.1.5 F 1.3.0 E 4.12.8 C 2.3.7

    B 5.4.5 Graph management: pruning
  37. 50.

    D 2.1.8 A 1.1.5 F 1.3.0 E 4.12.8 C 2.3.7

    B 5.4.5 Graph management: pruning B 5.4.6
  38. 51.

    D 2.1.8 A 1.1.5 F 1.3.0 E 4.12.8 C 2.3.7

    B 5.4.5 Graph management: pruning B 5.4.6
  39. 52.

    D 2.1.8 A 1.1.5 F 1.3.0 E 4.12.8 C 2.3.7

    B 5.4.5 Graph management: pruning B 5.4.6
  40. 53.

    D 2.1.8 A 1.1.5 F 1.3.0 E 4.12.8 C 2.3.7

    Graph management: pruning B 5.4.6
  41. 54.

    D 2.1.8 A 1.1.5 F 1.3.0 E 4.12.8 C 2.3.7

    Graph management: pruning B 5.4.6
  42. 55.

    D 2.1.8 A 1.1.5 F 1.3.0 E 4.12.8 C 2.3.7

    Job X 1.8.5 Graph management: pruning B 5.4.6
  43. 56.
  44. 67.

    X 3.2.0 X 3.2.1 X 3.3.0 X 4.0.0 X 4.1.0

    Time retain-latest X 3.2.0 X 3.2.1 X 3.3.0 X 4.0.0 X 4.1.0 retain-latest-two-major
  45. 68.

    X 3.2.0 X 3.2.1 X 3.3.0 X 4.0.0 X 4.1.0

    Time retain-latest X 3.2.0 X 3.2.1 X 3.3.0 X 4.0.0 X 4.1.0 retain-latest-two-major X 3.2.0 X 3.2.1 X 3.3.0 X 4.0.0 X 4.1.0 retain-latest-two-feature
  46. 69.

    D 2.1.8 A 1.1.5 F 1.2.4 E 4.12.8 C 2.3.7

    Job X 1.8.5 Graph management: pruning B 5.4.6
  47. 70.

    D 2.1.8 A 1.1.5 F 1.2.4 E 4.12.8 C 2.3.7

    Job X 1.8.5 Job X 1.9.0 Graph management: pruning B 5.4.6
  48. 71.

    D 2.1.8 A 1.1.5 F 1.2.4 E 4.12.8 C 2.3.7

    Job X 1.8.5 Job X 1.9.0 Job X 1.8.6 Graph management: pruning B 5.4.6
  49. 72.

    D 2.1.8 A 1.1.5 F 1.2.4 E 4.12.8 C 2.3.7

    Job X 1.9.0 Job X 1.8.6 Graph management: pruning B 5.4.6
  50. 73.

    D 2.1.8 A 1.1.5 F 1.2.4 E 4.12.8 C 2.3.7

    Job X 1.9.0 Job X 1.8.6 Graph management: pruning B 5.4.6
  51. 74.

    D 2.1.8 A 1.1.5 F 1.2.4 E 4.12.8 C 2.3.7

    Job X 1.9.0 Job X 1.8.6 Job X 1.9.1 Graph management: pruning B 5.4.6
  52. 75.

    D 2.1.8 A 1.1.5 F 1.2.4 E 4.12.8 C 2.3.7

    Job X 1.8.6 Job X 1.9.1 Graph management: pruning B 5.4.6
  53. 76.
  54. 77.

    for { _ <- status(id, Pending, "workflow about to start")

    i <- dockerOps(id, unit, dc.docker.registry) Launch workflow
  55. 78.

    for { _ <- status(id, Pending, "workflow about to start")

    i <- dockerOps(id, unit, dc.docker.registry) _ <- status(id, Deploying, s"writing alert definitions to ${dc.name}'s consul") _ <- writeAlertsToConsul(sn, ns.name, p.name, unit, p.environment.alertOptOuts) Launch workflow
  56. 79.

    for { _ <- status(id, Pending, "workflow about to start")

    i <- dockerOps(id, unit, dc.docker.registry) _ <- status(id, Deploying, s"writing alert definitions to ${dc.name}'s consul") _ <- writeAlertsToConsul(sn, ns.name, p.name, unit, p.environment.alertOptOuts) Launch workflow _ <- logToFile(...) _ <- writePolicyToVault(cfg = dc.policy, sn = sn, ns = ns.name, rs = rs)
  57. 80.

    for { _ <- status(id, Pending, "workflow about to start")

    i <- dockerOps(id, unit, dc.docker.registry) _ <- status(id, Deploying, s"writing alert definitions to ${dc.name}'s consul") _ <- writeAlertsToConsul(sn, ns.name, p.name, unit, p.environment.alertOptOuts) _ <- logToFile(...) _ <- writeDiscoveryToConsul(id, sn, ns.name, dc) Launch workflow _ <- logToFile(...) _ <- writePolicyToVault(cfg = dc.policy, sn = sn, ns = ns.name, rs = rs)
  58. 81.

    for { _ <- status(id, Pending, "workflow about to start")

    i <- dockerOps(id, unit, dc.docker.registry) _ <- status(id, Deploying, s"writing alert definitions to ${dc.name}'s consul") _ <- writeAlertsToConsul(sn, ns.name, p.name, unit, p.environment.alertOptOuts) _ <- logToFile(...) _ <- writeDiscoveryToConsul(id, sn, ns.name, dc) _ <- getTrafficShift.cata(...) _ <- logToFile(...) Launch workflow _ <- logToFile(...) _ <- writePolicyToVault(cfg = dc.policy, sn = sn, ns = ns.name, rs = rs)
  59. 82.

    for { _ <- status(id, Pending, "workflow about to start")

    i <- dockerOps(id, unit, dc.docker.registry) _ <- status(id, Deploying, s"writing alert definitions to ${dc.name}'s consul") _ <- writeAlertsToConsul(sn, ns.name, p.name, unit, p.environment.alertOptOuts) _ <- logToFile(...) _ <- writeDiscoveryToConsul(id, sn, ns.name, dc) _ <- getTrafficShift.cata(...) _ <- logToFile(...) Launch workflow _ <- logToFile(...) _ <- writePolicyToVault(cfg = dc.policy, sn = sn, ns = ns.name, rs = rs) l <- launch(i, dc, ns.name, vunit, p, hash) _ <- debug(s"response from scheduler $l")
  60. 83.

    for { _ <- status(id, Pending, "workflow about to start")

    i <- dockerOps(id, unit, dc.docker.registry) _ <- status(id, Deploying, s"writing alert definitions to ${dc.name}'s consul") _ <- writeAlertsToConsul(sn, ns.name, p.name, unit, p.environment.alertOptOuts) _ <- logToFile(...) _ <- writeDiscoveryToConsul(id, sn, ns.name, dc) _ <- getTrafficShift.cata(...) _ <- logToFile(...) _ <- status(id, getStatus(unit, p), "======> workflow completed <======") } yield () Launch workflow _ <- logToFile(...) _ <- writePolicyToVault(cfg = dc.policy, sn = sn, ns = ns.name, rs = rs) l <- launch(i, dc, ns.name, vunit, p, hash) _ <- debug(s"response from scheduler $l")
  61. 85.

    logToFile(d.id, s"removing policy from vault: ${vaultLoggingFields(...)}") >> deletePolicyFromVault(d.stackName, ns.name) >>

    Delete workflow logToFile(d.id, s"removing alerts from consul ${alerts.alertingKey(sn)}") >> deleteAlertsFromConsul(d.stackName) >>
  62. 86.

    logToFile(d.id, s"removing policy from vault: ${vaultLoggingFields(...)}") >> deletePolicyFromVault(d.stackName, ns.name) >>

    Delete workflow logToFile(d.id, ...) >> deleteDiscoveryInfoFromConsul(sn) >> logToFile(d.id, s"removing alerts from consul ${alerts.alertingKey(sn)}") >> deleteAlertsFromConsul(d.stackName) >>
  63. 87.

    logToFile(d.id, s"removing policy from vault: ${vaultLoggingFields(...)}") >> deletePolicyFromVault(d.stackName, ns.name) >>

    Delete workflow logToFile(d.id, ...) >> deleteDiscoveryInfoFromConsul(sn) >> logToFile(d.id, s"removing alerts from consul ${alerts.alertingKey(sn)}") >> deleteAlertsFromConsul(d.stackName) >> logToFile(d.id, s"instructing ${dc.name}'s scheduler to decommission ${sn}") >> delete(dc,d) >>
  64. 88.

    logToFile(d.id, s"removing policy from vault: ${vaultLoggingFields(...)}") >> deletePolicyFromVault(d.stackName, ns.name) >>

    Delete workflow logToFile(d.id, ...) >> deleteDiscoveryInfoFromConsul(sn) >> logToFile(d.id, s"removing alerts from consul ${alerts.alertingKey(sn)}") >> deleteAlertsFromConsul(d.stackName) >> logToFile(d.id, s"instructing ${dc.name}'s scheduler to decommission ${sn}") >> delete(dc,d) >> status(d.id, Terminated, s"Decommissioning deployment ${sn} in ${dc.name}")
  65. 95.

    runBackgroundJob("auditor", cfg.auditor.process(cfg.storage)) runBackgroundJob("pipeline_processor", Process.eval(Pipeline.task(cfg)(Pipeline.sinks.runAction(cfg)))) runBackgroundJob("workflow_logger", cfg.workflowLogger.process) runBackgroundJob("routing_cron", routing.cron.consulRefresh(cfg) to Http4sConsul.consulSink)

    runBackgroundJob("cleanup_pipeline", cleanup.CleanupCron.pipeline(cfg)) runBackgroundJob("sweeper", cleanup.Sweeper.process(cfg)) runBackgroundJob("deployment_monitor", DeploymentMonitor.loop(cfg)) Background processes
  66. 96.

    type Op0[A] = Coproduct[DockerOp, ConsulOp, A] type Op1[A] = Coproduct[LoggingOp,

    Op0,A] type Op2[A] = Coproduct[StoreOp, Op1, A] type Op3[A] = Coproduct[WorkflowControlOp, Op2,A] type Op4[A] = Coproduct[Vault, Op3, A] type WorkflowOp[A] = Coproduct[SchedulerOp, Op4, A] type WorkflowF[A] = Free.FreeC[WorkflowOp, A] Workflow algebra
  67. 97.

    final case class Interpreters( scheduler: SchedulerOp ~> Task, consul: ConsulOp

    ~> Task, vault: Vault ~> Task, storage: StoreOp ~> Task, logger: LoggingOp ~> Task, docker: DockerOp ~> Task, control: WorkflowControlOp ~> Task, health: HealthCheckOp ~> Task ) Workflow interpreter(s)
  68. 98.
  69. 99.
  70. 100.
  71. 101.
  72. 102.
  73. 103.
  74. 104.
  75. 105.
  76. 106.
  77. 107.