Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Nelson: Functional programming in system design (SBTB 2018)

Nelson: Functional programming in system design (SBTB 2018)

As functional programmers we work hard to keep our code compositional and modular. Unfortunately, these noble pursuits often stop with our code, the resulting program flung over the wall to be deployed in complex and prescriptive systems. In this talk we will look at Nelson, a deployment orchestration system that applies functional programming not only in its implementation, but also in its system behavior in managing the messy world of deployment infrastructure. Where Free algebras and streams allow Nelson to be extended to support different VCS platforms, schedulers, and health checkers, Nelson's emphasis on immutable deployments inform system-level decisions such as deployment workflows and service discovery. Having been run successfully at different companies under different configurations, Nelson serves as a prime example of how functional programming can help not just with the software we write, but also with the systems we maintain.

Adelbert Chang

November 16, 2018
Tweet

More Decks by Adelbert Chang

Other Decks in Programming

Transcript

  1. Nelson
    Functional programming in system design
    Adelbert Chang
    @adelbertchang
    Lead Data Engineer @ Target
    Distinguished YAML

    View Slide

  2. https://youtu.be/t8KRo-DXnEo

    View Slide

  3. View Slide

  4. View Slide

  5. View Slide

  6. View Slide

  7. View Slide

  8. View Slide

  9. View Slide

  10. View Slide

  11. View Slide

  12. View Slide

  13. View Slide

  14. Why are people scared of deployments?

    View Slide

  15. Why are people scared of deployments?
    Deployments are often manual, happen
    infrequently, and have cascading effects.

    View Slide

  16. B
    5.4.5
    D
    2.1.8
    F
    1.2.3
    G
    1.0.9
    C
    2.3.6
    A
    1.1.5
    Mutable deployments
    E
    4.12.7

    View Slide

  17. B
    5.4.5
    D
    2.1.8
    F
    1.2.3
    G
    1.0.9
    C
    2.3.6
    A
    1.1.5
    Mutable deployments
    E
    4.12.7
    D
    2.2.0

    View Slide

  18. B
    5.4.5
    D
    2.2.0
    F
    1.2.3
    G
    1.0.9
    C
    2.3.6
    A
    1.1.5
    Mutable deployments
    E
    4.12.7

    View Slide

  19. B
    5.4.5
    D
    2.2.0
    F
    1.2.3
    G
    1.0.9
    C
    2.3.6
    A
    1.1.5
    Mutable deployments
    E
    4.12.7

    View Slide

  20. F
    1.2.3
    G
    1.0.9
    C
    2.3.6
    Immutable deployments
    A
    1.1.5
    E
    4.12.7
    D
    2.1.8
    B
    5.4.5

    View Slide

  21. F
    1.2.3
    G
    1.0.9
    C
    2.3.6
    Immutable deployments
    A
    1.1.5
    E
    4.12.7
    D
    2.1.8
    B
    5.4.5
    D
    2.2.0

    View Slide

  22. F
    1.2.3
    G
    1.0.9
    C
    2.3.6
    Immutable deployments
    A
    1.1.5
    E
    4.12.7
    D
    2.1.8
    B
    5.4.5
    D
    2.2.0

    View Slide

  23. F
    1.2.3
    G
    1.0.9
    C
    2.3.6
    Immutable deployments
    A
    1.1.5
    E
    4.12.7
    D
    2.1.8
    B
    5.4.5
    D
    2.2.0
    E
    4.12.8

    View Slide

  24. F
    1.2.3
    G
    1.0.9
    C
    2.3.6
    Immutable deployments
    A
    1.1.5
    D
    2.1.8
    B
    5.4.5
    D
    2.2.0
    E
    4.12.8

    View Slide

  25. F
    1.2.3
    G
    1.0.9
    C
    2.3.6
    Immutable deployments
    A
    1.1.5
    D
    2.1.8
    B
    5.4.5
    D
    2.2.0
    E
    4.12.8
    B
    5.4.6

    View Slide

  26. F
    1.2.3
    G
    1.0.9
    C
    2.3.6
    Immutable deployments
    A
    1.1.5
    D
    2.1.8
    B
    5.4.5
    D
    2.2.0
    E
    4.12.8
    B
    5.4.6
    A
    1.1.6

    View Slide

  27. F
    1.2.3
    G
    1.0.9
    C
    2.3.6
    Immutable deployments
    D
    2.1.8
    B
    5.4.5
    D
    2.2.0
    E
    4.12.8
    B
    5.4.6
    A
    1.1.6

    View Slide

  28. F
    1.2.3
    G
    1.0.9
    C
    2.3.6
    Immutable deployments
    D
    2.1.8
    D
    2.2.0
    E
    4.12.8
    B
    5.4.6
    A
    1.1.6

    View Slide

  29. F
    1.2.3
    G
    1.0.9
    C
    2.3.6
    Immutable deployments
    D
    2.2.0
    E
    4.12.8
    B
    5.4.6
    A
    1.1.6

    View Slide

  30. Reify the call graph
    B
    5.4.5
    D
    2.1.8
    F
    1.2.3
    G
    1.0.9
    C
    2.3.6
    A
    1.1.5
    E
    4.12.7

    View Slide

  31. Reify the call graph
    B
    5.4.5
    D
    2.1.8
    F
    1.2.3
    G
    1.0.9
    C
    2.3.6
    A
    1.1.5
    E
    4.12.7

    View Slide

  32. View Slide

  33. Nelson: User's perspective

    View Slide

  34. .nelson.yml (checked-in)
    Nelson: User's perspective

    View Slide

  35. .nelson.yml (checked-in)
    Nelson: User's perspective
    What to deploy (logical)

    View Slide

  36. .nelson.yml (checked-in)
    Nelson: User's perspective
    What to deploy (logical)
    How to deploy

    View Slide

  37. .nelson.yml (checked-in)
    Nelson: User's perspective
    What to deploy (logical)
    How to deploy
    Where to deploy

    View Slide

  38. .nelson.yml (checked-in)
    Nelson: User's perspective
    What to deploy (logical)
    How to deploy
    foo.deployable.yml (per-deploy)
    Where to deploy

    View Slide

  39. .nelson.yml (checked-in)
    Nelson: User's perspective
    What to deploy (logical)
    How to deploy
    What to deploy (physical)
    foo.deployable.yml (per-deploy)
    Where to deploy

    View Slide

  40. Nelson: User's perspective
    Cloud image taken from https://en.wikipedia.org/wiki/File:Cartoon_cloud.svg

    View Slide

  41. Nelson: User's perspective
    1. Make a deployment
    Cloud image taken from https://en.wikipedia.org/wiki/File:Cartoon_cloud.svg

    View Slide

  42. Nelson: User's perspective
    1. Make a deployment
    Cloud image taken from https://en.wikipedia.org/wiki/File:Cartoon_cloud.svg
    2. Webhook calls Nelson

    View Slide

  43. Nelson: User's perspective
    1. Make a deployment
    Cloud image taken from https://en.wikipedia.org/wiki/File:Cartoon_cloud.svg
    2. Webhook calls Nelson
    3. Nelson fetches

    + parses manifest

    View Slide

  44. Nelson: User's perspective
    1. Make a deployment
    Cloud image taken from https://en.wikipedia.org/wiki/File:Cartoon_cloud.svg
    2. Webhook calls Nelson
    3. Nelson fetches

    + parses manifest
    4. Nelson deploys to scheduler

    + enters deployment into graph

    View Slide

  45. Nelson: User's perspective
    1. Make a deployment
    Cloud image taken from https://en.wikipedia.org/wiki/File:Cartoon_cloud.svg
    2. Webhook calls Nelson
    3. Nelson fetches

    + parses manifest
    4. Nelson deploys to scheduler

    + enters deployment into graph
    5. User uses Nelson CLI and other tools

    to interact with deployment

    View Slide

  46. Nelson: User's perspective - CLI

    View Slide

  47. Nelson: User's perspective - CLI

    View Slide

  48. Nelson: User's perspective - CLI

    View Slide

  49. Nelson: User's perspective - CLI

    View Slide

  50. Nelson: Operator's perspective

    View Slide

  51. Nelson: Operator's perspective

    View Slide

  52. Nelson: Operator's perspective
    1. Nelson receives

    + parses manifests

    View Slide

  53. Nelson: Operator's perspective
    1. Nelson receives

    + parses manifests
    2. Dynamically provision

    credentials

    View Slide

  54. Nelson: Operator's perspective
    1. Nelson receives

    + parses manifests
    2. Dynamically provision

    credentials
    3. Write service discovery

    information

    View Slide

  55. Nelson: Operator's perspective
    1. Nelson receives

    + parses manifests
    2. Dynamically provision

    credentials
    3. Write service discovery

    information
    4. Deploy

    View Slide

  56. Nelson: Blueprints

    View Slide

  57. Nelson: Blueprints

    View Slide

  58. Nelson: Blueprints

    View Slide

  59. Nelson: Blueprints
    Manifest contains blueprint

    View Slide

  60. Nelson workflow

    View Slide

  61. Nelson workflow
    1. Make a deployment

    View Slide

  62. Nelson workflow
    1. Make a deployment
    2. Webhook calls Nelson

    View Slide

  63. Nelson workflow
    1. Make a deployment
    2. Webhook calls Nelson
    3. Nelson fetches

    + parses manifest

    View Slide

  64. Nelson workflow
    1. Make a deployment
    2. Webhook calls Nelson
    3. Nelson fetches

    + parses manifest
    4. Dynamically provision

    credentials

    View Slide

  65. Nelson workflow
    1. Make a deployment
    2. Webhook calls Nelson
    3. Nelson fetches

    + parses manifest
    4. Dynamically provision

    credentials
    5. Write service discovery

    information

    View Slide

  66. Nelson workflow
    1. Make a deployment
    2. Webhook calls Nelson
    3. Nelson fetches

    + parses manifest
    4. Dynamically provision

    credentials
    5. Write service discovery

    information
    6. Deploy

    View Slide

  67. Nelson workflow
    1. Make a deployment
    2. Webhook calls Nelson
    3. Nelson fetches

    + parses manifest
    4. Dynamically provision

    credentials
    5. Write service discovery

    information
    6. Deploy
    In the background:

    • User uses Nelson CLI and other tools

    to interact with deployment

    • Traffic shifting, health checks, cleanup ...

    View Slide

  68. Nelson: Roadmap
    • Kubernetes backend: no one-size-fits-all answer for
    service-to-service right now

    • Labeled services? Envoy sidecar? Istio?

    • Test deployments off of branches or pull requests

    • Actual protocols: Protocol Buffers and gRPC API

    • Nelson UI

    • "Getting Started" improvements

    View Slide

  69. https://getnelson.io/

    View Slide

  70. https://getnelson.io/
    https://github.com/getnelson/
    https://gitter.im/getnelson/nelson
    Find me for Nelson stickers!
    Questions? Tim Perrett and I are around today and
    tomorrow!

    View Slide