Persistent Data Structures in System Design

Persistent Data Structures in System Design

Persistent data structures are a powerful tool in the functional programming toolbox, allowing us to work with changing data without sacrificing referential transparency. However the use of persistent data structures often stops with the program, the resulting code flung over the wall to be built and deployed in non-referentially transparent systems. In this talk we will see that it doesn't have to be this way and explore system designs that leverage the techniques of persistent data structures to recover referential transparency in the ever changing world of builds and deployments.

Fb8e986500c5059b2a6c0b2184bb0faf?s=128

Adelbert Chang

January 09, 2018
Tweet

Transcript

  1. Persistent Data Structures in System Design Adelbert Chang @adelbertchang East

    Bay Haskell Meetup, January 9, 2018
  2. None
  3. B A C D E F G H

  4. B A C D E F G H

  5. B A C D E F G H D’ I

  6. B A C D E F G H A’ D’

    I
  7. B C E F G H A’ D’ I

  8. Properties of persistent data structures

  9. Properties of persistent data structures • Non-destructive updates

  10. Properties of persistent data structures • Non-destructive updates • Explicit

    dependency tracking
  11. Properties of persistent data structures • Non-destructive updates • Explicit

    dependency tracking • Structural sharing
  12. Properties of persistent data structures • Non-destructive updates • Explicit

    dependency tracking • Structural sharing • Substructures unaware of the larger whole
  13. Properties of persistent data structures • Non-destructive updates • Explicit

    dependency tracking • Structural sharing • Substructures unaware of the larger whole • Automatic space/memory management
  14. None
  15. None
  16. None
  17. None
  18. A v0.1 C v0.1 B v0.1 Installed packages D v0.1

  19. A v0.2 C v0.2 B v0.1 Installed packages D v0.1

  20. A v0.2 C v0.2 B v0.1 Installed packages D v0.1

  21. B v0.1 Installed packages A v0.1 C v0.1 D v0.1

  22. B v0.1 Installed packages A v0.1 C v0.1 D v0.1

    A v0.2 C v0.2
  23. B v0.1 Installed packages A v0.1 C v0.1 D v0.1

    A v0.2 C v0.2 B v0.2
  24. Installed packages C v0.1 D v0.1 A v0.1 A v0.2

    C v0.2 B v0.2 B v0.1
  25. Installed packages C v0.1 D v0.1 A v0.1 A v0.2

    C v0.2 B v0.2
  26. Installed packages C v0.1 D v0.1 A v0.2 C v0.2

    B v0.2
  27. Installed packages D v0.1 A v0.2 C v0.2 B v0.2

  28. Nix: The Purely Functional Package Manager https://nixos.org/nix/

  29. Nix

  30. Nix • Persistent, referentially transparent build tool/package management

  31. Nix • Persistent, referentially transparent build tool/package management • Tracks

    graph of packages and their dependencies
  32. Nix • Persistent, referentially transparent build tool/package management • Tracks

    graph of packages and their dependencies • Packages are built from source
  33. Nix • Persistent, referentially transparent build tool/package management • Tracks

    graph of packages and their dependencies • Packages are built from source • Builds specified with derivations in Nix, a pure, lazy, functional language
  34. Nix • Persistent, referentially transparent build tool/package management • Tracks

    graph of packages and their dependencies • Packages are built from source • Builds specified with derivations in Nix, a pure, lazy, functional language • Uses lot of tricks top revent relying on PATH, HOME, etc.
  35. Nix • Persistent, referentially transparent build tool/package management • Tracks

    graph of packages and their dependencies • Packages are built from source • Builds specified with derivations in Nix, a pure, lazy, functional language • Uses lot of tricks top revent relying on PATH, HOME, etc. • Packages installed in a directory based on hash of its derivation
  36. { stdenv, fetchFromGitHub, caddy, asciidoctor , file, lessc, sass, multimarkdown,

    linkchecker , perlPackages, python27 }: stdenv.mkDerivation rec { name = "styx-${version}"; version = "0.6.0"; . . . installPhase = '' mkdir $out install -D -m 777 styx.sh $out/bin/styx mkdir -p $out/share/styx cp -r scaffold $out/share/styx cp -r nix $out/share/styx mkdir -p $out/share/doc/styx asciidoctor doc/index.adoc -o $out/share/doc/styx/index.html https://github.com/NixOS/nixpkgs/blob/1e95402803adf28bd4d78403fa9e9dcccff12101/pkgs/applications/misc/styx/default.nix
  37. { stdenv, fetchFromGitHub, caddy, asciidoctor , file, lessc, sass, multimarkdown,

    linkchecker , perlPackages, python27 }: stdenv.mkDerivation rec { name = "styx-${version}"; version = "0.6.0"; . . . installPhase = '' mkdir $out install -D -m 777 styx.sh $out/bin/styx mkdir -p $out/share/styx cp -r scaffold $out/share/styx cp -r nix $out/share/styx mkdir -p $out/share/doc/styx asciidoctor doc/index.adoc -o $out/share/doc/styx/index.html https://github.com/NixOS/nixpkgs/blob/1e95402803adf28bd4d78403fa9e9dcccff12101/pkgs/applications/misc/styx/default.nix
  38. { stdenv, fetchFromGitHub, caddy, asciidoctor , file, lessc, sass, multimarkdown,

    linkchecker , perlPackages, python27 }: stdenv.mkDerivation rec { name = "styx-${version}"; version = "0.6.0"; . . . installPhase = '' mkdir $out install -D -m 777 styx.sh $out/bin/styx mkdir -p $out/share/styx cp -r scaffold $out/share/styx cp -r nix $out/share/styx mkdir -p $out/share/doc/styx asciidoctor doc/index.adoc -o $out/share/doc/styx/index.html https://github.com/NixOS/nixpkgs/blob/1e95402803adf28bd4d78403fa9e9dcccff12101/pkgs/applications/misc/styx/default.nix
  39. /nix/store |— 0c1p5z4kda11…-user-env | |— bin | |— svn |—

    3aw2pdyx2jfc…-user-env | |— bin | |— firefox | |— svn |— ihvrn5awaw7y…-user-env | |— bin | |— firefox | |— svn |— 5mq2jcn36ldl…-subversion-1.1.2 | |— bin | |— svn |— g32imf68vvbw…-firefox-1.0.1 | |— bin | |— firefox |— dpmvp969yhdq…-subversion-1.1.3 | |— bin | |— svn https://nixos.org/nix/manual/#sec-profiles /nix/var/nix/profiles |— default |— default-0-link |— default-1-link |— default-2-link
  40. /nix/store |— 0c1p5z4kda11…-user-env | |— bin | |— svn |—

    3aw2pdyx2jfc…-user-env | |— bin | |— firefox | |— svn |— ihvrn5awaw7y…-user-env | |— bin | |— firefox | |— svn |— 5mq2jcn36ldl…-subversion-1.1.2 | |— bin | |— svn |— g32imf68vvbw…-firefox-1.0.1 | |— bin | |— firefox |— dpmvp969yhdq…-subversion-1.1.3 | |— bin | |— svn https://nixos.org/nix/manual/#sec-profiles /nix/var/nix/profiles |— default |— default-0-link |— default-1-link |— default-2-link
  41. /nix/store |— 0c1p5z4kda11…-user-env | |— bin | |— svn |—

    3aw2pdyx2jfc…-user-env | |— bin | |— firefox | |— svn |— ihvrn5awaw7y…-user-env | |— bin | |— firefox | |— svn |— 5mq2jcn36ldl…-subversion-1.1.2 | |— bin | |— svn |— g32imf68vvbw…-firefox-1.0.1 | |— bin | |— firefox |— dpmvp969yhdq…-subversion-1.1.3 | |— bin | |— svn https://nixos.org/nix/manual/#sec-profiles /nix/var/nix/profiles |— default |— default-0-link |— default-1-link |— default-2-link
  42. /nix/store |— 0c1p5z4kda11…-user-env | |— bin | |— svn |—

    3aw2pdyx2jfc…-user-env | |— bin | |— firefox | |— svn |— ihvrn5awaw7y…-user-env | |— bin | |— firefox | |— svn |— 5mq2jcn36ldl…-subversion-1.1.2 | |— bin | |— svn |— g32imf68vvbw…-firefox-1.0.1 | |— bin | |— firefox |— dpmvp969yhdq…-subversion-1.1.3 | |— bin | |— svn https://nixos.org/nix/manual/#sec-profiles /nix/var/nix/profiles |— default |— default-0-link |— default-1-link |— default-2-link
  43. /nix/store |— 0c1p5z4kda11…-user-env | |— bin | |— svn |—

    3aw2pdyx2jfc…-user-env | |— bin | |— firefox | |— svn |— ihvrn5awaw7y…-user-env | |— bin | |— firefox | |— svn |— 5mq2jcn36ldl…-subversion-1.1.2 | |— bin | |— svn |— g32imf68vvbw…-firefox-1.0.1 | |— bin | |— firefox |— dpmvp969yhdq…-subversion-1.1.3 | |— bin | |— svn https://nixos.org/nix/manual/#sec-profiles /nix/var/nix/profiles |— default |— default-0-link |— default-1-link
  44. /nix/store |— 0c1p5z4kda11…-user-env | |— bin | |— svn |—

    3aw2pdyx2jfc…-user-env | |— bin | |— firefox | |— svn |— ihvrn5awaw7y…-user-env | |— bin | |— firefox | |— svn |— 5mq2jcn36ldl…-subversion-1.1.2 | |— bin | |— svn |— g32imf68vvbw…-firefox-1.0.1 | |— bin | |— firefox |— dpmvp969yhdq…-subversion-1.1.3 | |— bin | |— svn https://nixos.org/nix/manual/#sec-profiles /nix/var/nix/profiles |— default |— default-0-link |— default-1-link
  45. What did we gain?

  46. What did we gain? • Package installs will not clobber

    other packages
  47. What did we gain? • Package installs will not clobber

    other packages • Trivial rollbacks and uninstalls
  48. What did we gain? • Package installs will not clobber

    other packages • Trivial rollbacks and uninstalls • Deterministic, reproducible builds (up to OS ABI)
  49. What did we gain? • Package installs will not clobber

    other packages • Trivial rollbacks and uninstalls • Deterministic, reproducible builds (up to OS ABI) • Safe:
  50. What did we gain? • Package installs will not clobber

    other packages • Trivial rollbacks and uninstalls • Deterministic, reproducible builds (up to OS ABI) • Safe: • Sharing packages across users
  51. What did we gain? • Package installs will not clobber

    other packages • Trivial rollbacks and uninstalls • Deterministic, reproducible builds (up to OS ABI) • Safe: • Sharing packages across users • Continuous integration
  52. What did we gain? • Package installs will not clobber

    other packages • Trivial rollbacks and uninstalls • Deterministic, reproducible builds (up to OS ABI) • Safe: • Sharing packages across users • Continuous integration • Caching packages
  53. None
  54. A v0.1 C v0.1 D v0.1 B v0.1 Ingress

  55. A v0.1 C v0.1 D v0.1 B v0.1 A v0.2

    Ingress
  56. A v0.2 C v0.1 D v0.1 B v0.1 Ingress

  57. A v0.2 C v0.1 D v0.1 B v0.1 Ingress

  58. C v0.1 D v0.1 B v0.1 A v0.1 Ingress A

    v0.2
  59. C v0.1 D v0.1 B v0.1 A v0.2 Ingress A

    v0.1
  60. B v0.1 A v0.2 Ingress C v0.1 D v0.1 A

    v0.1
  61. B v0.1 A v0.2 Ingress B v0.2 C v0.1 D

    v0.1 A v0.1
  62. B v0.1 A v0.2 Ingress B v0.2 C v0.1 D

    v0.1 A v0.1
  63. B v0.1 A v0.2 Ingress B v0.2 C v0.1 D

    v0.1 A v0.1
  64. A v0.2 Ingress B v0.2 C v0.1 D v0.1 A

    v0.1 B v0.1
  65. A v0.2 Ingress B v0.2 C v0.1 D v0.1 A

    v0.1
  66. A v0.2 Ingress B v0.2 C v0.1 D v0.1

  67. A v0.2 Ingress B v0.2 D v0.1

  68. Nelson: Rigorous Deployment for a Functional World https://getnelson.github.io/

  69. Nelson

  70. Nelson • Continuous delivery system for immutable infrastructure

  71. Nelson • Continuous delivery system for immutable infrastructure • Immutable

    infrastructure taken to its logical conclusion
  72. Nelson • Continuous delivery system for immutable infrastructure • Immutable

    infrastructure taken to its logical conclusion • Tracks graph of service dependencies
  73. Nelson • Continuous delivery system for immutable infrastructure • Immutable

    infrastructure taken to its logical conclusion • Tracks graph of service dependencies • Semantic versioning (MAJOR.MINOR.PATCH)
  74. Nelson • Continuous delivery system for immutable infrastructure • Immutable

    infrastructure taken to its logical conclusion • Tracks graph of service dependencies • Semantic versioning (MAJOR.MINOR.PATCH) • Declare all dependencies
  75. Nelson • Continuous delivery system for immutable infrastructure • Immutable

    infrastructure taken to its logical conclusion • Tracks graph of service dependencies • Semantic versioning (MAJOR.MINOR.PATCH) • Declare all dependencies • Ingress/load balancers and jobs form the “root” of the graph
  76. None
  77. units: - name: conductor description: description dependencies: - ref: db.example@1.0

    ports: - default->9000/http loadbalancers: - name: lb routes: - name: expose default expose: default->9000/http destination: conductor->default
  78. None
  79. Ingress OSN v0.1.0

  80. News Feed v0.1.0 Ingress OSN v0.1.0

  81. News Feed v0.1.0 Ingress OSN v0.1.0 OSN v0.1.1

  82. News Feed v0.1.0 Ingress OSN v0.1.1

  83. News Feed v0.1.0 Ingress OSN v0.1.1

  84. News Feed v0.1.0 Ingress OSN v0.1.1 Friends v0.1.0

  85. News Feed v0.1.0 Ingress OSN v0.1.1 Friends v0.1.0 OSN v0.1.2

  86. News Feed v0.1.0 Ingress Friends v0.1.0 OSN v0.1.2

  87. News Feed v0.1.0 Ingress Friends v0.1.0 OSN v0.1.2

  88. News Feed v0.1.0 Ingress Friends v0.1.0 ML v0.1.0 OSN v0.1.2

  89. News Feed v0.1.0 Ingress Friends v0.1.0 ML v0.1.0 Friends v0.2.0

    OSN v0.1.2
  90. News Feed v0.1.0 Ingress Friends v0.1.0 ML v0.1.0 Friends v0.2.0

    OSN v0.1.3 OSN v0.1.2
  91. News Feed v0.1.0 Ingress Friends v0.1.0 ML v0.1.0 Friends v0.2.0

    OSN v0.1.3
  92. News Feed v0.1.0 Ingress ML v0.1.0 Friends v0.2.0 OSN v0.1.3

  93. News Feed v0.1.0 Ingress ML v0.1.0 Friends v0.2.0 OSN v0.1.3

  94. News Feed v0.1.0 Ingress ML v0.1.0 Friends v0.2.0 OSN v0.1.3

    Friends v0.3.0
  95. News Feed v0.1.0 Ingress ML v0.1.0 Friends v0.2.0 OSN v0.1.3

    Friends v0.3.0 Msngr v0.1.0
  96. News Feed v0.1.0 Ingress ML v0.1.0 Friends v0.2.0 OSN v0.1.3

    Friends v0.3.0 Msngr v0.1.0 OSN v0.2.0
  97. News Feed v0.1.0 Ingress ML v0.1.0 Friends v0.2.0 Friends v0.3.0

    Msngr v0.1.0 OSN v0.2.0
  98. News Feed v0.1.0 Ingress ML v0.1.0 Friends v0.3.0 Msngr v0.1.0

    OSN v0.2.0
  99. News Feed v0.1.0 Ingress Friends v0.3.0 Msngr v0.1.0 OSN v0.2.0

  100. What did we gain?

  101. What did we gain? • Deployments do not clobber other

    deployments
  102. What did we gain? • Deployments do not clobber other

    deployments • Reverting a deployment is just bleeding traffic back
  103. What did we gain? • Deployments do not clobber other

    deployments • Reverting a deployment is just bleeding traffic back • Fully automated application lifecycle
  104. What did we gain? • Deployments do not clobber other

    deployments • Reverting a deployment is just bleeding traffic back • Fully automated application lifecycle • Clear distinction between service owners vs. infra
  105. What did we gain? • Deployments do not clobber other

    deployments • Reverting a deployment is just bleeding traffic back • Fully automated application lifecycle • Clear distinction between service owners vs. infra • Engineering teams in charge of releasing, monitoring, etc.
  106. What did we gain? • Deployments do not clobber other

    deployments • Reverting a deployment is just bleeding traffic back • Fully automated application lifecycle • Clear distinction between service owners vs. infra • Engineering teams in charge of releasing, monitoring, etc. • No build team, ticket filing for deployments, etc.
  107. What did we gain? • Deployments do not clobber other

    deployments • Reverting a deployment is just bleeding traffic back • Fully automated application lifecycle • Clear distinction between service owners vs. infra • Engineering teams in charge of releasing, monitoring, etc. • No build team, ticket filing for deployments, etc. • Infra team focuses on infra!
  108. Summary

  109. Summary • Immutability, persistence, referential transparency, etc. are not just

    for code
  110. Summary • Immutability, persistence, referential transparency, etc. are not just

    for code • See also: event sourcing, Datomic, categorical databases
  111. Summary • Immutability, persistence, referential transparency, etc. are not just

    for code • See also: event sourcing, Datomic, categorical databases • Cross-pollination is good, FP ⇘ Systems
  112. Summary • Immutability, persistence, referential transparency, etc. are not just

    for code • See also: event sourcing, Datomic, categorical databases • Cross-pollination is good, FP ⇘ Systems • PL and FP folks: learn about systems, machine learning, etc. and share your knowledge
  113. Summary • Immutability, persistence, referential transparency, etc. are not just

    for code • See also: event sourcing, Datomic, categorical databases • Cross-pollination is good, FP ⇘ Systems • PL and FP folks: learn about systems, machine learning, etc. and share your knowledge • Systems folks: learn about programming languages, take ideas, build systems from solid foundations
  114. Further Reading • Nix • Website: https://nixos.org/ • Nix Pills:

    https://nixos.org/nixos/nix-pills/ • Dr. Eelco Dolstra’s PhD thesis: https://nixos.org/~eelco/ pubs/phd-thesis.pdf • Nelson • Website: https://getnelson.github.io/ • SF Infra as Code talk: https://youtu.be/lTIvKZHedJQ • Scale by the Bay talk: https://youtu.be/3EHtAA4oE0k
  115. EOF