Persistent Data Structures in System Design

Persistent data structures are a powerful tool in the functional programming toolbox, allowing us to work with changing data without sacrificing referential transparency. However the use of persistent data structures often stops with the program, the resulting code flung over the wall to be built and deployed in non-referentially transparent systems. In this talk we will see that it doesn't have to be this way and explore system designs that leverage the techniques of persistent data structures to recover referential transparency in the ever changing world of builds and deployments.

Transcript

1. Persistent Data Structures in System Design Adelbert Chang @adelbertchang East Bay Haskell Meetup, January 9, 2018
3. B A C D E F G H
4. B A C D E F G H
5. B A C D E F G H D’ I
6. B A C D E F G H A’ D’ I
7. B C E F G H A’ D’ I
8. Properties of persistent data structures
9. Properties of persistent data structures • Non-destructive updates
10. Properties of persistent data structures • Non-destructive updates • Explicit dependency tracking
11. Properties of persistent data structures • Non-destructive updates • Explicit dependency tracking • Structural sharing
12. Properties of persistent data structures • Non-destructive updates • Explicit dependency tracking • Structural sharing • Substructures unaware of the larger whole
13. Properties of persistent data structures • Non-destructive updates • Explicit dependency tracking • Structural sharing • Substructures unaware of the larger whole • Automatic space/memory management
18. A v0.1 C v0.1 B v0.1 Installed packages D v0.1
19. A v0.2 C v0.2 B v0.1 Installed packages D v0.1
20. A v0.2 C v0.2 B v0.1 Installed packages D v0.1
21. B v0.1 Installed packages A v0.1 C v0.1 D v0.1
22. B v0.1 Installed packages A v0.1 C v0.1 D v0.1 A v0.2 C v0.2
23. B v0.1 Installed packages A v0.1 C v0.1 D v0.1 A v0.2 C v0.2 B v0.2
24. Installed packages C v0.1 D v0.1 A v0.1 A v0.2 C v0.2 B v0.2 B v0.1
25. Installed packages C v0.1 D v0.1 A v0.1 A v0.2 C v0.2 B v0.2
26. Installed packages C v0.1 D v0.1 A v0.2 C v0.2 B v0.2
27. Installed packages D v0.1 A v0.2 C v0.2 B v0.2
28. Nix: The Purely Functional Package Manager https://nixos.org/nix/
29. Nix
30. Nix • Persistent, referentially transparent build tool/package management
31. Nix • Persistent, referentially transparent build tool/package management • Tracks graph of packages and their dependencies
32. Nix • Persistent, referentially transparent build tool/package management • Tracks graph of packages and their dependencies • Packages are built from source
33. Nix • Persistent, referentially transparent build tool/package management • Tracks graph of packages and their dependencies • Packages are built from source • Builds specified with derivations in Nix, a pure, lazy, functional language
34. Nix • Persistent, referentially transparent build tool/package management • Tracks graph of packages and their dependencies • Packages are built from source • Builds specified with derivations in Nix, a pure, lazy, functional language • Uses lot of tricks top revent relying on PATH, HOME, etc.
35. Nix • Persistent, referentially transparent build tool/package management • Tracks graph of packages and their dependencies • Packages are built from source • Builds specified with derivations in Nix, a pure, lazy, functional language • Uses lot of tricks top revent relying on PATH, HOME, etc. • Packages installed in a directory based on hash of its derivation
36. { stdenv, fetchFromGitHub, caddy, asciidoctor , file, lessc, sass, multimarkdown, linkchecker , perlPackages, python27 }: stdenv.mkDerivation rec { name = "styx-${version}"; version = "0.6.0"; . . . installPhase = '' mkdir $out install -D -m 777 styx.sh $out/bin/styx mkdir -p $out/share/styx cp -r scaffold $out/share/styx cp -r nix $out/share/styx mkdir -p $out/share/doc/styx asciidoctor doc/index.adoc -o $out/share/doc/styx/index.html https://github.com/NixOS/nixpkgs/blob/1e95402803adf28bd4d78403fa9e9dcccff12101/pkgs/applications/misc/styx/default.nix
37. { stdenv, fetchFromGitHub, caddy, asciidoctor , file, lessc, sass, multimarkdown, linkchecker , perlPackages, python27 }: stdenv.mkDerivation rec { name = "styx-${version}"; version = "0.6.0"; . . . installPhase = '' mkdir $out install -D -m 777 styx.sh $out/bin/styx mkdir -p $out/share/styx cp -r scaffold $out/share/styx cp -r nix $out/share/styx mkdir -p $out/share/doc/styx asciidoctor doc/index.adoc -o $out/share/doc/styx/index.html https://github.com/NixOS/nixpkgs/blob/1e95402803adf28bd4d78403fa9e9dcccff12101/pkgs/applications/misc/styx/default.nix
38. { stdenv, fetchFromGitHub, caddy, asciidoctor , file, lessc, sass, multimarkdown, linkchecker , perlPackages, python27 }: stdenv.mkDerivation rec { name = "styx-${version}"; version = "0.6.0"; . . . installPhase = '' mkdir $out install -D -m 777 styx.sh $out/bin/styx mkdir -p $out/share/styx cp -r scaffold $out/share/styx cp -r nix $out/share/styx mkdir -p $out/share/doc/styx asciidoctor doc/index.adoc -o $out/share/doc/styx/index.html https://github.com/NixOS/nixpkgs/blob/1e95402803adf28bd4d78403fa9e9dcccff12101/pkgs/applications/misc/styx/default.nix
39. /nix/store |— 0c1p5z4kda11…-user-env | |— bin | |— svn |— 3aw2pdyx2jfc…-user-env | |— bin | |— firefox | |— svn |— ihvrn5awaw7y…-user-env | |— bin | |— firefox | |— svn |— 5mq2jcn36ldl…-subversion-1.1.2 | |— bin | |— svn |— g32imf68vvbw…-firefox-1.0.1 | |— bin | |— firefox |— dpmvp969yhdq…-subversion-1.1.3 | |— bin | |— svn https://nixos.org/nix/manual/#sec-profiles /nix/var/nix/profiles |— default |— default-0-link |— default-1-link |— default-2-link
40. /nix/store |— 0c1p5z4kda11…-user-env | |— bin | |— svn |— 3aw2pdyx2jfc…-user-env | |— bin | |— firefox | |— svn |— ihvrn5awaw7y…-user-env | |— bin | |— firefox | |— svn |— 5mq2jcn36ldl…-subversion-1.1.2 | |— bin | |— svn |— g32imf68vvbw…-firefox-1.0.1 | |— bin | |— firefox |— dpmvp969yhdq…-subversion-1.1.3 | |— bin | |— svn https://nixos.org/nix/manual/#sec-profiles /nix/var/nix/profiles |— default |— default-0-link |— default-1-link |— default-2-link
41. /nix/store |— 0c1p5z4kda11…-user-env | |— bin | |— svn |— 3aw2pdyx2jfc…-user-env | |— bin | |— firefox | |— svn |— ihvrn5awaw7y…-user-env | |— bin | |— firefox | |— svn |— 5mq2jcn36ldl…-subversion-1.1.2 | |— bin | |— svn |— g32imf68vvbw…-firefox-1.0.1 | |— bin | |— firefox |— dpmvp969yhdq…-subversion-1.1.3 | |— bin | |— svn https://nixos.org/nix/manual/#sec-profiles /nix/var/nix/profiles |— default |— default-0-link |— default-1-link |— default-2-link
42. /nix/store |— 0c1p5z4kda11…-user-env | |— bin | |— svn |— 3aw2pdyx2jfc…-user-env | |— bin | |— firefox | |— svn |— ihvrn5awaw7y…-user-env | |— bin | |— firefox | |— svn |— 5mq2jcn36ldl…-subversion-1.1.2 | |— bin | |— svn |— g32imf68vvbw…-firefox-1.0.1 | |— bin | |— firefox |— dpmvp969yhdq…-subversion-1.1.3 | |— bin | |— svn https://nixos.org/nix/manual/#sec-profiles /nix/var/nix/profiles |— default |— default-0-link |— default-1-link |— default-2-link
43. /nix/store |— 0c1p5z4kda11…-user-env | |— bin | |— svn |— 3aw2pdyx2jfc…-user-env | |— bin | |— firefox | |— svn |— ihvrn5awaw7y…-user-env | |— bin | |— firefox | |— svn |— 5mq2jcn36ldl…-subversion-1.1.2 | |— bin | |— svn |— g32imf68vvbw…-firefox-1.0.1 | |— bin | |— firefox |— dpmvp969yhdq…-subversion-1.1.3 | |— bin | |— svn https://nixos.org/nix/manual/#sec-profiles /nix/var/nix/profiles |— default |— default-0-link |— default-1-link
44. /nix/store |— 0c1p5z4kda11…-user-env | |— bin | |— svn |— 3aw2pdyx2jfc…-user-env | |— bin | |— firefox | |— svn |— ihvrn5awaw7y…-user-env | |— bin | |— firefox | |— svn |— 5mq2jcn36ldl…-subversion-1.1.2 | |— bin | |— svn |— g32imf68vvbw…-firefox-1.0.1 | |— bin | |— firefox |— dpmvp969yhdq…-subversion-1.1.3 | |— bin | |— svn https://nixos.org/nix/manual/#sec-profiles /nix/var/nix/profiles |— default |— default-0-link |— default-1-link
45. What did we gain?
46. What did we gain? • Package installs will not clobber other packages
47. What did we gain? • Package installs will not clobber other packages • Trivial rollbacks and uninstalls
48. What did we gain? • Package installs will not clobber other packages • Trivial rollbacks and uninstalls • Deterministic, reproducible builds (up to OS ABI)
49. What did we gain? • Package installs will not clobber other packages • Trivial rollbacks and uninstalls • Deterministic, reproducible builds (up to OS ABI) • Safe:
50. What did we gain? • Package installs will not clobber other packages • Trivial rollbacks and uninstalls • Deterministic, reproducible builds (up to OS ABI) • Safe: • Sharing packages across users
51. What did we gain? • Package installs will not clobber other packages • Trivial rollbacks and uninstalls • Deterministic, reproducible builds (up to OS ABI) • Safe: • Sharing packages across users • Continuous integration
52. What did we gain? • Package installs will not clobber other packages • Trivial rollbacks and uninstalls • Deterministic, reproducible builds (up to OS ABI) • Safe: • Sharing packages across users • Continuous integration • Caching packages
54. A v0.1 C v0.1 D v0.1 B v0.1 Ingress
55. A v0.1 C v0.1 D v0.1 B v0.1 A v0.2 Ingress
56. A v0.2 C v0.1 D v0.1 B v0.1 Ingress
57. A v0.2 C v0.1 D v0.1 B v0.1 Ingress
58. C v0.1 D v0.1 B v0.1 A v0.1 Ingress A v0.2
59. C v0.1 D v0.1 B v0.1 A v0.2 Ingress A v0.1
60. B v0.1 A v0.2 Ingress C v0.1 D v0.1 A v0.1
61. B v0.1 A v0.2 Ingress B v0.2 C v0.1 D v0.1 A v0.1
62. B v0.1 A v0.2 Ingress B v0.2 C v0.1 D v0.1 A v0.1
63. B v0.1 A v0.2 Ingress B v0.2 C v0.1 D v0.1 A v0.1
64. A v0.2 Ingress B v0.2 C v0.1 D v0.1 A v0.1 B v0.1
65. A v0.2 Ingress B v0.2 C v0.1 D v0.1 A v0.1
66. A v0.2 Ingress B v0.2 C v0.1 D v0.1
67. A v0.2 Ingress B v0.2 D v0.1
68. Nelson: Rigorous Deployment for a Functional World https://getnelson.github.io/
69. Nelson
70. Nelson • Continuous delivery system for immutable infrastructure
71. Nelson • Continuous delivery system for immutable infrastructure • Immutable infrastructure taken to its logical conclusion
72. Nelson • Continuous delivery system for immutable infrastructure • Immutable infrastructure taken to its logical conclusion • Tracks graph of service dependencies
73. Nelson • Continuous delivery system for immutable infrastructure • Immutable infrastructure taken to its logical conclusion • Tracks graph of service dependencies • Semantic versioning (MAJOR.MINOR.PATCH)
74. Nelson • Continuous delivery system for immutable infrastructure • Immutable infrastructure taken to its logical conclusion • Tracks graph of service dependencies • Semantic versioning (MAJOR.MINOR.PATCH) • Declare all dependencies
75. Nelson • Continuous delivery system for immutable infrastructure • Immutable infrastructure taken to its logical conclusion • Tracks graph of service dependencies • Semantic versioning (MAJOR.MINOR.PATCH) • Declare all dependencies • Ingress/load balancers and jobs form the “root” of the graph
77. units: - name: conductor description: description dependencies: - ref: db.example@1.0 ports: - default->9000/http loadbalancers: - name: lb routes: - name: expose default expose: default->9000/http destination: conductor->default
79. Ingress OSN v0.1.0
80. News Feed v0.1.0 Ingress OSN v0.1.0
81. News Feed v0.1.0 Ingress OSN v0.1.0 OSN v0.1.1
82. News Feed v0.1.0 Ingress OSN v0.1.1
83. News Feed v0.1.0 Ingress OSN v0.1.1
84. News Feed v0.1.0 Ingress OSN v0.1.1 Friends v0.1.0
85. News Feed v0.1.0 Ingress OSN v0.1.1 Friends v0.1.0 OSN v0.1.2
86. News Feed v0.1.0 Ingress Friends v0.1.0 OSN v0.1.2
87. News Feed v0.1.0 Ingress Friends v0.1.0 OSN v0.1.2
88. News Feed v0.1.0 Ingress Friends v0.1.0 ML v0.1.0 OSN v0.1.2
89. News Feed v0.1.0 Ingress Friends v0.1.0 ML v0.1.0 Friends v0.2.0 OSN v0.1.2
90. News Feed v0.1.0 Ingress Friends v0.1.0 ML v0.1.0 Friends v0.2.0 OSN v0.1.3 OSN v0.1.2
91. News Feed v0.1.0 Ingress Friends v0.1.0 ML v0.1.0 Friends v0.2.0 OSN v0.1.3
92. News Feed v0.1.0 Ingress ML v0.1.0 Friends v0.2.0 OSN v0.1.3
93. News Feed v0.1.0 Ingress ML v0.1.0 Friends v0.2.0 OSN v0.1.3
94. News Feed v0.1.0 Ingress ML v0.1.0 Friends v0.2.0 OSN v0.1.3 Friends v0.3.0
95. News Feed v0.1.0 Ingress ML v0.1.0 Friends v0.2.0 OSN v0.1.3 Friends v0.3.0 Msngr v0.1.0
96. News Feed v0.1.0 Ingress ML v0.1.0 Friends v0.2.0 OSN v0.1.3 Friends v0.3.0 Msngr v0.1.0 OSN v0.2.0
97. News Feed v0.1.0 Ingress ML v0.1.0 Friends v0.2.0 Friends v0.3.0 Msngr v0.1.0 OSN v0.2.0
98. News Feed v0.1.0 Ingress ML v0.1.0 Friends v0.3.0 Msngr v0.1.0 OSN v0.2.0
99. News Feed v0.1.0 Ingress Friends v0.3.0 Msngr v0.1.0 OSN v0.2.0
100. What did we gain?
101. What did we gain? • Deployments do not clobber other deployments
102. What did we gain? • Deployments do not clobber other deployments • Reverting a deployment is just bleeding traffic back
103. What did we gain? • Deployments do not clobber other deployments • Reverting a deployment is just bleeding traffic back • Fully automated application lifecycle
104. What did we gain? • Deployments do not clobber other deployments • Reverting a deployment is just bleeding traffic back • Fully automated application lifecycle • Clear distinction between service owners vs. infra
105. What did we gain? • Deployments do not clobber other deployments • Reverting a deployment is just bleeding traffic back • Fully automated application lifecycle • Clear distinction between service owners vs. infra • Engineering teams in charge of releasing, monitoring, etc.
106. What did we gain? • Deployments do not clobber other deployments • Reverting a deployment is just bleeding traffic back • Fully automated application lifecycle • Clear distinction between service owners vs. infra • Engineering teams in charge of releasing, monitoring, etc. • No build team, ticket filing for deployments, etc.
107. What did we gain? • Deployments do not clobber other deployments • Reverting a deployment is just bleeding traffic back • Fully automated application lifecycle • Clear distinction between service owners vs. infra • Engineering teams in charge of releasing, monitoring, etc. • No build team, ticket filing for deployments, etc. • Infra team focuses on infra!
108. Summary
109. Summary • Immutability, persistence, referential transparency, etc. are not just for code
110. Summary • Immutability, persistence, referential transparency, etc. are not just for code • See also: event sourcing, Datomic, categorical databases
111. Summary • Immutability, persistence, referential transparency, etc. are not just for code • See also: event sourcing, Datomic, categorical databases • Cross-pollination is good, FP ⇘ Systems
112. Summary • Immutability, persistence, referential transparency, etc. are not just for code • See also: event sourcing, Datomic, categorical databases • Cross-pollination is good, FP ⇘ Systems • PL and FP folks: learn about systems, machine learning, etc. and share your knowledge
113. Summary • Immutability, persistence, referential transparency, etc. are not just for code • See also: event sourcing, Datomic, categorical databases • Cross-pollination is good, FP ⇘ Systems • PL and FP folks: learn about systems, machine learning, etc. and share your knowledge • Systems folks: learn about programming languages, take ideas, build systems from solid foundations
114. Further Reading • Nix • Website: https://nixos.org/ • Nix Pills: https://nixos.org/nixos/nix-pills/ • Dr. Eelco Dolstra’s PhD thesis: https://nixos.org/~eelco/ pubs/phd-thesis.pdf • Nelson • Website: https://getnelson.github.io/ • SF Infra as Code talk: https://youtu.be/lTIvKZHedJQ • Scale by the Bay talk: https://youtu.be/3EHtAA4oE0k
115. EOF