Persistent Data Structures in System Design

Persistent data structures are a powerful tool in the functional programming toolbox, allowing us to work with changing data without sacrificing referential transparency. However the use of persistent data structures often stops with the program, the resulting code flung over the wall to be built and deployed in non-referentially transparent systems. In this talk we will see that it doesn't have to be this way and explore system designs that leverage the techniques of persistent data structures to recover referential transparency in the ever changing world of builds and deployments.

Transcript

1. Persistent Data Structures
   in System Design
   Adelbert Chang
   @adelbertchang
   East Bay Haskell Meetup, January 9, 2018
   

   View Slide

2. View Slide

3. B
   A
   C D
   E F G H
   

   View Slide

4. B
   A
   C D
   E F G H
   

   View Slide

5. B
   A
   C D
   E F G H
   D’
   I
   

   View Slide

6. B
   A
   C D
   E F G H
   A’
   D’
   I
   

   View Slide

7. B C
   E F G H
   A’
   D’
   I
   

   View Slide

8. Properties of persistent data structures
   

   View Slide

9. Properties of persistent data structures
   • Non-destructive updates
   

   View Slide

10. Properties of persistent data structures
    • Non-destructive updates
    • Explicit dependency tracking
    

    View Slide

11. Properties of persistent data structures
    • Non-destructive updates
    • Explicit dependency tracking
    • Structural sharing
    

    View Slide

12. Properties of persistent data structures
    • Non-destructive updates
    • Explicit dependency tracking
    • Structural sharing
    • Substructures unaware of the larger whole
    

    View Slide

13. Properties of persistent data structures
    • Non-destructive updates
    • Explicit dependency tracking
    • Structural sharing
    • Substructures unaware of the larger whole
    • Automatic space/memory management
    

    View Slide

14. View Slide

15. View Slide

16. View Slide

17. View Slide

18. A v0.1
    C v0.1
    B v0.1
    Installed packages
    D v0.1
    

    View Slide

19. A v0.2
    C v0.2
    B v0.1
    Installed packages
    D v0.1
    

    View Slide

20. A v0.2
    C v0.2
    B v0.1
    Installed packages
    D v0.1
    

    View Slide

21. B v0.1
    Installed packages
    A v0.1
    C v0.1 D v0.1
    

    View Slide

22. B v0.1
    Installed packages
    A v0.1
    C v0.1 D v0.1
    A v0.2
    C v0.2
    

    View Slide

23. B v0.1
    Installed packages
    A v0.1
    C v0.1 D v0.1
    A v0.2
    C v0.2
    B v0.2
    

    View Slide

24. Installed packages
    C v0.1 D v0.1
    A v0.1
    A v0.2
    C v0.2
    B v0.2
    B v0.1
    

    View Slide

25. Installed packages
    C v0.1 D v0.1
    A v0.1
    A v0.2
    C v0.2
    B v0.2
    

    View Slide

26. Installed packages
    C v0.1 D v0.1
    A v0.2
    C v0.2
    B v0.2
    

    View Slide

27. Installed packages
    D v0.1
    A v0.2
    C v0.2
    B v0.2
    

    View Slide

28. Nix: The Purely Functional Package Manager
    https://nixos.org/nix/
    

    View Slide

29. Nix
    

    View Slide

30. Nix
    • Persistent, referentially transparent build tool/package
    management
    

    View Slide

31. Nix
    • Persistent, referentially transparent build tool/package
    management
    • Tracks graph of packages and their dependencies
    

    View Slide

32. Nix
    • Persistent, referentially transparent build tool/package
    management
    • Tracks graph of packages and their dependencies
    • Packages are built from source
    

    View Slide

33. Nix
    • Persistent, referentially transparent build tool/package
    management
    • Tracks graph of packages and their dependencies
    • Packages are built from source
    • Builds specified with derivations in Nix, a pure, lazy,
    functional language
    

    View Slide

34. Nix
    • Persistent, referentially transparent build tool/package
    management
    • Tracks graph of packages and their dependencies
    • Packages are built from source
    • Builds specified with derivations in Nix, a pure, lazy,
    functional language
    • Uses lot of tricks top revent relying on PATH, HOME, etc.
    

    View Slide

35. Nix
    • Persistent, referentially transparent build tool/package
    management
    • Tracks graph of packages and their dependencies
    • Packages are built from source
    • Builds specified with derivations in Nix, a pure, lazy,
    functional language
    • Uses lot of tricks top revent relying on PATH, HOME, etc.
    • Packages installed in a directory based on hash of its
    derivation
    

    View Slide

36. { stdenv, fetchFromGitHub, caddy, asciidoctor
    , file, lessc, sass, multimarkdown, linkchecker
    , perlPackages, python27 }:
    stdenv.mkDerivation rec {
    name = "styx-${version}";
    version = "0.6.0";
    . . .
    installPhase = ''
    mkdir $out
    install -D -m 777 styx.sh $out/bin/styx
    mkdir -p $out/share/styx
    cp -r scaffold $out/share/styx
    cp -r nix $out/share/styx
    mkdir -p $out/share/doc/styx
    asciidoctor doc/index.adoc -o $out/share/doc/styx/index.html
    https://github.com/NixOS/nixpkgs/blob/1e95402803adf28bd4d78403fa9e9dcccff12101/pkgs/applications/misc/styx/default.nix
    

    View Slide

37. { stdenv, fetchFromGitHub, caddy, asciidoctor
    , file, lessc, sass, multimarkdown, linkchecker
    , perlPackages, python27 }:
    stdenv.mkDerivation rec {
    name = "styx-${version}";
    version = "0.6.0";
    . . .
    installPhase = ''
    mkdir $out
    install -D -m 777 styx.sh $out/bin/styx
    mkdir -p $out/share/styx
    cp -r scaffold $out/share/styx
    cp -r nix $out/share/styx
    mkdir -p $out/share/doc/styx
    asciidoctor doc/index.adoc -o $out/share/doc/styx/index.html
    https://github.com/NixOS/nixpkgs/blob/1e95402803adf28bd4d78403fa9e9dcccff12101/pkgs/applications/misc/styx/default.nix
    

    View Slide

38. { stdenv, fetchFromGitHub, caddy, asciidoctor
    , file, lessc, sass, multimarkdown, linkchecker
    , perlPackages, python27 }:
    stdenv.mkDerivation rec {
    name = "styx-${version}";
    version = "0.6.0";
    . . .
    installPhase = ''
    mkdir $out
    install -D -m 777 styx.sh $out/bin/styx
    mkdir -p $out/share/styx
    cp -r scaffold $out/share/styx
    cp -r nix $out/share/styx
    mkdir -p $out/share/doc/styx
    asciidoctor doc/index.adoc -o $out/share/doc/styx/index.html
    https://github.com/NixOS/nixpkgs/blob/1e95402803adf28bd4d78403fa9e9dcccff12101/pkgs/applications/misc/styx/default.nix
    

    View Slide

39. /nix/store
    |— 0c1p5z4kda11…-user-env
    | |— bin
    | |— svn
    |— 3aw2pdyx2jfc…-user-env
    | |— bin
    | |— firefox
    | |— svn
    |— ihvrn5awaw7y…-user-env
    | |— bin
    | |— firefox
    | |— svn
    |— 5mq2jcn36ldl…-subversion-1.1.2
    | |— bin
    | |— svn
    |— g32imf68vvbw…-firefox-1.0.1
    | |— bin
    | |— firefox
    |— dpmvp969yhdq…-subversion-1.1.3
    | |— bin
    | |— svn
    https://nixos.org/nix/manual/#sec-profiles
    /nix/var/nix/profiles
    |— default
    |— default-0-link
    |— default-1-link
    |— default-2-link
    

    View Slide

40. /nix/store
    |— 0c1p5z4kda11…-user-env
    | |— bin
    | |— svn
    |— 3aw2pdyx2jfc…-user-env
    | |— bin
    | |— firefox
    | |— svn
    |— ihvrn5awaw7y…-user-env
    | |— bin
    | |— firefox
    | |— svn
    |— 5mq2jcn36ldl…-subversion-1.1.2
    | |— bin
    | |— svn
    |— g32imf68vvbw…-firefox-1.0.1
    | |— bin
    | |— firefox
    |— dpmvp969yhdq…-subversion-1.1.3
    | |— bin
    | |— svn
    https://nixos.org/nix/manual/#sec-profiles
    /nix/var/nix/profiles
    |— default
    |— default-0-link
    |— default-1-link
    |— default-2-link
    

    View Slide

41. /nix/store
    |— 0c1p5z4kda11…-user-env
    | |— bin
    | |— svn
    |— 3aw2pdyx2jfc…-user-env
    | |— bin
    | |— firefox
    | |— svn
    |— ihvrn5awaw7y…-user-env
    | |— bin
    | |— firefox
    | |— svn
    |— 5mq2jcn36ldl…-subversion-1.1.2
    | |— bin
    | |— svn
    |— g32imf68vvbw…-firefox-1.0.1
    | |— bin
    | |— firefox
    |— dpmvp969yhdq…-subversion-1.1.3
    | |— bin
    | |— svn
    https://nixos.org/nix/manual/#sec-profiles
    /nix/var/nix/profiles
    |— default
    |— default-0-link
    |— default-1-link
    |— default-2-link
    

    View Slide

42. /nix/store
    |— 0c1p5z4kda11…-user-env
    | |— bin
    | |— svn
    |— 3aw2pdyx2jfc…-user-env
    | |— bin
    | |— firefox
    | |— svn
    |— ihvrn5awaw7y…-user-env
    | |— bin
    | |— firefox
    | |— svn
    |— 5mq2jcn36ldl…-subversion-1.1.2
    | |— bin
    | |— svn
    |— g32imf68vvbw…-firefox-1.0.1
    | |— bin
    | |— firefox
    |— dpmvp969yhdq…-subversion-1.1.3
    | |— bin
    | |— svn
    https://nixos.org/nix/manual/#sec-profiles
    /nix/var/nix/profiles
    |— default
    |— default-0-link
    |— default-1-link
    |— default-2-link
    

    View Slide

43. /nix/store
    |— 0c1p5z4kda11…-user-env
    | |— bin
    | |— svn
    |— 3aw2pdyx2jfc…-user-env
    | |— bin
    | |— firefox
    | |— svn
    |— ihvrn5awaw7y…-user-env
    | |— bin
    | |— firefox
    | |— svn
    |— 5mq2jcn36ldl…-subversion-1.1.2
    | |— bin
    | |— svn
    |— g32imf68vvbw…-firefox-1.0.1
    | |— bin
    | |— firefox
    |— dpmvp969yhdq…-subversion-1.1.3
    | |— bin
    | |— svn
    https://nixos.org/nix/manual/#sec-profiles
    /nix/var/nix/profiles
    |— default
    |— default-0-link
    |— default-1-link
    

    View Slide

44. /nix/store
    |— 0c1p5z4kda11…-user-env
    | |— bin
    | |— svn
    |— 3aw2pdyx2jfc…-user-env
    | |— bin
    | |— firefox
    | |— svn
    |— ihvrn5awaw7y…-user-env
    | |— bin
    | |— firefox
    | |— svn
    |— 5mq2jcn36ldl…-subversion-1.1.2
    | |— bin
    | |— svn
    |— g32imf68vvbw…-firefox-1.0.1
    | |— bin
    | |— firefox
    |— dpmvp969yhdq…-subversion-1.1.3
    | |— bin
    | |— svn
    https://nixos.org/nix/manual/#sec-profiles
    /nix/var/nix/profiles
    |— default
    |— default-0-link
    |— default-1-link
    

    View Slide

45. What did we gain?
    

    View Slide

46. What did we gain?
    • Package installs will not clobber other packages
    

    View Slide

47. What did we gain?
    • Package installs will not clobber other packages
    • Trivial rollbacks and uninstalls
    

    View Slide

48. What did we gain?
    • Package installs will not clobber other packages
    • Trivial rollbacks and uninstalls
    • Deterministic, reproducible builds (up to OS ABI)
    

    View Slide

49. What did we gain?
    • Package installs will not clobber other packages
    • Trivial rollbacks and uninstalls
    • Deterministic, reproducible builds (up to OS ABI)
    • Safe:
    

    View Slide

50. What did we gain?
    • Package installs will not clobber other packages
    • Trivial rollbacks and uninstalls
    • Deterministic, reproducible builds (up to OS ABI)
    • Safe:
    • Sharing packages across users
    

    View Slide

51. What did we gain?
    • Package installs will not clobber other packages
    • Trivial rollbacks and uninstalls
    • Deterministic, reproducible builds (up to OS ABI)
    • Safe:
    • Sharing packages across users
    • Continuous integration
    

    View Slide

52. What did we gain?
    • Package installs will not clobber other packages
    • Trivial rollbacks and uninstalls
    • Deterministic, reproducible builds (up to OS ABI)
    • Safe:
    • Sharing packages across users
    • Continuous integration
    • Caching packages
    

    View Slide

53. View Slide

54. A
    v0.1
    C
    v0.1
    D
    v0.1
    B
    v0.1
    Ingress
    

    View Slide

55. A
    v0.1
    C
    v0.1
    D
    v0.1
    B
    v0.1
    A
    v0.2
    Ingress
    

    View Slide

56. A
    v0.2
    C
    v0.1
    D
    v0.1
    B
    v0.1
    Ingress
    

    View Slide

57. A
    v0.2
    C
    v0.1
    D
    v0.1
    B
    v0.1
    Ingress
    

    View Slide

58. C
    v0.1
    D
    v0.1
    B
    v0.1
    A
    v0.1
    Ingress
    A
    v0.2
    

    View Slide

59. C
    v0.1
    D
    v0.1
    B
    v0.1
    A
    v0.2
    Ingress
    A
    v0.1
    

    View Slide

60. B
    v0.1
    A
    v0.2
    Ingress
    C
    v0.1
    D
    v0.1
    A
    v0.1
    

    View Slide

61. B
    v0.1
    A
    v0.2
    Ingress
    B
    v0.2
    C
    v0.1
    D
    v0.1
    A
    v0.1
    

    View Slide

62. B
    v0.1
    A
    v0.2
    Ingress
    B
    v0.2
    C
    v0.1
    D
    v0.1
    A
    v0.1
    

    View Slide

63. B
    v0.1
    A
    v0.2
    Ingress
    B
    v0.2
    C
    v0.1
    D
    v0.1
    A
    v0.1
    

    View Slide

64. A
    v0.2
    Ingress
    B
    v0.2
    C
    v0.1
    D
    v0.1
    A
    v0.1
    B
    v0.1
    

    View Slide

65. A
    v0.2
    Ingress
    B
    v0.2
    C
    v0.1
    D
    v0.1
    A
    v0.1
    

    View Slide

66. A
    v0.2
    Ingress
    B
    v0.2
    C
    v0.1
    D
    v0.1
    

    View Slide

67. A
    v0.2
    Ingress
    B
    v0.2
    D
    v0.1
    

    View Slide

68. Nelson: Rigorous Deployment for a Functional World
    https://getnelson.github.io/
    

    View Slide

69. Nelson
    

    View Slide

70. Nelson
    • Continuous delivery system for immutable infrastructure
    

    View Slide

71. Nelson
    • Continuous delivery system for immutable infrastructure
    • Immutable infrastructure taken to its logical conclusion
    

    View Slide

72. Nelson
    • Continuous delivery system for immutable infrastructure
    • Immutable infrastructure taken to its logical conclusion
    • Tracks graph of service dependencies
    

    View Slide

73. Nelson
    • Continuous delivery system for immutable infrastructure
    • Immutable infrastructure taken to its logical conclusion
    • Tracks graph of service dependencies
    • Semantic versioning (MAJOR.MINOR.PATCH)
    

    View Slide

74. Nelson
    • Continuous delivery system for immutable infrastructure
    • Immutable infrastructure taken to its logical conclusion
    • Tracks graph of service dependencies
    • Semantic versioning (MAJOR.MINOR.PATCH)
    • Declare all dependencies
    

    View Slide

75. Nelson
    • Continuous delivery system for immutable infrastructure
    • Immutable infrastructure taken to its logical conclusion
    • Tracks graph of service dependencies
    • Semantic versioning (MAJOR.MINOR.PATCH)
    • Declare all dependencies
    • Ingress/load balancers and jobs form the “root” of the
    graph
    

    View Slide

76. View Slide

77. units:
    - name: conductor
    description: description
    dependencies:
    - ref: db.example@1.0
    ports:
    - default->9000/http
    loadbalancers:
    - name: lb
    routes:
    - name: expose default
    expose: default->9000/http
    destination: conductor->default
    

    View Slide

78. View Slide

79. Ingress
    OSN
    v0.1.0
    

    View Slide

80. News
    Feed
    v0.1.0
    Ingress
    OSN
    v0.1.0
    

    View Slide

81. News
    Feed
    v0.1.0
    Ingress
    OSN
    v0.1.0
    OSN
    v0.1.1
    

    View Slide

82. News
    Feed
    v0.1.0
    Ingress
    OSN
    v0.1.1
    

    View Slide

83. News
    Feed
    v0.1.0
    Ingress
    OSN
    v0.1.1
    

    View Slide

84. News
    Feed
    v0.1.0
    Ingress
    OSN
    v0.1.1
    Friends
    v0.1.0
    

    View Slide

85. News
    Feed
    v0.1.0
    Ingress
    OSN
    v0.1.1
    Friends
    v0.1.0
    OSN
    v0.1.2
    

    View Slide

86. News
    Feed
    v0.1.0
    Ingress
    Friends
    v0.1.0
    OSN
    v0.1.2
    

    View Slide

87. News
    Feed
    v0.1.0
    Ingress
    Friends
    v0.1.0
    OSN
    v0.1.2
    

    View Slide

88. News
    Feed
    v0.1.0
    Ingress
    Friends
    v0.1.0
    ML
    v0.1.0
    OSN
    v0.1.2
    

    View Slide

89. News
    Feed
    v0.1.0
    Ingress
    Friends
    v0.1.0
    ML
    v0.1.0
    Friends
    v0.2.0
    OSN
    v0.1.2
    

    View Slide

90. News
    Feed
    v0.1.0
    Ingress
    Friends
    v0.1.0
    ML
    v0.1.0
    Friends
    v0.2.0
    OSN
    v0.1.3
    OSN
    v0.1.2
    

    View Slide

91. News
    Feed
    v0.1.0
    Ingress
    Friends
    v0.1.0
    ML
    v0.1.0
    Friends
    v0.2.0
    OSN
    v0.1.3
    

    View Slide

92. News
    Feed
    v0.1.0
    Ingress
    ML
    v0.1.0
    Friends
    v0.2.0
    OSN
    v0.1.3
    

    View Slide

93. News
    Feed
    v0.1.0
    Ingress
    ML
    v0.1.0
    Friends
    v0.2.0
    OSN
    v0.1.3
    

    View Slide

94. News
    Feed
    v0.1.0
    Ingress
    ML
    v0.1.0
    Friends
    v0.2.0
    OSN
    v0.1.3
    Friends
    v0.3.0
    

    View Slide

95. News
    Feed
    v0.1.0
    Ingress
    ML
    v0.1.0
    Friends
    v0.2.0
    OSN
    v0.1.3
    Friends
    v0.3.0
    Msngr
    v0.1.0
    

    View Slide

96. News
    Feed
    v0.1.0
    Ingress
    ML
    v0.1.0
    Friends
    v0.2.0
    OSN
    v0.1.3
    Friends
    v0.3.0
    Msngr
    v0.1.0
    OSN
    v0.2.0
    

    View Slide

97. News
    Feed
    v0.1.0
    Ingress
    ML
    v0.1.0
    Friends
    v0.2.0
    Friends
    v0.3.0
    Msngr
    v0.1.0
    OSN
    v0.2.0
    

    View Slide

98. News
    Feed
    v0.1.0
    Ingress
    ML
    v0.1.0
    Friends
    v0.3.0
    Msngr
    v0.1.0
    OSN
    v0.2.0
    

    View Slide

99. News
    Feed
    v0.1.0
    Ingress
    Friends
    v0.3.0
    Msngr
    v0.1.0
    OSN
    v0.2.0
    

    View Slide

100. What did we gain?
     

     View Slide

101. What did we gain?
     • Deployments do not clobber other deployments
     

     View Slide

102. What did we gain?
     • Deployments do not clobber other deployments
     • Reverting a deployment is just bleeding traffic back
     

     View Slide

103. What did we gain?
     • Deployments do not clobber other deployments
     • Reverting a deployment is just bleeding traffic back
     • Fully automated application lifecycle
     

     View Slide

104. What did we gain?
     • Deployments do not clobber other deployments
     • Reverting a deployment is just bleeding traffic back
     • Fully automated application lifecycle
     • Clear distinction between service owners vs. infra
     

     View Slide

105. What did we gain?
     • Deployments do not clobber other deployments
     • Reverting a deployment is just bleeding traffic back
     • Fully automated application lifecycle
     • Clear distinction between service owners vs. infra
     • Engineering teams in charge of releasing, monitoring,
     etc.
     

     View Slide

106. What did we gain?
     • Deployments do not clobber other deployments
     • Reverting a deployment is just bleeding traffic back
     • Fully automated application lifecycle
     • Clear distinction between service owners vs. infra
     • Engineering teams in charge of releasing, monitoring,
     etc.
     • No build team, ticket filing for deployments, etc.
     

     View Slide

107. What did we gain?
     • Deployments do not clobber other deployments
     • Reverting a deployment is just bleeding traffic back
     • Fully automated application lifecycle
     • Clear distinction between service owners vs. infra
     • Engineering teams in charge of releasing, monitoring,
     etc.
     • No build team, ticket filing for deployments, etc.
     • Infra team focuses on infra!
     

     View Slide

108. Summary
     

     View Slide

109. Summary
     • Immutability, persistence, referential transparency, etc. are
     not just for code
     

     View Slide

110. Summary
     • Immutability, persistence, referential transparency, etc. are
     not just for code
     • See also: event sourcing, Datomic, categorical
     databases
     

     View Slide

111. Summary
     • Immutability, persistence, referential transparency, etc. are
     not just for code
     • See also: event sourcing, Datomic, categorical
     databases
     • Cross-pollination is good, FP ⇘ Systems
     

     View Slide

112. Summary
     • Immutability, persistence, referential transparency, etc. are
     not just for code
     • See also: event sourcing, Datomic, categorical
     databases
     • Cross-pollination is good, FP ⇘ Systems
     • PL and FP folks: learn about systems, machine learning,
     etc. and share your knowledge
     

     View Slide

113. Summary
     • Immutability, persistence, referential transparency, etc. are
     not just for code
     • See also: event sourcing, Datomic, categorical
     databases
     • Cross-pollination is good, FP ⇘ Systems
     • PL and FP folks: learn about systems, machine learning,
     etc. and share your knowledge
     • Systems folks: learn about programming languages, take
     ideas, build systems from solid foundations
     

     View Slide

114. Further Reading
     • Nix
     • Website: https://nixos.org/
     • Nix Pills: https://nixos.org/nixos/nix-pills/
     • Dr. Eelco Dolstra’s PhD thesis: https://nixos.org/~eelco/
     pubs/phd-thesis.pdf
     • Nelson
     • Website: https://getnelson.github.io/
     • SF Infra as Code talk: https://youtu.be/lTIvKZHedJQ
     • Scale by the Bay talk: https://youtu.be/3EHtAA4oE0k
     

     View Slide

115. EOF
     

     View Slide