Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Persistent Data Structures in System Design

Persistent Data Structures in System Design

Persistent data structures are a powerful tool in the functional programming toolbox, allowing us to work with changing data without sacrificing referential transparency. However the use of persistent data structures often stops with the program, the resulting code flung over the wall to be built and deployed in non-referentially transparent systems. In this talk we will see that it doesn't have to be this way and explore system designs that leverage the techniques of persistent data structures to recover referential transparency in the ever changing world of builds and deployments.

Adelbert Chang

January 09, 2018
Tweet

More Decks by Adelbert Chang

Other Decks in Programming

Transcript

  1. Persistent Data Structures
    in System Design
    Adelbert Chang
    @adelbertchang
    East Bay Haskell Meetup, January 9, 2018

    View full-size slide

  2. B
    A
    C D
    E F G H

    View full-size slide

  3. B
    A
    C D
    E F G H

    View full-size slide

  4. B
    A
    C D
    E F G H
    D’
    I

    View full-size slide

  5. B
    A
    C D
    E F G H
    A’
    D’
    I

    View full-size slide

  6. B C
    E F G H
    A’
    D’
    I

    View full-size slide

  7. Properties of persistent data structures

    View full-size slide

  8. Properties of persistent data structures
    • Non-destructive updates

    View full-size slide

  9. Properties of persistent data structures
    • Non-destructive updates
    • Explicit dependency tracking

    View full-size slide

  10. Properties of persistent data structures
    • Non-destructive updates
    • Explicit dependency tracking
    • Structural sharing

    View full-size slide

  11. Properties of persistent data structures
    • Non-destructive updates
    • Explicit dependency tracking
    • Structural sharing
    • Substructures unaware of the larger whole

    View full-size slide

  12. Properties of persistent data structures
    • Non-destructive updates
    • Explicit dependency tracking
    • Structural sharing
    • Substructures unaware of the larger whole
    • Automatic space/memory management

    View full-size slide

  13. A v0.1
    C v0.1
    B v0.1
    Installed packages
    D v0.1

    View full-size slide

  14. A v0.2
    C v0.2
    B v0.1
    Installed packages
    D v0.1

    View full-size slide

  15. A v0.2
    C v0.2
    B v0.1
    Installed packages
    D v0.1

    View full-size slide

  16. B v0.1
    Installed packages
    A v0.1
    C v0.1 D v0.1

    View full-size slide

  17. B v0.1
    Installed packages
    A v0.1
    C v0.1 D v0.1
    A v0.2
    C v0.2

    View full-size slide

  18. B v0.1
    Installed packages
    A v0.1
    C v0.1 D v0.1
    A v0.2
    C v0.2
    B v0.2

    View full-size slide

  19. Installed packages
    C v0.1 D v0.1
    A v0.1
    A v0.2
    C v0.2
    B v0.2
    B v0.1

    View full-size slide

  20. Installed packages
    C v0.1 D v0.1
    A v0.1
    A v0.2
    C v0.2
    B v0.2

    View full-size slide

  21. Installed packages
    C v0.1 D v0.1
    A v0.2
    C v0.2
    B v0.2

    View full-size slide

  22. Installed packages
    D v0.1
    A v0.2
    C v0.2
    B v0.2

    View full-size slide

  23. Nix: The Purely Functional Package Manager
    https://nixos.org/nix/

    View full-size slide

  24. Nix
    • Persistent, referentially transparent build tool/package
    management

    View full-size slide

  25. Nix
    • Persistent, referentially transparent build tool/package
    management
    • Tracks graph of packages and their dependencies

    View full-size slide

  26. Nix
    • Persistent, referentially transparent build tool/package
    management
    • Tracks graph of packages and their dependencies
    • Packages are built from source

    View full-size slide

  27. Nix
    • Persistent, referentially transparent build tool/package
    management
    • Tracks graph of packages and their dependencies
    • Packages are built from source
    • Builds specified with derivations in Nix, a pure, lazy,
    functional language

    View full-size slide

  28. Nix
    • Persistent, referentially transparent build tool/package
    management
    • Tracks graph of packages and their dependencies
    • Packages are built from source
    • Builds specified with derivations in Nix, a pure, lazy,
    functional language
    • Uses lot of tricks top revent relying on PATH, HOME, etc.

    View full-size slide

  29. Nix
    • Persistent, referentially transparent build tool/package
    management
    • Tracks graph of packages and their dependencies
    • Packages are built from source
    • Builds specified with derivations in Nix, a pure, lazy,
    functional language
    • Uses lot of tricks top revent relying on PATH, HOME, etc.
    • Packages installed in a directory based on hash of its
    derivation

    View full-size slide

  30. { stdenv, fetchFromGitHub, caddy, asciidoctor
    , file, lessc, sass, multimarkdown, linkchecker
    , perlPackages, python27 }:
    stdenv.mkDerivation rec {
    name = "styx-${version}";
    version = "0.6.0";
    . . .
    installPhase = ''
    mkdir $out
    install -D -m 777 styx.sh $out/bin/styx
    mkdir -p $out/share/styx
    cp -r scaffold $out/share/styx
    cp -r nix $out/share/styx
    mkdir -p $out/share/doc/styx
    asciidoctor doc/index.adoc -o $out/share/doc/styx/index.html
    https://github.com/NixOS/nixpkgs/blob/1e95402803adf28bd4d78403fa9e9dcccff12101/pkgs/applications/misc/styx/default.nix

    View full-size slide

  31. { stdenv, fetchFromGitHub, caddy, asciidoctor
    , file, lessc, sass, multimarkdown, linkchecker
    , perlPackages, python27 }:
    stdenv.mkDerivation rec {
    name = "styx-${version}";
    version = "0.6.0";
    . . .
    installPhase = ''
    mkdir $out
    install -D -m 777 styx.sh $out/bin/styx
    mkdir -p $out/share/styx
    cp -r scaffold $out/share/styx
    cp -r nix $out/share/styx
    mkdir -p $out/share/doc/styx
    asciidoctor doc/index.adoc -o $out/share/doc/styx/index.html
    https://github.com/NixOS/nixpkgs/blob/1e95402803adf28bd4d78403fa9e9dcccff12101/pkgs/applications/misc/styx/default.nix

    View full-size slide

  32. { stdenv, fetchFromGitHub, caddy, asciidoctor
    , file, lessc, sass, multimarkdown, linkchecker
    , perlPackages, python27 }:
    stdenv.mkDerivation rec {
    name = "styx-${version}";
    version = "0.6.0";
    . . .
    installPhase = ''
    mkdir $out
    install -D -m 777 styx.sh $out/bin/styx
    mkdir -p $out/share/styx
    cp -r scaffold $out/share/styx
    cp -r nix $out/share/styx
    mkdir -p $out/share/doc/styx
    asciidoctor doc/index.adoc -o $out/share/doc/styx/index.html
    https://github.com/NixOS/nixpkgs/blob/1e95402803adf28bd4d78403fa9e9dcccff12101/pkgs/applications/misc/styx/default.nix

    View full-size slide

  33. /nix/store
    |— 0c1p5z4kda11…-user-env
    | |— bin
    | |— svn
    |— 3aw2pdyx2jfc…-user-env
    | |— bin
    | |— firefox
    | |— svn
    |— ihvrn5awaw7y…-user-env
    | |— bin
    | |— firefox
    | |— svn
    |— 5mq2jcn36ldl…-subversion-1.1.2
    | |— bin
    | |— svn
    |— g32imf68vvbw…-firefox-1.0.1
    | |— bin
    | |— firefox
    |— dpmvp969yhdq…-subversion-1.1.3
    | |— bin
    | |— svn
    https://nixos.org/nix/manual/#sec-profiles
    /nix/var/nix/profiles
    |— default
    |— default-0-link
    |— default-1-link
    |— default-2-link

    View full-size slide

  34. /nix/store
    |— 0c1p5z4kda11…-user-env
    | |— bin
    | |— svn
    |— 3aw2pdyx2jfc…-user-env
    | |— bin
    | |— firefox
    | |— svn
    |— ihvrn5awaw7y…-user-env
    | |— bin
    | |— firefox
    | |— svn
    |— 5mq2jcn36ldl…-subversion-1.1.2
    | |— bin
    | |— svn
    |— g32imf68vvbw…-firefox-1.0.1
    | |— bin
    | |— firefox
    |— dpmvp969yhdq…-subversion-1.1.3
    | |— bin
    | |— svn
    https://nixos.org/nix/manual/#sec-profiles
    /nix/var/nix/profiles
    |— default
    |— default-0-link
    |— default-1-link
    |— default-2-link

    View full-size slide

  35. /nix/store
    |— 0c1p5z4kda11…-user-env
    | |— bin
    | |— svn
    |— 3aw2pdyx2jfc…-user-env
    | |— bin
    | |— firefox
    | |— svn
    |— ihvrn5awaw7y…-user-env
    | |— bin
    | |— firefox
    | |— svn
    |— 5mq2jcn36ldl…-subversion-1.1.2
    | |— bin
    | |— svn
    |— g32imf68vvbw…-firefox-1.0.1
    | |— bin
    | |— firefox
    |— dpmvp969yhdq…-subversion-1.1.3
    | |— bin
    | |— svn
    https://nixos.org/nix/manual/#sec-profiles
    /nix/var/nix/profiles
    |— default
    |— default-0-link
    |— default-1-link
    |— default-2-link

    View full-size slide

  36. /nix/store
    |— 0c1p5z4kda11…-user-env
    | |— bin
    | |— svn
    |— 3aw2pdyx2jfc…-user-env
    | |— bin
    | |— firefox
    | |— svn
    |— ihvrn5awaw7y…-user-env
    | |— bin
    | |— firefox
    | |— svn
    |— 5mq2jcn36ldl…-subversion-1.1.2
    | |— bin
    | |— svn
    |— g32imf68vvbw…-firefox-1.0.1
    | |— bin
    | |— firefox
    |— dpmvp969yhdq…-subversion-1.1.3
    | |— bin
    | |— svn
    https://nixos.org/nix/manual/#sec-profiles
    /nix/var/nix/profiles
    |— default
    |— default-0-link
    |— default-1-link
    |— default-2-link

    View full-size slide

  37. /nix/store
    |— 0c1p5z4kda11…-user-env
    | |— bin
    | |— svn
    |— 3aw2pdyx2jfc…-user-env
    | |— bin
    | |— firefox
    | |— svn
    |— ihvrn5awaw7y…-user-env
    | |— bin
    | |— firefox
    | |— svn
    |— 5mq2jcn36ldl…-subversion-1.1.2
    | |— bin
    | |— svn
    |— g32imf68vvbw…-firefox-1.0.1
    | |— bin
    | |— firefox
    |— dpmvp969yhdq…-subversion-1.1.3
    | |— bin
    | |— svn
    https://nixos.org/nix/manual/#sec-profiles
    /nix/var/nix/profiles
    |— default
    |— default-0-link
    |— default-1-link

    View full-size slide

  38. /nix/store
    |— 0c1p5z4kda11…-user-env
    | |— bin
    | |— svn
    |— 3aw2pdyx2jfc…-user-env
    | |— bin
    | |— firefox
    | |— svn
    |— ihvrn5awaw7y…-user-env
    | |— bin
    | |— firefox
    | |— svn
    |— 5mq2jcn36ldl…-subversion-1.1.2
    | |— bin
    | |— svn
    |— g32imf68vvbw…-firefox-1.0.1
    | |— bin
    | |— firefox
    |— dpmvp969yhdq…-subversion-1.1.3
    | |— bin
    | |— svn
    https://nixos.org/nix/manual/#sec-profiles
    /nix/var/nix/profiles
    |— default
    |— default-0-link
    |— default-1-link

    View full-size slide

  39. What did we gain?

    View full-size slide

  40. What did we gain?
    • Package installs will not clobber other packages

    View full-size slide

  41. What did we gain?
    • Package installs will not clobber other packages
    • Trivial rollbacks and uninstalls

    View full-size slide

  42. What did we gain?
    • Package installs will not clobber other packages
    • Trivial rollbacks and uninstalls
    • Deterministic, reproducible builds (up to OS ABI)

    View full-size slide

  43. What did we gain?
    • Package installs will not clobber other packages
    • Trivial rollbacks and uninstalls
    • Deterministic, reproducible builds (up to OS ABI)
    • Safe:

    View full-size slide

  44. What did we gain?
    • Package installs will not clobber other packages
    • Trivial rollbacks and uninstalls
    • Deterministic, reproducible builds (up to OS ABI)
    • Safe:
    • Sharing packages across users

    View full-size slide

  45. What did we gain?
    • Package installs will not clobber other packages
    • Trivial rollbacks and uninstalls
    • Deterministic, reproducible builds (up to OS ABI)
    • Safe:
    • Sharing packages across users
    • Continuous integration

    View full-size slide

  46. What did we gain?
    • Package installs will not clobber other packages
    • Trivial rollbacks and uninstalls
    • Deterministic, reproducible builds (up to OS ABI)
    • Safe:
    • Sharing packages across users
    • Continuous integration
    • Caching packages

    View full-size slide

  47. A
    v0.1
    C
    v0.1
    D
    v0.1
    B
    v0.1
    Ingress

    View full-size slide

  48. A
    v0.1
    C
    v0.1
    D
    v0.1
    B
    v0.1
    A
    v0.2
    Ingress

    View full-size slide

  49. A
    v0.2
    C
    v0.1
    D
    v0.1
    B
    v0.1
    Ingress

    View full-size slide

  50. A
    v0.2
    C
    v0.1
    D
    v0.1
    B
    v0.1
    Ingress

    View full-size slide

  51. C
    v0.1
    D
    v0.1
    B
    v0.1
    A
    v0.1
    Ingress
    A
    v0.2

    View full-size slide

  52. C
    v0.1
    D
    v0.1
    B
    v0.1
    A
    v0.2
    Ingress
    A
    v0.1

    View full-size slide

  53. B
    v0.1
    A
    v0.2
    Ingress
    C
    v0.1
    D
    v0.1
    A
    v0.1

    View full-size slide

  54. B
    v0.1
    A
    v0.2
    Ingress
    B
    v0.2
    C
    v0.1
    D
    v0.1
    A
    v0.1

    View full-size slide

  55. B
    v0.1
    A
    v0.2
    Ingress
    B
    v0.2
    C
    v0.1
    D
    v0.1
    A
    v0.1

    View full-size slide

  56. B
    v0.1
    A
    v0.2
    Ingress
    B
    v0.2
    C
    v0.1
    D
    v0.1
    A
    v0.1

    View full-size slide

  57. A
    v0.2
    Ingress
    B
    v0.2
    C
    v0.1
    D
    v0.1
    A
    v0.1
    B
    v0.1

    View full-size slide

  58. A
    v0.2
    Ingress
    B
    v0.2
    C
    v0.1
    D
    v0.1
    A
    v0.1

    View full-size slide

  59. A
    v0.2
    Ingress
    B
    v0.2
    C
    v0.1
    D
    v0.1

    View full-size slide

  60. A
    v0.2
    Ingress
    B
    v0.2
    D
    v0.1

    View full-size slide

  61. Nelson: Rigorous Deployment for a Functional World
    https://getnelson.github.io/

    View full-size slide

  62. Nelson
    • Continuous delivery system for immutable infrastructure

    View full-size slide

  63. Nelson
    • Continuous delivery system for immutable infrastructure
    • Immutable infrastructure taken to its logical conclusion

    View full-size slide

  64. Nelson
    • Continuous delivery system for immutable infrastructure
    • Immutable infrastructure taken to its logical conclusion
    • Tracks graph of service dependencies

    View full-size slide

  65. Nelson
    • Continuous delivery system for immutable infrastructure
    • Immutable infrastructure taken to its logical conclusion
    • Tracks graph of service dependencies
    • Semantic versioning (MAJOR.MINOR.PATCH)

    View full-size slide

  66. Nelson
    • Continuous delivery system for immutable infrastructure
    • Immutable infrastructure taken to its logical conclusion
    • Tracks graph of service dependencies
    • Semantic versioning (MAJOR.MINOR.PATCH)
    • Declare all dependencies

    View full-size slide

  67. Nelson
    • Continuous delivery system for immutable infrastructure
    • Immutable infrastructure taken to its logical conclusion
    • Tracks graph of service dependencies
    • Semantic versioning (MAJOR.MINOR.PATCH)
    • Declare all dependencies
    • Ingress/load balancers and jobs form the “root” of the
    graph

    View full-size slide

  68. units:
    - name: conductor
    description: description
    dependencies:
    - ref: [email protected]
    ports:
    - default->9000/http
    loadbalancers:
    - name: lb
    routes:
    - name: expose default
    expose: default->9000/http
    destination: conductor->default

    View full-size slide

  69. Ingress
    OSN
    v0.1.0

    View full-size slide

  70. News
    Feed
    v0.1.0
    Ingress
    OSN
    v0.1.0

    View full-size slide

  71. News
    Feed
    v0.1.0
    Ingress
    OSN
    v0.1.0
    OSN
    v0.1.1

    View full-size slide

  72. News
    Feed
    v0.1.0
    Ingress
    OSN
    v0.1.1

    View full-size slide

  73. News
    Feed
    v0.1.0
    Ingress
    OSN
    v0.1.1

    View full-size slide

  74. News
    Feed
    v0.1.0
    Ingress
    OSN
    v0.1.1
    Friends
    v0.1.0

    View full-size slide

  75. News
    Feed
    v0.1.0
    Ingress
    OSN
    v0.1.1
    Friends
    v0.1.0
    OSN
    v0.1.2

    View full-size slide

  76. News
    Feed
    v0.1.0
    Ingress
    Friends
    v0.1.0
    OSN
    v0.1.2

    View full-size slide

  77. News
    Feed
    v0.1.0
    Ingress
    Friends
    v0.1.0
    OSN
    v0.1.2

    View full-size slide

  78. News
    Feed
    v0.1.0
    Ingress
    Friends
    v0.1.0
    ML
    v0.1.0
    OSN
    v0.1.2

    View full-size slide

  79. News
    Feed
    v0.1.0
    Ingress
    Friends
    v0.1.0
    ML
    v0.1.0
    Friends
    v0.2.0
    OSN
    v0.1.2

    View full-size slide

  80. News
    Feed
    v0.1.0
    Ingress
    Friends
    v0.1.0
    ML
    v0.1.0
    Friends
    v0.2.0
    OSN
    v0.1.3
    OSN
    v0.1.2

    View full-size slide

  81. News
    Feed
    v0.1.0
    Ingress
    Friends
    v0.1.0
    ML
    v0.1.0
    Friends
    v0.2.0
    OSN
    v0.1.3

    View full-size slide

  82. News
    Feed
    v0.1.0
    Ingress
    ML
    v0.1.0
    Friends
    v0.2.0
    OSN
    v0.1.3

    View full-size slide

  83. News
    Feed
    v0.1.0
    Ingress
    ML
    v0.1.0
    Friends
    v0.2.0
    OSN
    v0.1.3

    View full-size slide

  84. News
    Feed
    v0.1.0
    Ingress
    ML
    v0.1.0
    Friends
    v0.2.0
    OSN
    v0.1.3
    Friends
    v0.3.0

    View full-size slide

  85. News
    Feed
    v0.1.0
    Ingress
    ML
    v0.1.0
    Friends
    v0.2.0
    OSN
    v0.1.3
    Friends
    v0.3.0
    Msngr
    v0.1.0

    View full-size slide

  86. News
    Feed
    v0.1.0
    Ingress
    ML
    v0.1.0
    Friends
    v0.2.0
    OSN
    v0.1.3
    Friends
    v0.3.0
    Msngr
    v0.1.0
    OSN
    v0.2.0

    View full-size slide

  87. News
    Feed
    v0.1.0
    Ingress
    ML
    v0.1.0
    Friends
    v0.2.0
    Friends
    v0.3.0
    Msngr
    v0.1.0
    OSN
    v0.2.0

    View full-size slide

  88. News
    Feed
    v0.1.0
    Ingress
    ML
    v0.1.0
    Friends
    v0.3.0
    Msngr
    v0.1.0
    OSN
    v0.2.0

    View full-size slide

  89. News
    Feed
    v0.1.0
    Ingress
    Friends
    v0.3.0
    Msngr
    v0.1.0
    OSN
    v0.2.0

    View full-size slide

  90. What did we gain?

    View full-size slide

  91. What did we gain?
    • Deployments do not clobber other deployments

    View full-size slide

  92. What did we gain?
    • Deployments do not clobber other deployments
    • Reverting a deployment is just bleeding traffic back

    View full-size slide

  93. What did we gain?
    • Deployments do not clobber other deployments
    • Reverting a deployment is just bleeding traffic back
    • Fully automated application lifecycle

    View full-size slide

  94. What did we gain?
    • Deployments do not clobber other deployments
    • Reverting a deployment is just bleeding traffic back
    • Fully automated application lifecycle
    • Clear distinction between service owners vs. infra

    View full-size slide

  95. What did we gain?
    • Deployments do not clobber other deployments
    • Reverting a deployment is just bleeding traffic back
    • Fully automated application lifecycle
    • Clear distinction between service owners vs. infra
    • Engineering teams in charge of releasing, monitoring,
    etc.

    View full-size slide

  96. What did we gain?
    • Deployments do not clobber other deployments
    • Reverting a deployment is just bleeding traffic back
    • Fully automated application lifecycle
    • Clear distinction between service owners vs. infra
    • Engineering teams in charge of releasing, monitoring,
    etc.
    • No build team, ticket filing for deployments, etc.

    View full-size slide

  97. What did we gain?
    • Deployments do not clobber other deployments
    • Reverting a deployment is just bleeding traffic back
    • Fully automated application lifecycle
    • Clear distinction between service owners vs. infra
    • Engineering teams in charge of releasing, monitoring,
    etc.
    • No build team, ticket filing for deployments, etc.
    • Infra team focuses on infra!

    View full-size slide

  98. Summary
    • Immutability, persistence, referential transparency, etc. are
    not just for code

    View full-size slide

  99. Summary
    • Immutability, persistence, referential transparency, etc. are
    not just for code
    • See also: event sourcing, Datomic, categorical
    databases

    View full-size slide

  100. Summary
    • Immutability, persistence, referential transparency, etc. are
    not just for code
    • See also: event sourcing, Datomic, categorical
    databases
    • Cross-pollination is good, FP ⇘ Systems

    View full-size slide

  101. Summary
    • Immutability, persistence, referential transparency, etc. are
    not just for code
    • See also: event sourcing, Datomic, categorical
    databases
    • Cross-pollination is good, FP ⇘ Systems
    • PL and FP folks: learn about systems, machine learning,
    etc. and share your knowledge

    View full-size slide

  102. Summary
    • Immutability, persistence, referential transparency, etc. are
    not just for code
    • See also: event sourcing, Datomic, categorical
    databases
    • Cross-pollination is good, FP ⇘ Systems
    • PL and FP folks: learn about systems, machine learning,
    etc. and share your knowledge
    • Systems folks: learn about programming languages, take
    ideas, build systems from solid foundations

    View full-size slide

  103. Further Reading
    • Nix
    • Website: https://nixos.org/
    • Nix Pills: https://nixos.org/nixos/nix-pills/
    • Dr. Eelco Dolstra’s PhD thesis: https://nixos.org/~eelco/
    pubs/phd-thesis.pdf
    • Nelson
    • Website: https://getnelson.github.io/
    • SF Infra as Code talk: https://youtu.be/lTIvKZHedJQ
    • Scale by the Bay talk: https://youtu.be/3EHtAA4oE0k

    View full-size slide