Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Modularity in Software: A Retrospective

Modularity in Software: A Retrospective

Signify

May 28, 2020
Tweet

More Decks by Signify

Other Decks in Technology

Transcript

  1. [email protected]
    @deanwampler
    Modularity: A Retrospective

    View full-size slide

  2. • A little history
    • What is “modularity”? Why is it useful?
    • Modularity in Software:
    • What we’ve tried: the good and the bad
    • What we still need to do
    Outline

    View full-size slide

  3. @deanwampler
    A Little History

    View full-size slide

  4. @deanwampler
    • 1760s: French General Jean-
    Baptiste Vaquette de Gribeauval
    • Standardized cannon bores, shells
    • Eli Whitney - early 1800s
    • Muskets
    Interchangeable Parts
    https://en.wikipedia.org/wiki/Artillery
    https://en.wikipedia.org/wiki/Musket

    View full-size slide

  5. @deanwampler
    • Electronics
    • → PCs
    Interchangeable Parts
    https://fccid.io/NYYTP100/Block-Diagram/BLOCK-DIAGRAM-133927
    http://oldcomputers.net/ibm5150.html

    View full-size slide

  6. @deanwampler
    • Electronics
    • → PCs
    Interchangeable Parts
    https://mitpress.mit.edu/books/design-rules-volume-1
    http://claytonchristensen.com/books/the-innovators-dilemma/

    View full-size slide

  7. @deanwampler
    One Global Pattern…
    Duality of “Forces”

    View full-size slide

  8. @deanwampler
    • RNA evolves energy-stable
    structures that resist
    mutation
    • But mutation still possible
    In Biology
    https://mitpress.mit.edu/books/modularity
    Chapter 6, Evolutionary Lock-in and Origin of Modularity in RNA Structure

    View full-size slide

  9. @deanwampler
    • Exploit rewarding paths
    • … but explore other paths
    that might be more
    rewarding
    Reinforcement Learning
    https://mitpress.mit.edu/books/reinforcement-learning-second-edition

    View full-size slide

  10. @deanwampler
    • A module should open for
    extension
    • … but closed for modification
    Open-Closed Principle
    Wikipedia Page

    View full-size slide

  11. @deanwampler
    • All systems have design
    patterns, including FP
    • Specific patterns will vary

    FP “patterns”? Monads, monoids,
    applicatives, functors, …

    View full-size slide

  12. @deanwampler
    Modularity Defined

    View full-size slide

  13. @deanwampler
    A module is a unit whose
    structural elements are
    powerfully connected among
    themselves and relatively weakly
    connected to elements in other
    units.
    Definition
    I.e., Cohesion vs. Coupling

    View full-size slide

  14. @deanwampler
    • Design Rules - fixed decisions
    • Architecture - the pieces & how
    they are wired together
    • Interfaces - How they talk to
    each other
    Terms

    View full-size slide

  15. @deanwampler
    • Allows greater complexity
    • Allows concurrent work
    • Embrace Conway’s Law
    Benefits
    Wikipedia Page

    View full-size slide

  16. @deanwampler
    • Accomodates uncertainty
    • the flexibility to evolve
    internally
    Benefits

    View full-size slide

  17. @deanwampler
    • Not stated in the book are “DevOps”
    concerns, e.g.,
    • Independent scalability
    • Independent replacement
    Benefits

    View full-size slide

  18. @deanwampler
    Insights
    • Modules can be
    • structural
    • procedural

    View full-size slide

  19. @deanwampler
    Insights
    • Procedures =
    Libraries of components
    + Scripts as glue for
    procedures
    What is the type of a procedure??

    View full-size slide

  20. @deanwampler
    Software

    View full-size slide

  21. @deanwampler
    Examples
    • Microservices?
    • DLL Hell…
    • OOP vs. FP vs. … what?
    • New Modularities

    View full-size slide

  22. @deanwampler
    Microservices

    View full-size slide

  23. @deanwampler
    https://corecursive.com/045-david-heinemeier-hansson-software-contrarian/
    Microservices

    View full-size slide

  24. @deanwampler
    Microservices
    • Why Microservices?
    • Conway’s Law
    • Independent scaling, etc.

    View full-size slide

  25. @deanwampler
    Istio
    https://istio.io/
    • Abandoned Microservices?
    • Because most deployments
    have one administrator
    • They now release as one
    deployable service

    View full-size slide

  26. @deanwampler
    DLL hell

    View full-size slide

  27. @deanwampler
    DLL hell
    • Is semantic versioning really
    enough?

    View full-size slide

  28. @deanwampler
    DLL hell
    • Clojure - once you release a
    function, type, etc., never
    change it.
    • Can we define more rigorous
    specifications of interfaces?

    View full-size slide

  29. @deanwampler
    DLL hell
    • Unison language
    • Why do we rely on names?
    https://www.unisonweb.org/2020/04/10/reducing-churn/

    View full-size slide

  30. @deanwampler
    OOP vs. FP vs. ??

    View full-size slide

  31. @deanwampler
    OOP vs. FP vs. ??
    • OOP - Why didn’t we get true
    modularity and reusability?
    • Inheritance too fragile
    • Behavior not sequential
    • Lack of constraints…

    View full-size slide

  32. @deanwampler
    Constraints

    View full-size slide

  33. @deanwampler
    https://www.youtube.com/watch?v=GqmsQeSzMdw
    Constraints

    View full-size slide

  34. @deanwampler
    Constraints

    View full-size slide

  35. @deanwampler
    Constraints

    View full-size slide

  36. @deanwampler
    Constraints
    def foo(array: Array[Int]): Int
    def foo[A](array: Array[A]): Int
    vs.

    View full-size slide

  37. @deanwampler
    Constraints
    • Side effects? - they undermine
    • reuse
    • compositionality
    • concurrency
    • “algebraic reasoning”

    View full-size slide

  38. @deanwampler
    Success Stories

    View full-size slide

  39. @deanwampler
    Databases vs. Hadoop

    View full-size slide

  40. @deanwampler
    submit to…
    YARN
    HDFS
    MapReduce jobs
    Spark jobs

    Worker Node #1
    Disk
    Disk
    Disk
    Disk
    Disk
    Node
    Manager
    Data Node
    Master Node
    Resource
    Manager
    Name Node

    #2
    Databases vs. Hadoop

    View full-size slide

  41. @deanwampler
    submit to…
    YARN
    HDFS
    MapReduce jobs
    Spark jobs

    Worker Node #1
    Disk
    Disk
    Disk
    Disk
    Disk
    Node
    Manager
    Data Node
    Master Node
    Resource
    Manager
    Name Node

    #2
    Database
    Deconstructed!
    Databases vs. Hadoop

    View full-size slide

  42. @deanwampler
    Ray

    View full-size slide

  43. @deanwampler
    Microservices (again)
    REST
    API Gateway
    µ-service 1
    µ-service 2
    µ-service 3

    View full-size slide

  44. @deanwampler
    Microservices (again)
    Need to Scale??
    Why do YOU have to
    manage all these
    instances yourself??
    REST
    API Gateway
    µ-service 1
    µ-service 2
    µ-service 3
    µ-service 1
    µ-service 2
    µ-service 3
    µ-service 1
    µ-service 2
    µ-service 3
    µ-service 1
    µ-service 2
    µ-service 1

    View full-size slide

  45. @deanwampler
    Ray
    • Pretend you have ONE
    instance of the
    microservice.
    • Let systems like Ray
    manage horizontal
    scaling.
    ray.io
    Ray Cluster
    task/
    actor
    task/
    actor
    task/
    actor
    task/
    actor
    task/
    actor
    task/
    actor
    task/
    actor
    task/
    actor
    task/
    actor
    task/
    actor
    task/
    actor
    task/
    actor
    task/
    actor
    task/
    actor
    REST
    API Gateway
    µ-service 1
    µ-service 2
    µ-service 3

    View full-size slide

  46. @deanwampler
    Apache Arrow
    • Standardize in-memory
    data
    • Storage, schema, …
    • Reuse across apps & libs
    arrow.apache.org

    View full-size slide

  47. @deanwampler
    polyglotprogramming.com/talks
    anyscale.com
    ray.io

    View full-size slide