Modularity in Software: A Retrospective

by Signify

Slide 1

Slide 1 text

[email protected] @deanwampler Modularity: A Retrospective

Slide 2

Slide 2 text

• A little history • What is “modularity”? Why is it useful? • Modularity in Software: • What we’ve tried: the good and the bad • What we still need to do Outline

Slide 3

Slide 3 text

@deanwampler A Little History

Slide 4

Slide 4 text

@deanwampler • 1760s: French General Jean- Baptiste Vaquette de Gribeauval • Standardized cannon bores, shells • Eli Whitney - early 1800s • Muskets Interchangeable Parts https://en.wikipedia.org/wiki/Artillery https://en.wikipedia.org/wiki/Musket

Slide 5

Slide 5 text

@deanwampler • Electronics • → PCs Interchangeable Parts https://fccid.io/NYYTP100/Block-Diagram/BLOCK-DIAGRAM-133927 http://oldcomputers.net/ibm5150.html

Slide 6

Slide 6 text

@deanwampler • Electronics • → PCs Interchangeable Parts https://mitpress.mit.edu/books/design-rules-volume-1 http://claytonchristensen.com/books/the-innovators-dilemma/

Slide 7

Slide 7 text

@deanwampler One Global Pattern… Duality of “Forces”

Slide 8

Slide 8 text

@deanwampler • RNA evolves energy-stable structures that resist mutation • But mutation still possible In Biology https://mitpress.mit.edu/books/modularity Chapter 6, Evolutionary Lock-in and Origin of Modularity in RNA Structure

Slide 9

Slide 9 text

@deanwampler • Exploit rewarding paths • … but explore other paths that might be more rewarding Reinforcement Learning https://mitpress.mit.edu/books/reinforcement-learning-second-edition

Slide 10

Slide 10 text

@deanwampler • A module should open for extension • … but closed for modification Open-Closed Principle Wikipedia Page

Slide 11

Slide 11 text

@deanwampler • All systems have design patterns, including FP • Specific patterns will vary FP “patterns”? Monads, monoids, applicatives, functors, …

Slide 12

Slide 12 text

@deanwampler Modularity Defined

Slide 13

Slide 13 text

@deanwampler A module is a unit whose structural elements are powerfully connected among themselves and relatively weakly connected to elements in other units. Definition I.e., Cohesion vs. Coupling

Slide 14

Slide 14 text

@deanwampler • Design Rules - fixed decisions • Architecture - the pieces & how they are wired together • Interfaces - How they talk to each other Terms

Slide 15

Slide 15 text

@deanwampler • Allows greater complexity • Allows concurrent work • Embrace Conway’s Law Benefits Wikipedia Page

Slide 16

Slide 16 text

@deanwampler • Accomodates uncertainty • the flexibility to evolve internally Benefits

Slide 17

Slide 17 text

@deanwampler • Not stated in the book are “DevOps” concerns, e.g., • Independent scalability • Independent replacement Benefits

Slide 18

Slide 18 text

@deanwampler Insights • Modules can be • structural • procedural

Slide 19

Slide 19 text

@deanwampler Insights • Procedures = Libraries of components + Scripts as glue for procedures What is the type of a procedure??

Slide 20

Slide 20 text

@deanwampler Software

Slide 21

Slide 21 text

@deanwampler Examples • Microservices? • DLL Hell… • OOP vs. FP vs. … what? • New Modularities

Slide 22

Slide 22 text

@deanwampler Microservices

Slide 23

Slide 23 text

@deanwampler https://corecursive.com/045-david-heinemeier-hansson-software-contrarian/ Microservices

Slide 24

Slide 24 text

@deanwampler Microservices • Why Microservices? • Conway’s Law • Independent scaling, etc.

Slide 25

Slide 25 text

@deanwampler Istio https://istio.io/ • Abandoned Microservices? • Because most deployments have one administrator • They now release as one deployable service

Slide 26

Slide 26 text

@deanwampler DLL hell

Slide 27

Slide 27 text

@deanwampler DLL hell • Is semantic versioning really enough?

Slide 28

Slide 28 text

@deanwampler DLL hell • Clojure - once you release a function, type, etc., never change it. • Can we define more rigorous specifications of interfaces?

Slide 29

Slide 29 text

@deanwampler DLL hell • Unison language • Why do we rely on names? https://www.unisonweb.org/2020/04/10/reducing-churn/

Slide 30

Slide 30 text

@deanwampler OOP vs. FP vs. ??

Slide 31

Slide 31 text

@deanwampler OOP vs. FP vs. ?? • OOP - Why didn’t we get true modularity and reusability? • Inheritance too fragile • Behavior not sequential • Lack of constraints…

Slide 32

Slide 32 text

@deanwampler Constraints

Slide 33

Slide 33 text

@deanwampler https://www.youtube.com/watch?v=GqmsQeSzMdw Constraints

Slide 34

Slide 34 text

@deanwampler Constraints

Slide 35

Slide 35 text

@deanwampler Constraints

Slide 36

Slide 36 text

@deanwampler Constraints def foo(array: Array[Int]): Int def foo[A](array: Array[A]): Int vs.

Slide 37

Slide 37 text

@deanwampler Constraints • Side effects? - they undermine • reuse • compositionality • concurrency • “algebraic reasoning”

Slide 38

Slide 38 text

@deanwampler Success Stories

Slide 39

Slide 39 text

@deanwampler Databases vs. Hadoop

Slide 40

Slide 40 text

@deanwampler submit to… YARN HDFS MapReduce jobs Spark jobs … Worker Node #1 Disk Disk Disk Disk Disk Node Manager Data Node Master Node Resource Manager Name Node … #2 Databases vs. Hadoop

Slide 41

Slide 41 text

@deanwampler submit to… YARN HDFS MapReduce jobs Spark jobs … Worker Node #1 Disk Disk Disk Disk Disk Node Manager Data Node Master Node Resource Manager Name Node … #2 Database Deconstructed! Databases vs. Hadoop

Slide 42

Slide 42 text

@deanwampler Ray

Slide 43

Slide 43 text

@deanwampler Microservices (again) REST API Gateway µ-service 1 µ-service 2 µ-service 3

Slide 44

Slide 44 text

@deanwampler Microservices (again) Need to Scale?? Why do YOU have to manage all these instances yourself?? REST API Gateway µ-service 1 µ-service 2 µ-service 3 µ-service 1 µ-service 2 µ-service 3 µ-service 1 µ-service 2 µ-service 3 µ-service 1 µ-service 2 µ-service 1

Slide 45

Slide 45 text

@deanwampler Ray • Pretend you have ONE instance of the microservice. • Let systems like Ray manage horizontal scaling. ray.io Ray Cluster task/ actor task/ actor task/ actor task/ actor task/ actor task/ actor task/ actor task/ actor task/ actor task/ actor task/ actor task/ actor task/ actor task/ actor REST API Gateway µ-service 1 µ-service 2 µ-service 3

Slide 46

Slide 46 text

@deanwampler Apache Arrow • Standardize in-memory data • Storage, schema, … • Reuse across apps & libs arrow.apache.org

Slide 47

Slide 47 text

@deanwampler polyglotprogramming.com/talks anyscale.com ray.io