Beautiful Complexity

Experienced software developers have been exhorting us for decades to build simple, clear, modular systems and to avoid and reduce complexity at every turn. In software development, and in the English language, we have often ignored this advice and created ambiguous, vague, complex monoliths: The very word 'complexity' is one of them.

In this talk, we'll explore three faces of complexity of interest to programmers: "Big O" complexity, "Tar Pit" complexity, and "NetSci" complexity. In Network Science, I have found inherent, essential complexity that is inspiring, insightful, surprising, ubiquitous...and most importantly for engineers...pragmatic and useful. I'd like to share with you new tools from this emerging discipline that I've been using to describe, understand, and extend elegant, performant software systems and teams.

The presentation is done in a terse Lessig-style: Read the accompanying blog post at http://bobbynorton.com/posts/beautiful-complexity/ for more details.

Transcript

1. People fear complexity…it’s overwhelming
   Beautiful Complexity
   !
   Bobby Norton 8th Light University
   April 17, 2015
   

   View Slide

2. “Controlling complexity is the
   essence of computer programming.”
   !
   Brian Kernighan
   1976
   

   View Slide

3. View Slide

4. View Slide

5. !
   Rule 2. Measure. Don't tune for speed
   until you've measured, and even then
   don't unless one part of the code
   overwhelms the rest.
   !
   Rule 3. Fancy algorithms are slow when n
   is small, and n is usually small.
   Rob Pike - Notes on C Programming
   

   View Slide

6. View Slide

7. !
   1. Any live cell with fewer than two live neighbors dies, as if caused by under-population.
   2. Any live cell with two or three live neighbors lives on to the next generation.
   3. Any live cell with more than three live neighbors dies, as if by overcrowding.
   4. Any dead cell with exactly three live neighbors becomes a live cell, as if by reproduction.
   

   View Slide

8. View Slide

9. “The stable pattern that results from bunnies
   has 1744 cells and consists of 136 blinkers
   (including 14 traffic lights), 109 blocks, 65
   beehives (including three honey farms), 40
   gliders, 18 boats, 18 loaves, seven ships, four
   tubs, three ponds and two toads.”
   !
   http://www.conwaylife.com/wiki/Bunnies
   

   View Slide

10. View Slide

11. Each of the 10,000 cells in this 100x100
    grid can be in 2 possible starting states.
    2
    10,000
    

    View Slide

12. 2^8,192 ~ 10^2,466
    

    View Slide

13. Number of atoms in the known universe:
    10^80
    

    View Slide

14. View Slide

15. View Slide

16. The Art of Unix Programming!
    

    View Slide

17. Q Ethan McCallum, Bobby Norton, et al: Bad Data Handbook, Chapter 13: Crouching Table, Hidden Network
    

    View Slide

18. View Slide

19. has_and_belongs_to_many :assets
    !
    has_and_belongs_to_many :services
    

    View Slide

20. View Slide

21. View Slide

22. has_and_belongs_to_many :assets
    !
    has_and_belongs_to_many :services
    

    View Slide

23. View Slide

24. View Slide

25. has_and_belongs_to_many :assets
    !
    has_and_belongs_to_many :services
    

    View Slide

26. View Slide

27. !
    Rule 5. Data dominates. If you've chosen
    the right data structures and organized
    things well, the algorithms will almost
    always be self-evident. Data structures,
    not algorithms, are central to
    programming.
    Rob Pike - Notes on C Programming
    

    View Slide

28. نابز یریگدای لاح رد امش ایآ
    ؟دیتسه یسراف
    ؟درذگیم امش نیرمت زا تدم هچ
    ؟دیا هتفرگ دای ردقچ نونک ات
    ار نتشون و ندناوخ ییاناوت امش ایآ
    ؟دیراد
    

    View Slide

29. Aya shoma dar hale yad giri-e zabane
    farsi hastid?
    !
    Che modat az tamrine shoma
    migozarad?
    !
    Ta konoon cheghadr yad gerefte ied?
    !
    Aya shoma tavanaie-e khandan va
    neveshtan ra darid?
    

    View Slide

30. Are you learning Farsi?
    !
    How long have you been practicing?
    !
    How much have you learned so far?
    !
    Are you able to read and write?
    

    View Slide

31. View Slide

32. The rate at which
    the rate of change
    of the rate of change
    is changing…
    

    View Slide

33. View Slide

34. There is a twofold reason for departing
    from the usual mathematical practice of
    using single letters for atomic symbols.
    First, computer programs frequently
    require hundreds of distinguishable
    symbols that must be formed from the 47
    characters that are printable by the IBM
    704 computer. Second, it is convenient to
    allow English words and phrases to stand
    for atomic entities for mnemonic reasons.
    !
    John McCarthy, 1960
    Recursive Functions of Symbolic Expressions and Their
    Computation by Machine, Part I
    

    View Slide

35. (define ((L-harmonic m k) local)
    (let ((q (coordinate local))
    (v (velocity local)))
    (- (* 1/2 m (square v))
    (* 1/2 k (square q)))))
    Functional Differential Geometry
    Gerald Jay Sussman and Jack Wisdom
    with Will Farr
    

    View Slide

36. Functional Differential Geometry
    Gerald Jay Sussman and Jack Wisdom
    with Will Farr
    The traditional use of Leibnitz’s notation and Newton’s
    notation is convenient in simple situations, but in more
    complicated situations it can be a serious handicap
    to clear reasoning.
    

    View Slide

37. Is it ‘complex’…as in, difficult?
    !
    Or just unfamiliar?
    

    View Slide

38. If you give someone Fortran, he has Fortran.
    !
    If you give someone Lisp, he has any language he pleases.
    !
    - Guy L. Steele Jr.
    

    View Slide

39. user> (source source)
    !
    (defmacro source
    "Prints the source code for the given symbol, if it can
    find it.
    This requires that the symbol resolve to a Var defined in a
    namespace for which the .clj is in the classpath.
    !
    Example: (source filter)"
    [n]
    `(println (or (source-fn '~n) (str "Source not found"))))
    

    View Slide

40. Lisp isn't a language, it's a building
    material.
    !
    - Alan Kay
    

    View Slide

41. The big idea is “messaging”…
    !
    The key in making great and growable systems is much more to
    design how its modules communicate rather than what their
    internal properties and behaviors should be. The Japanese have a
    small word - ma - for "that which is in between" - perhaps the
    nearest English equivalent is "interstitial".
    !
    Think of the internet - to live, it:
    !
    (a) has to allow many different kinds of ideas and realizations that
    are beyond any single standard and
    !
    (b) to allow varying degrees of safe interoperability between these
    ideas.
    !
    - Alan Kay, 1998, Squeak mailing list
    (http://c2.com/cgi/wiki?AlanKayOnMessaging)
    

    View Slide

42. [ -z $(diff -w <(openssl md5 derp-0.6.3.tgz | cut -d ' ' -f 2) derp-0.6.3.tgz.md5) ]
    

    View Slide

43. If a problem space is predictable, in theory we can
    find a solution via optimization.
    !
    If a problem space is not predictable, or it changes
    too fast, very probably optimization will offer
    obsolete solutions.
    !
    Carlos Gershenson
    Facing Complexity: Prediction vs. Adaptation
    

    View Slide

44. View Slide

45. View Slide

46. View Slide

47. View Slide

48. View Slide

49. View Slide

50. View Slide

51. View Slide

52. View Slide

53. netsci complexity
    Mapping the Structural Core of Human Cerebral Cortex!
    Hagmann, et al, PLOS Biology
    

    View Slide

54. View Slide

55. Many real networks, from the cell to the Internet, independent of
    their age, function, and scope, converge to similar architectures.
    !
    It is this universality that allowed researchers from different
    disciplines to embrace network theory as a common paradigm.
    Albert-László Barabási
    Scale-Free Networks: A Decade
    and Beyond
    24 JULY 2009 VOL 325 SCIENCE
    

    View Slide

56. An essential requirement of these products is that they meet the
    needs of those members of society who will actually use them.
    !
    This concept of fitness for use is universal.
    It applies to all goods and services, without exception.
    !
    The popular term for fitness for use is Quality,
    and our basic definition becomes:
    !
    Quality means fitness for use.
    !
    - Joseph Juran
    

    View Slide

57. View Slide

58. View Slide

59. View Slide

60. View Slide

61. View Slide

62. View Slide

63. View Slide

64. View Slide

65. A distributed system is one in which the failure
    of a computer you didn’t even know existed can
    render your own computer unusable.
    !
    Leslie Lamport
    Known for the Byzantine Generals' Problem, vector clocks, and the Paxos algorithm
    

    View Slide

66. View Slide

67. View Slide

68. View Slide

69. View Slide

70. View Slide

71. View Slide

72. You want to isolate your modules from the complexities
    of the organization as a whole, and design your systems
    such that each module is responsible (responds to) the
    needs of just that one business function.
    The crux of the Single Responsibility Principle.
    !
    This principle is about people.
    - “Uncle Bob” Martin
    http://blog.8thlight.com/uncle-bob/2014/05/08/SingleReponsibilityPrinciple.html
    

    View Slide

73. Maybe enabling and embracing
    complexity is the essence of computer
    programming: This implies composition
    from simple parts, clear interfaces,
    modularity, clarity, self-organization,
    iteration, and emergence.
    !
    Curb your reductionism.
    

    View Slide

74. View Slide

75. @bobbynorton
    bobby@testedminds.com
    

    View Slide

76. Image Credits
    1: Hurricane Danielle - @Astro_Wheels - Douglas H. Wheelock / NASA!
    2: Brian Kernighan - Forbes India magazine issue of 18 November, 2011 - !
    http://forbesindia.com/media/images/2011/Nov/topimg_17442_bwk_frank_600x400.jpg!
    3: Bubble sort - http://upload.wikimedia.org/wikipedia/commons/thumb/8/83/Bubblesort-edited-color.svg/512px-Bubblesort-edited-color.svg.png!
    4: http://en.wikipedia.org/wiki/Bubble_sort!
    6,7: http://www.conwaylife.com/wiki!
    8: original work - "Rabbits" - https://github.com/bobbyno/cellblock!
    10: Game of Life universal Turing machine, by Paul Rendell - http://pentadecathlon.com/lifeNews/2011/01/2011-01-10-overview.PNG!
    14: Hubble Ultra Deep Field - http://upload.wikimedia.org/wikipedia/commons/thumb/6/69/NASA-HS201427a-HubbleUltraDeepField2014-20140603.jpg/!
    1122px-NASA-HS201427a-HubbleUltraDeepField2014-20140603.jpg!
    15: La Brea Tar Pits - http://img163.imageshack.us/img163/9770/labreatarbubble.jpg!
    16: Sofware Tools - http://ecx.images-amazon.com/images/I/41%2BYuOIrEXL.jpg!
    17: original work - Bad Data Handbook, O’Reilly 2012 Chapter 13: Crouching Table, Hidden Network!
    18: Ruby on Rails - http://www.hotforsecurity.com/wp-content/uploads/2013/01/ruby-on-rails-steams-critical-security-patch.jpg!
    33: http://en.wikipedia.org/wiki/History_of_mathematical_notation!
    44: Berlin Wall at Brandenburg Gate - http://origins.osu.edu/sites/origins.osu.edu/files/ber.jpg!
    45: https://ardillanegra.files.wordpress.com/2014/05/2007-2009-financial-crisis.gif!
    46: http://cdn.theatlantic.com/static/mt/assets/science/assets_c/2013/06/healthcare-gov-homepage-thumb-570x348-125912.jpg!
    47: AP Photo/Korean Central News Agency - http://www.nti.org/media/images/Apr_24--NK.jpg!
    48: http://assets.nydailynews.com/polopoly_fs/1.1701127!/img/httpImage/image.jpg_gen/derivatives/article_635/north-korea.jpg!
    49: http://feelgrafix.com/data_images/out/26/953992-minimalism.jpg!
    50: https://lh3.googleusercontent.com/-SotJJ2DlNrg/U18vq1JM5PI/AAAAAAAAANk/bCVDnkK6GLY/s0-d/18364-doge-simplistic-doge.png!
    51: Rube Goldberg - http://blog.codinghorror.com/content/images/uploads/2005/04/6a0120a85dcdae970b0128776fae11970c-pi.png!
    52: http://www.goodwp.com/large/201206/22319.jpg!
    53: Mapping the Structural Core of Human Cerebral Cortex, Hagmann, et al, PLOS Biology!
    54: Albert-László Barabási, Scale-Free Networks: A Decade and Beyond, 24 JULY 2009 VOL 325 SCIENCE!
    57: Arpanet Logical Map, 1977 - http://en.wikipedia.org/wiki/ARPANET#/media/File:Arpanet_logical_map,_march_1977.png!
    58: Internet map - https://duchafria.files.wordpress.com/2013/07/internet.jpg?w=300&h=300!
    59,60: 415,808 nodes, 283,317 edges in a Twitter network - https://dhs.stanford.edu/gephi-workshop/twitter-network-gallery/!
    61: Gource: Openvswitch Repo Visualization (fast) - http://i.ytimg.com/vi/orRphCzIA38/maxresdefault.jpg!
    62: CPAN Explorer, screenshot - http://cpan-explorer.org/2009/07/28/new-version-of-the-distributions-map-for-yapceu/!
    63: CPAN Explorer, screenshot - http://cpan-explorer.org/2009/07/28/version-of-the-authors-graph-for-yapceu/!
    64: Manuel Lima, the Power of Networks, via http://blogs.perl.org/users/philip_durbin/images/perl-mslima.jpg!
    66: Distributed application architecture diagram - original work!
    67: RabbitMQ messaging topology - original work!
    68: yEd Graph Editor: http://www.yworks.com/en/products/yfiles/yed/!
    69-71: Obama for America AWS diagram by Miles Ward, via http://www.williamhertling.com/2013/07/printable-obama-for-america-aws-architecture/!
    74: http://blogs.bfischool.org.s3.amazonaws.com/wp-content/uploads/2012/04/nautilus-shell-golden-ratio.png!
    !
    This presentation may contain copyrighted images the use of which has not always been specifically authorized by the copyright owner.!
    This material is being used for commentary, criticism, and educational purposes and constitutes a 'fair use' of any such copyrighted material !
    as provided for in section 107 of the US Copyright Law.
    

    View Slide