Beautiful Complexity

Experienced software developers have been exhorting us for decades to build simple, clear, modular systems and to avoid and reduce complexity at every turn. In software development, and in the English language, we have often ignored this advice and created ambiguous, vague, complex monoliths: The very word 'complexity' is one of them.

In this talk, we'll explore three faces of complexity of interest to programmers: "Big O" complexity, "Tar Pit" complexity, and "NetSci" complexity. In Network Science, I have found inherent, essential complexity that is inspiring, insightful, surprising, ubiquitous...and most importantly for engineers...pragmatic and useful. I'd like to share with you new tools from this emerging discipline that I've been using to describe, understand, and extend elegant, performant software systems and teams.

The presentation is done in a terse Lessig-style: Read the accompanying blog post at http://bobbynorton.com/posts/beautiful-complexity/ for more details.

Transcript

1. People fear complexity…it’s overwhelming Beautiful Complexity ! Bobby Norton 8th Light University April 17, 2015
2. “Controlling complexity is the essence of computer programming.” ! Brian Kernighan 1976
5. ! Rule 2. Measure. Don't tune for speed until you've measured, and even then don't unless one part of the code overwhelms the rest. ! Rule 3. Fancy algorithms are slow when n is small, and n is usually small. Rob Pike - Notes on C Programming
7. ! 1. Any live cell with fewer than two live neighbors dies, as if caused by under-population. 2. Any live cell with two or three live neighbors lives on to the next generation. 3. Any live cell with more than three live neighbors dies, as if by overcrowding. 4. Any dead cell with exactly three live neighbors becomes a live cell, as if by reproduction.
9. “The stable pattern that results from bunnies has 1744 cells and consists of 136 blinkers (including 14 traffic lights), 109 blocks, 65 beehives (including three honey farms), 40 gliders, 18 boats, 18 loaves, seven ships, four tubs, three ponds and two toads.” ! http://www.conwaylife.com/wiki/Bunnies
11. Each of the 10,000 cells in this 100x100 grid can be in 2 possible starting states. 2 10,000
12. 2^8,192 ~ 10^2,466
13. Number of atoms in the known universe: 10^80
16. The Art of Unix Programming!
17. Q Ethan McCallum, Bobby Norton, et al: Bad Data Handbook, Chapter 13: Crouching Table, Hidden Network
19. has_and_belongs_to_many :assets ! has_and_belongs_to_many :services
22. has_and_belongs_to_many :assets ! has_and_belongs_to_many :services
25. has_and_belongs_to_many :assets ! has_and_belongs_to_many :services
27. ! Rule 5. Data dominates. If you've chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming. Rob Pike - Notes on C Programming
28. نابز یریگدای لاح رد امش ایآ ؟دیتسه یسراف ؟درذگیم امش نیرمت زا تدم هچ ؟دیا هتفرگ دای ردقچ نونک ات ار نتشون و ندناوخ ییاناوت امش ایآ ؟دیراد
29. Aya shoma dar hale yad giri-e zabane farsi hastid? ! Che modat az tamrine shoma migozarad? ! Ta konoon cheghadr yad gerefte ied? ! Aya shoma tavanaie-e khandan va neveshtan ra darid?
30. Are you learning Farsi? ! How long have you been practicing? ! How much have you learned so far? ! Are you able to read and write?
32. The rate at which the rate of change of the rate of change is changing…
34. There is a twofold reason for departing from the usual mathematical practice of using single letters for atomic symbols. First, computer programs frequently require hundreds of distinguishable symbols that must be formed from the 47 characters that are printable by the IBM 704 computer. Second, it is convenient to allow English words and phrases to stand for atomic entities for mnemonic reasons. ! John McCarthy, 1960 Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I
35. (define ((L-harmonic m k) local) (let ((q (coordinate local)) (v (velocity local))) (- (* 1/2 m (square v)) (* 1/2 k (square q))))) Functional Differential Geometry Gerald Jay Sussman and Jack Wisdom with Will Farr
36. Functional Differential Geometry Gerald Jay Sussman and Jack Wisdom with Will Farr The traditional use of Leibnitz’s notation and Newton’s notation is convenient in simple situations, but in more complicated situations it can be a serious handicap to clear reasoning.
37. Is it ‘complex’…as in, difficult? ! Or just unfamiliar?
38. If you give someone Fortran, he has Fortran. ! If you give someone Lisp, he has any language he pleases. ! - Guy L. Steele Jr.
39. user> (source source) ! (defmacro source "Prints the source code for the given symbol, if it can find it. This requires that the symbol resolve to a Var defined in a namespace for which the .clj is in the classpath. ! Example: (source filter)" [n] `(println (or (source-fn '~n) (str "Source not found"))))
40. Lisp isn't a language, it's a building material. ! - Alan Kay
41. The big idea is “messaging”… ! The key in making great and growable systems is much more to design how its modules communicate rather than what their internal properties and behaviors should be. The Japanese have a small word - ma - for "that which is in between" - perhaps the nearest English equivalent is "interstitial". ! Think of the internet - to live, it: ! (a) has to allow many different kinds of ideas and realizations that are beyond any single standard and ! (b) to allow varying degrees of safe interoperability between these ideas. ! - Alan Kay, 1998, Squeak mailing list (http://c2.com/cgi/wiki?AlanKayOnMessaging)
42. [ -z $(diff -w <(openssl md5 derp-0.6.3.tgz | cut -d ' ' -f 2) derp-0.6.3.tgz.md5) ]
43. If a problem space is predictable, in theory we can find a solution via optimization. ! If a problem space is not predictable, or it changes too fast, very probably optimization will offer obsolete solutions. ! Carlos Gershenson Facing Complexity: Prediction vs. Adaptation
53. netsci complexity Mapping the Structural Core of Human Cerebral Cortex! Hagmann, et al, PLOS Biology
55. Many real networks, from the cell to the Internet, independent of their age, function, and scope, converge to similar architectures. ! It is this universality that allowed researchers from different disciplines to embrace network theory as a common paradigm. Albert-László Barabási Scale-Free Networks: A Decade and Beyond 24 JULY 2009 VOL 325 SCIENCE
56. An essential requirement of these products is that they meet the needs of those members of society who will actually use them. ! This concept of fitness for use is universal. It applies to all goods and services, without exception. ! The popular term for fitness for use is Quality, and our basic definition becomes: ! Quality means fitness for use. ! - Joseph Juran
65. A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable. ! Leslie Lamport Known for the Byzantine Generals' Problem, vector clocks, and the Paxos algorithm
72. You want to isolate your modules from the complexities of the organization as a whole, and design your systems such that each module is responsible (responds to) the needs of just that one business function. The crux of the Single Responsibility Principle. ! This principle is about people. - “Uncle Bob” Martin http://blog.8thlight.com/uncle-bob/2014/05/08/SingleReponsibilityPrinciple.html
73. Maybe enabling and embracing complexity is the essence of computer programming: This implies composition from simple parts, clear interfaces, modularity, clarity, self-organization, iteration, and emergence. ! Curb your reductionism.
75. @bobbynorton bobby@testedminds.com
76. Image Credits 1: Hurricane Danielle - @Astro_Wheels - Douglas H. Wheelock / NASA! 2: Brian Kernighan - Forbes India magazine issue of 18 November, 2011 - ! http://forbesindia.com/media/images/2011/Nov/topimg_17442_bwk_frank_600x400.jpg! 3: Bubble sort - http://upload.wikimedia.org/wikipedia/commons/thumb/8/83/Bubblesort-edited-color.svg/512px-Bubblesort-edited-color.svg.png! 4: http://en.wikipedia.org/wiki/Bubble_sort! 6,7: http://www.conwaylife.com/wiki! 8: original work - "Rabbits" - https://github.com/bobbyno/cellblock! 10: Game of Life universal Turing machine, by Paul Rendell - http://pentadecathlon.com/lifeNews/2011/01/2011-01-10-overview.PNG! 14: Hubble Ultra Deep Field - http://upload.wikimedia.org/wikipedia/commons/thumb/6/69/NASA-HS201427a-HubbleUltraDeepField2014-20140603.jpg/! 1122px-NASA-HS201427a-HubbleUltraDeepField2014-20140603.jpg! 15: La Brea Tar Pits - http://img163.imageshack.us/img163/9770/labreatarbubble.jpg! 16: Sofware Tools - http://ecx.images-amazon.com/images/I/41%2BYuOIrEXL.jpg! 17: original work - Bad Data Handbook, O’Reilly 2012 Chapter 13: Crouching Table, Hidden Network! 18: Ruby on Rails - http://www.hotforsecurity.com/wp-content/uploads/2013/01/ruby-on-rails-steams-critical-security-patch.jpg! 33: http://en.wikipedia.org/wiki/History_of_mathematical_notation! 44: Berlin Wall at Brandenburg Gate - http://origins.osu.edu/sites/origins.osu.edu/files/ber.jpg! 45: https://ardillanegra.files.wordpress.com/2014/05/2007-2009-financial-crisis.gif! 46: http://cdn.theatlantic.com/static/mt/assets/science/assets_c/2013/06/healthcare-gov-homepage-thumb-570x348-125912.jpg! 47: AP Photo/Korean Central News Agency - http://www.nti.org/media/images/Apr_24--NK.jpg! 48: http://assets.nydailynews.com/polopoly_fs/1.1701127!/img/httpImage/image.jpg_gen/derivatives/article_635/north-korea.jpg! 49: http://feelgrafix.com/data_images/out/26/953992-minimalism.jpg! 50: https://lh3.googleusercontent.com/-SotJJ2DlNrg/U18vq1JM5PI/AAAAAAAAANk/bCVDnkK6GLY/s0-d/18364-doge-simplistic-doge.png! 51: Rube Goldberg - http://blog.codinghorror.com/content/images/uploads/2005/04/6a0120a85dcdae970b0128776fae11970c-pi.png! 52: http://www.goodwp.com/large/201206/22319.jpg! 53: Mapping the Structural Core of Human Cerebral Cortex, Hagmann, et al, PLOS Biology! 54: Albert-László Barabási, Scale-Free Networks: A Decade and Beyond, 24 JULY 2009 VOL 325 SCIENCE! 57: Arpanet Logical Map, 1977 - http://en.wikipedia.org/wiki/ARPANET#/media/File:Arpanet_logical_map,_march_1977.png! 58: Internet map - https://duchafria.files.wordpress.com/2013/07/internet.jpg?w=300&h=300! 59,60: 415,808 nodes, 283,317 edges in a Twitter network - https://dhs.stanford.edu/gephi-workshop/twitter-network-gallery/! 61: Gource: Openvswitch Repo Visualization (fast) - http://i.ytimg.com/vi/orRphCzIA38/maxresdefault.jpg! 62: CPAN Explorer, screenshot - http://cpan-explorer.org/2009/07/28/new-version-of-the-distributions-map-for-yapceu/! 63: CPAN Explorer, screenshot - http://cpan-explorer.org/2009/07/28/version-of-the-authors-graph-for-yapceu/! 64: Manuel Lima, the Power of Networks, via http://blogs.perl.org/users/philip_durbin/images/perl-mslima.jpg! 66: Distributed application architecture diagram - original work! 67: RabbitMQ messaging topology - original work! 68: yEd Graph Editor: http://www.yworks.com/en/products/yfiles/yed/! 69-71: Obama for America AWS diagram by Miles Ward, via http://www.williamhertling.com/2013/07/printable-obama-for-america-aws-architecture/! 74: http://blogs.bfischool.org.s3.amazonaws.com/wp-content/uploads/2012/04/nautilus-shell-golden-ratio.png! ! This presentation may contain copyrighted images the use of which has not always been specifically authorized by the copyright owner.! This material is being used for commentary, criticism, and educational purposes and constitutes a 'fair use' of any such copyrighted material ! as provided for in section 107 of the US Copyright Law.