Speaking for the Dead: Is "Waterfall" and "Monolithic" Actually Good? MICHAEL DEHAAN @LASERLLAMA - 9.6.2016

PRELUDE u In Early 2012, I created a project called Ansible. u I named it after Ender’s Game, one of my favorite books. u Orson Scott Card wrote “Speaker for the Dead” as a sequel, which is not one of my favorite books.

PRELUDE u Speaking for the dead was essentially a recounting of someone’s life. u An honest eulogy. Tell the good and bad things, describe the person as they were. Be empathetic and know them as they knew themselves. u I began writing software full time at IBM in 2001. Not that long ago. I’m also not dead. u But we’ve declared Waterfall and Monolithic Architecture dead. Perhaps prematurely. Let’s talk about that.

TODAY u Two specific topics today: u “Waterfall” vs “Agile” u “Monolithic” vs “Microservices” u Two general topic: u Labels, spin, and assumptions u An appreciation of the very recent past

First: A Theory u My theory is, we get bored at work. u We tend to like a challenge. u Things became too easy, and we tried to do them differently. u Different was new, or different was hard. u Different is not always better. u Lots of new people coming into the field all of the time. u We appear to have an age-range problem that reduces our pool of historical experience. u We also read less these days. u And we get caught up in labels and spin for the “one way” to do things. u There is no one way.

Labels Apply Spin Good Connotation Bad Connotation Affordable Care Act “Obamacare” USA PATRIOT ACT Agile Waterfall Microservices Monolithic Big Design Up Front Enterprisey

TRUE NAMES u The term “Ansible” was originally coined by Ursula LeGuinn. u In Earthsea, she writes of “True Names” – knowing the real true name of something gives us complete power over it. u We must know things as they are, rather than reducing them to labels. u “Waterfall” and “Monolithic” have been stamped with negative connotation that prevents us from thinking about them properly.

“Waterfall” and “Iterative Development” u In the early 2000s, waterfall was the norm. u Requirements – Design – Implementation - Test u Engineering was engineering – greater responsibility, less hand holding. u Plenty of time to design solutions and research. u Marketing Requirements Documents u Engineering Requirements Documents u Questions, comments, documented mutual agreement u Status meetings usually once a week u Accurate view of strengths of various team members u Accurate view of dependencies between tasks, task order

The Gantt Chart Why don’t we do these anymore?

Agile (especially Scrum) u “Big Design Up Front” spins the idea of taking weeks in design as a “bad” thing. Development becomes less computer-sciencey. u Engineers are viewed as interchangeable, and estimate each other’s tasks far away from skillsets. This is less human. u Estimates are encouraged to be thought of in “points” not time while release dates remain time focused in most industries, leading to dissatisfaction with inaccurate constantly slipping estimates, which erodes trust. u Scope and what is delivered is more unpredictable. u Focus on status is actually increased – standups, weekly reviews, retrospectives u Games make work feel more like Kindergarten - “fist of five”, “planning poker” u Crunch time becomes constant

Even In Lightweight Agile Systems u Backlog does not typically model task inter-relationships u Management does not endorse long periods of design or experimentation u Estimates from engineers are scrutinized and continually asked to be made shorter u Requirements are often single sentences u There is very little feedback on requirements. u There is no sign-off required on understanding of requirements. u Testing gets crunched to the end of a cycle and has to struggle to fit in.

Lessons from Waterfall u Design matters. u Not all engineers are created alike. u Tasks have dependencies. u Estimates in days imply realism. u Treat engineers like professionals. u Requirements should not change constantly. u Extensive test cycles are fine, transition into bugfix modes. u ”Iterative Development” is basically “Lightweight Agile” u Let’s quit throwing “Waterfall” under the bus.

Lessons Refined With Agile u Don’t let one person do all the design because that’s not fair, and probably not optimal u Do have the team ask lots of questions of the product owner u It is good to be able to re-order priorities between iterations or be able to insert new ones u The more “web” an application is and the smaller the features are, the more an agile release cycle makes sense. u Test driven development makes for more reliable code.

What’s So Bad About A Monolith?

Monolithic Application Architecture Load Balancer app app app Message bus tasks tasks tasks

Microservices

Professor Plum with the Metadata Service in us-east-1

Example Worldwide Microservices Shop u ~150 services u Many thousands of Jenkins builds per day u 8 Amazon regions (dev, stage, many prod regions) u ~1000s of EC2 instances u Realistically could be modelled in ~50 EC2 instances u Jenkins configurations drift u Security enforcement drifts u Code sharing suboptimal u “DevOps” writing LOTS of custom code to manage it.

Inefficiency Kills Polar Bears

Lessons Learned From Monoliths u Increased ability to develop on one laptop u Interfaces can benefit from static typing vs design-by-contract u Very small numbers of different server/image types in production u Local function calls are much faster than RPC u Not many instances sitting around with 5% utilization u No need for “containers” to get around above utilization problem u Greatly simplified debugging u Easy code sharing u Very simple version upgrades

Microservices u Was created as a technical solution to a people problem u Teams did not want to do group code review u Teams wanted to deploy on their own schedules u Teams wanted to pick different languages u Teams wanted to communicate with other teams less u Design by contract was effective, but increased test demands u However API changes and upgrades actually INCREASE need for communication u Avoids controversial design debates by also avoiding object design u Creates an infinite sea of new management challenges.

Hmm. Metal VM Container Cloud SW Container Container App Not Simpler.

Lessons Learned From Microservices u Design by contract can prevent teams from being too tied down to language choices as a form of expression and productivity. u Design by contract encourages documenting of interfaces between components. u An API focus encourages good decoupling from UI layers and avoids components becoming too tightly coupled. But this could also have been done with architect roles.

Conclusions u Waterfall/Iterative Development has been maligned, but comes from an era where there was more respect for engineering, and more discipline in it. Understand dates and requirements and there is less stress. u Monolithic architectures are more efficient, allow greater Object Oriented Design principles, allow better type checking and debugging, are simpler and are easier to work on. They are also cheaper and more environmentally friendly. u Scrum came from consultant-ware and uses religious-feeling intimiation to gain a foothold, if you don’t like it, you must not be doing it right. Still, ”Hakuna Matata” vibes from lightweight agile gets people to care a little less about scope/dates are well meaning, if not unrealistic. u Microservices come from teams who have had trouble coordinating and who did not want top-down design dictations. This is a natural and understandable part of the human experience, but needs to be checked somewhat or it risks sabotaging better design opportunities and the development experience. u Remember what worked from things from the past before discarding them. Allow Computer Science to be both science (design) and formal engineering (time), not merely labor and timesheets. u There’s something to learn from everything. Try to separate the labels from the true meanings of things to avoid connotation clouding thought.