An extended version of my slides for DevOps Days Raleigh with more-text for an online audience.
Speaking for the Dead: Is
"Waterfall" and "Monolithic"
@LASERLLAMA - 9.6.2016
u In Early 2012, I created a project called Ansible.
u I named it after Ender’s Game, one of my favorite books.
u Orson Scott Card wrote “Speaker for the Dead” as a sequel, which is not
one of my favorite books.
u Speaking for the dead was essentially a recounting of someone’s life.
u An honest eulogy. Tell the good and bad things, describe the person as
they were. Be empathetic and know them as they knew themselves.
u I began writing software full time at IBM in 2001. Not that long ago. I’m
also not dead.
u But we’ve declared Waterfall and Monolithic Architecture dead. Perhaps
prematurely. Let’s talk about that.
u Two specific topics today:
u “Waterfall” vs “Agile”
u “Monolithic” vs “Microservices”
u Two general topic:
u Labels, spin, and assumptions
u An appreciation of the very recent past
First: A Theory
u My theory is, we get bored at work.
u We tend to like a challenge.
u Things became too easy, and we tried to do them differently.
u Different was new, or different was hard.
u Different is not always better.
u Lots of new people coming into the field all of the time.
u We appear to have an age-range problem that reduces our pool of historical
u We also read less these days.
u And we get caught up in labels and spin for the “one way” to do things.
u There is no one way.
Labels Apply Spin
Good Connotation Bad Connotation
Affordable Care Act “Obamacare”
USA PATRIOT ACT
Big Design Up Front
u The term “Ansible” was originally coined by Ursula LeGuinn.
u In Earthsea, she writes of “True Names” – knowing the real true name of
something gives us complete power over it.
u We must know things as they are, rather than reducing them to labels.
u “Waterfall” and “Monolithic” have been stamped with negative
connotation that prevents us from thinking about them properly.
“Waterfall” and “Iterative
u In the early 2000s, waterfall was the norm.
u Requirements – Design – Implementation - Test
u Engineering was engineering – greater responsibility, less hand holding.
u Plenty of time to design solutions and research.
u Marketing Requirements Documents
u Engineering Requirements Documents
u Questions, comments, documented mutual agreement
u Status meetings usually once a week
u Accurate view of strengths of various team members
u Accurate view of dependencies between tasks, task order
The Gantt Chart
Why don’t we
Agile (especially Scrum)
u “Big Design Up Front” spins the idea of taking weeks in design as a “bad”
thing. Development becomes less computer-sciencey.
u Engineers are viewed as interchangeable, and estimate each other’s tasks far
away from skillsets. This is less human.
u Estimates are encouraged to be thought of in “points” not time while release
dates remain time focused in most industries, leading to dissatisfaction with
inaccurate constantly slipping estimates, which erodes trust.
u Scope and what is delivered is more unpredictable.
u Focus on status is actually increased – standups, weekly reviews, retrospectives
u Games make work feel more like Kindergarten - “fist of five”, “planning poker”
u Crunch time becomes constant
Even In Lightweight Agile Systems
u Backlog does not typically model task inter-relationships
u Management does not endorse long periods of design or experimentation
u Estimates from engineers are scrutinized and continually asked to be
u Requirements are often single sentences
u There is very little feedback on requirements.
u There is no sign-off required on understanding of requirements.
u Testing gets crunched to the end of a cycle and has to struggle to fit in.
Lessons from Waterfall
u Design matters.
u Not all engineers are created alike.
u Tasks have dependencies.
u Estimates in days imply realism.
u Treat engineers like professionals.
u Requirements should not change constantly.
u Extensive test cycles are fine, transition into bugfix modes.
u ”Iterative Development” is basically “Lightweight Agile”
u Let’s quit throwing “Waterfall” under the bus.
Lessons Refined With Agile
u Don’t let one person do all the design because that’s not fair, and
probably not optimal
u Do have the team ask lots of questions of the product owner
u It is good to be able to re-order priorities between iterations or be able to
insert new ones
u The more “web” an application is and the smaller the features are, the
more an agile release cycle makes sense.
u Test driven development makes for more reliable code.
What’s So Bad About A Monolith?
Monolithic Application Architecture
Professor Plum with the Metadata
Service in us-east-1
Example Worldwide Microservices
u ~150 services
u Many thousands of Jenkins builds per day
u 8 Amazon regions (dev, stage, many prod regions)
u ~1000s of EC2 instances
u Realistically could be modelled in ~50 EC2 instances
u Jenkins configurations drift
u Security enforcement drifts
u Code sharing suboptimal
u “DevOps” writing LOTS of custom code to manage it.
Inefficiency Kills Polar Bears
Lessons Learned From Monoliths
u Increased ability to develop on one laptop
u Interfaces can benefit from static typing vs design-by-contract
u Very small numbers of different server/image types in production
u Local function calls are much faster than RPC
u Not many instances sitting around with 5% utilization
u No need for “containers” to get around above utilization problem
u Greatly simplified debugging
u Easy code sharing
u Very simple version upgrades
u Was created as a technical solution to a people problem
u Teams did not want to do group code review
u Teams wanted to deploy on their own schedules
u Teams wanted to pick different languages
u Teams wanted to communicate with other teams less
u Design by contract was effective, but increased test demands
u However API changes and upgrades actually INCREASE need for
u Avoids controversial design debates by also avoiding object design
u Creates an infinite sea of new management challenges.
Lessons Learned From Microservices
u Design by contract can prevent teams from being too tied down to
language choices as a form of expression and productivity.
u Design by contract encourages documenting of interfaces between
u An API focus encourages good decoupling from UI layers and avoids
components becoming too tightly coupled. But this could also have
been done with architect roles.
u Waterfall/Iterative Development has been maligned, but comes from an era where there was more respect for
engineering, and more discipline in it. Understand dates and requirements and there is less stress.
u Monolithic architectures are more efficient, allow greater Object Oriented Design principles, allow better type checking
and debugging, are simpler and are easier to work on. They are also cheaper and more environmentally friendly.
u Scrum came from consultant-ware and uses religious-feeling intimiation to gain a foothold, if you don’t like it, you must
not be doing it right. Still, ”Hakuna Matata” vibes from lightweight agile gets people to care a little less about
scope/dates are well meaning, if not unrealistic.
u Microservices come from teams who have had trouble coordinating and who did not want top-down design dictations.
This is a natural and understandable part of the human experience, but needs to be checked somewhat or it risks
sabotaging better design opportunities and the development experience.
u Remember what worked from things from the past before discarding them. Allow Computer Science to be both science
(design) and formal engineering (time), not merely labor and timesheets.
u There’s something to learn from everything. Try to separate the labels from the true meanings of things to avoid
connotation clouding thought.