Share
Embed
Transcript
43 slides
-
Fear Driven Development Confronting our inner daemons this is a
talk I’ve wanted to give since noticing a lot of what core services did while I ran it was dictated by fear (mostly of BDB/partie)…it struck me that fear caused a lot of unnecessary paralysis, which could be avoided if we considered how to confront fear in software development -
What is fear? A feeling of agitation and anxiety caused
by the presence of imminent danger the recognition of a perceived threat…it’s a survival mechanism that initiates our fight-or-flight response -
What causes fear? most people know what fear is but
perhaps don’t know what actually causes it -
Specific fears develop as a result of learning fear can
be acquired by a frightening traumatic accident…if you were bitten by a dog as a child, it would be natural for you to subsequently fear dogs -
Certain fears are more common than others (e.g. animals, heights)
Also easier to induce in the laboratory…a phenomenon is known as preparedness. Because early humans that were quick to fear dangerous situations were more likely to survive and reproduce, preparedness is theorized to be a genetic effect that is the result of natural selection. -
Emotional contagion the tendency for two individuals to emotionally converge
if you hang out with people who are afraid of the dark you’re likely to become afraid of the dark -
Fear is contagious We learn fear regardless of whether we
have experienced trauma or if we have observed the fear in others if you see a surfer get eaten by a shark, you’re going to be scared of sharks -
Negative emotions are usually more infectious than positive ones people
with negative emotions may be more likely to drag down people with positive emotions -
This is terrifying for a large engineering organization negative emotions,
like fear, elicit a stronger reaction and are more contagious than positive emotions…in larger organizations it’s like trying to stop the flu from making the rounds at the office -
Fear is additive • you can be scared of the
dark • you can be scared of the bogeyman • if you fear both…how does the bogeyman in the dark make you feel? you’re afraid of sharks…you’re afraid of water…SHARKS IN THE WATER…FUCK THE BEACH! you’re afraid of dogs…you’re afraid of spiders…DOGS WITH 8 LEGS! -
Fear compounds even small fears left to linger over long
periods of time can become large fears urban legends run amok someone dies in a tunnel…cause unknown…mystery case…HAUNTED CAVE -
I fear databases Ironically I was paid to build one
here databases are terrifying…they’re complicated and we call on them to store things that would put us out of business if we lost them…we put them on pedestals, believe only experts can build them ! I fear other things as well, but my experience working on partie developed most of my thoughts around fear and software development -
What does Yammer fear? why are we actually here? why
am I giving this talk? -
Refactoring how many ways can we convince ourselves out of
breaking up workfeed? how long have feedie/flatterie lingered on Scala? -
Performance raise your hand if something you did backed up
one or more important queues -
Scale raise your hand if you've taken the site down
by directing too much traffic at something -
Upgrades who wants to go upgrade BDB with me after
this? who's running database migrations today? what about backfills? -
Deploying basically roll up every fear we just discussed and
experience them all at once -
So what? What's the downside of fear in software development?
so what? we’re afraid. is it actually causing problems? -
Fear induced loss of agility Analysis paralysis looking for the
best/safest solution BDB upgrades in core services decomposing work feed -
Fear erects barricades around the code and systems that need
the most attention Fear will keep you from improving things…"that shit's scary, DON'T GO IN THERE!" feedie’s data model workfeed’s data model artie’s ability to DDoS the site -
Fear creates local maximums you won't explore areas you're afraid
of, even if that's where the biggest gains are to be had -
How I’ve dealt with my fears if I came only
with problems and no solutions then this would be a terrible talk -
Exposure therapy exposure to the feared object or context, without
any danger, in order to overcome anxiety -
Testing as a safety harness back in the day, partie
was the best tested code in the company a broken test is cheap and won't hurt anyone/anything most importantly, I had a series of tests that broke BDB every way I could…tests verified that BDB behaved how I expected and that partie could at least identify the failure and potentially fix it -
Development and staging are your safe zones I broke partie
in every way I could conceive before it left my computer…and then I put it back together reduce things to a smaller stage…it’s learning on the practice field vs. learning in Yankee Stadium make a mistake in Yankee Stadium and get sent back to the minors…make a mistake on the practice field and you do the drill 10 more times -
Build confidence on non-critical systems we did partie/BDB/Java upgrades on
services like prankie and whoville first because we knew we could lose all the data and still be fine -
Double dispatch when replacing existing functionality we kept writing feeds
to the old postgresql messaging db while also writing to feedie we slowly increased read traffic to feedie, switching it back when problems arose durable queues meant we could take feedie down entirely without losing deliveries even after switching to 100% reads on feedie, we kept double dispatching for months because we could -
Hack Day! it's low risk, even if you can't put
it back together you still had fun -
Sharing my fears Someone else has probably seen or done
this or something like it before I got in touch with the BDB developers, they reassured you there's nothing to be afraid of -
Organizational selective amnesia Let the new person do it (Hi,
Vivek!) they don't know that repo/library/database is haunted if things aren’t really to be scared, having someone who isn’t scared is invaluable -
Decatastrophizing "If the feared event or object happened, what would
occur then?" partie's 3-way replication and quorum elections/acks was tremendous for my confidence…once I'd seen it work with my own eyes -
Make the time to test catch problems early nobody's scared
of well tested code -
Do your research ahead of time read up on things
and perform actual research…like in a laboratory we suck at doing and scheduling research…we have to get better at it -
More hack day projects focusing on the things we fear
get under that hood and start yanking cables…is it as bad as we've made it out to be? best case…it's not bad and we can ship whatever you do worst case…it's bad, but now we know a lot more about the bad -
None of this should impact your personal life we need
to set up the structures that make this part of the expected day-to-day experience heroes can do this today, but heroes aren't sustainable -
Don't let anyone bully you out of what you feel
to be adequate testing or research it’s up to us to address the fear, otherwise we’re going to live in fear forever -
Ending the vicious cycle requires us to confront our biggest
fear… authority we are all empowered to address this fear as we encounter and experience it…if you feel otherwise, remember…engineers have the numbers